Puppy 245mpi cluster made easy

A home for all kinds of Puppy related projects
Message
Author
whatupuppy
Posts: 17
Joined: Tue 18 Jan 2011, 02:07

#41 Post by whatupuppy »

Update:
Well things did not go well with my friend this weekend.

So this have to be put on hold for now.

Also, failing hard drive on old laptops causing puppy to hang at Uncompressing linux..OK.booting Linux kernel.

Any suggestion beside buying a new harddrive is welcomed.

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#42 Post by Master_wrong »

I was about to suggest buying flashdisk....

so there is no cdrom, i guess ?
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#43 Post by Master_wrong »

i spend last night testing like couple dozens of program....
it end up only couple of program that compiled successfully

here is 2 interesting program that might be useful,
1. tentakel :
Tentakel is a program for executing the same command on many hosts in parallel
using various remote methods. It can make use of several sets of hosts that are
defined in a configuration file as groups.

It also supports an interactive mode that can be used for repeated commands.

The author uses tentakel to simultaneously install software on many
workstations or ask and set parameters on a linux compute cluster. With the
power of format strings tentakel can also be used for monitoring purposes.

Supported remote methods are ssh(1) and rsh(1). Both must be configured to
allow for password-less logins. Password-protected keyfiles for ssh can be
used with ssh-agent(1).

A plugin mechanism allows users to implement their own remote methods in
addition to the builtin ones.

For more information on available options please refer to the manpage
tentakel(1).



The project homepage is: http://tentakel.biskalar.de/
too bad i cannot get anything to work, i simply not smart enough to create config file, if anyone capable please let me know





2. unfs3-0.9.22.tar.gz :Great NFS for puppy, it capable of export and mount /root

INTRODUCTION
============

UNFS3 is a user-space implementation of the NFSv3 server
specification.

UNFS3 supports all NFSv3 procedures with the exception of the
READDIRPLUS procedure. It tries to provide as much information
to NFS clients as possible, within the limits possible from
user-space.

See the unfsd(8) manpage for restrictions imposed on NFS
operations (section RESTRICTIONS) and for possible races
with local file system activity (section BUGS).

It is not possible to export to netgroups or wildcard hostnames
via /etc/exports, all other addressing methods should work. The
following options are recognized in the exports file: ro, rw,
root_squash, all_squash, no_root_squash, no_all_squash. If
other options are present, they are ignored.

Cluster extensions compatible to the older ClusterNFS project
are supported when the source is configured with --enable-cluster.
all you need to do

1. create /etc/exports file: contain=====> /root (ro) --->for read only
open terminal at unfsd directrory
2. ./configure
3. make
4. make install
5. ./unfsd -s -d -p -n 4711 -m 4711 -e /etc/exports


6. at client : mkdir /mnt/try
7. copy unfsd,./configure,make,make install
8. mount -o port=4711,mountport=4711,mountvers=3,nfsvers=3,nolock,tcp 192.168.2.130:/root /mnt/try
Attachments
unfs3-test.png
(25.94 KiB) Downloaded 1110 times
tentakel.zip
(42.15 KiB) Downloaded 449 times
unfs3-0.9.22.tar.gz
(164.5 KiB) Downloaded 473 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#44 Post by Master_wrong »

made pet of unfs3 at http://www.murga-linux.com/puppy/viewtopic.php?t=64491

btw pxe booting work using
http://www.murga-linux.com/puppy/viewtopic.php?t=64107

also other program tested
pconsole does not work through daemons. The pconsole.sh script opens up a
number of connections to each node of your cluster of workstations. Then,
the pconsole binary attaches to the tty devices that these windows are
using, and it copies the input you type to all open connections......
.......
By default, pconsole tries to use SSH to make connections. If you do not have
'ssh', it will try to use 'telnet'.
Attachments
pconsole-test.png
(32.67 KiB) Downloaded 488 times
Last edited by Master_wrong on Sun 06 Feb 2011, 06:12, edited 2 times in total.

gcmartin

What is a cluster

#45 Post by gcmartin »

@Master _Wrong has done all of us a very valuable service. I am showing this picture here because over the past couple months, many who come here have an inaccurate perception of a cluster.

Image

This picture is worth a thousand words, as, it shows where to place the monitor, keyboard, and mouse for the cluster. In a true cluster, the nodes/workers/slave PCs DO NOT HAVE A MONITOR, KEYBOARD, OR MOUSE. But they all work together on behalf of the workload within the cluster.

Hope this helps
Attachments
Beowulf.png
(86.79 KiB) Downloaded 521 times
Last edited by gcmartin on Sat 05 Feb 2011, 18:28, edited 1 time in total.

gcmartin

#46 Post by gcmartin »

.

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#47 Post by Master_wrong »

Thank you gcmartin,

your effort in pxe documentation also help this project somehow.

btw here is some article about cluster:
http://www.clustermonkey.net//content/view/89/32/

the site also has ton of info about mpi
http://www.clustermonkey.net//content/view/14/28/
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
Aitch
Posts: 6518
Joined: Wed 04 Apr 2007, 15:57
Location: Chatham, Kent, UK

#48 Post by Aitch »

Wow, Master_wrong....you HAVE been busy, and got away from this thread...

http://www.murga-linux.com/puppy/viewtopic.php?t=34647

Since my initial interest, I've had a change in hardware direction, and now don't use much of my old scsi server boxes, as they use too much power, and make too much noise :wink:

However, I still think clustering is under-rated, especially in the puppyworld, with so much old hardware available

A couple or 3/more of useful links

http://www.linuxclusters.com/books/clus ... index.html

http://freshmeat.net/articles/linux-clustering-software

http://www.jumpingbean.co.za/linux-clus ... ailability

http://linuxreviews.org/howtos/virtual_ ... ter_howto/

http://www.ultramonkey.org/

http://www.gdargaud.net/Hack/ClusterNotes.html#Software

http://www.cs.wisc.edu/condor/

Good luck guys....some spaghetti examples

http://www.calvin.edu/~adams/research/m ... f/related/

Aitch :)

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#49 Post by Master_wrong »

:D
Hi Aitch, welcome aboard...
Wow, Master_wrong....you HAVE been busy, and got away from this thread...
yes it quite complicated...
but it was fun, still one thing is missing... howto incorporate pxe into it, yes i can do it already but how to make it more simple ?

I'm thinking about IBS atm, eh no no not irritable bowel syndrome :lol:

here:
http://www.stsx.org/projects/ibs/ibs-manual.html
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
Aitch
Posts: 6518
Joined: Wed 04 Apr 2007, 15:57
Location: Chatham, Kent, UK

#50 Post by Aitch »

Try iPXE

http://ipxe.org/

I just saw it demo'd at a linux fair in London....it looks cool, but don't ask me how to combine it with what you've done...that's quite an achievement, already :D

btw, here's a tempting project to build to run it on

http://www.mini-itx.com/projects/cluster/?p

Aitch :)

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#51 Post by Master_wrong »

@aitch
You can use iPXE to replace the existing PXE ROM on your network card, or you can chainload into iPXE to obtain the features of iPXE without the hassle of reflashing.
from above statement we know that
1. its used to replace existing pxe rom on network card ----> client side
2. chainload into ipxe ----> which mean pxe chainload into ipxe ---->client side

what we need is server side... such as IBS ----> Instant Boot Server

once we have the server surely we can use ipxe... maybe to create cloud puppy supercomputer ?
that would be cool 8)

Btw speaking of pxe....

I just realize that when remastering, when program ask about separate zl245332.... you have to answer "no"...
that's quite an achievement, already Very Happy
thanks aitch, the best part is you can actually install other newer version of mpich and use my script because the setting is the same :idea:
all you need is correct version of python...

for example you want version 1.1.1 of mpich2, you will need python 2.6 at least...
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
Aitch
Posts: 6518
Joined: Wed 04 Apr 2007, 15:57
Location: Chatham, Kent, UK

#52 Post by Aitch »

Master_wrong

Ah yes, I see

I realize this also plugs into Sunburnt's previous posts, here

http://www.murga-linux.com/puppy/viewtopic.php?t=5797

and

http://www.murga-linux.com/puppy/viewtopic.php?t=9198

which is back in 2006....

Haven't seen Sunburnt around for a while though....but this
Sunburnt wrote:I think a Puppy server - client setup on a CD is what John Murga has in mind.
Keep an eye out for possable [sic] developments.
seems to have finally become a distinct possible

Maybe time to think about some old thin client hardware on ebay....?

'....cloud puppy supercomputer' ?

mmm, please :lol:

Aitch :)

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#53 Post by Master_wrong »


seems to have finally become a distinct possible
here is the IBS, i try but cannot understand how to create chroot dir...
supposedly this can create pxe server with nfs and dhcp all in one.
Attachments
ibs.zip
(173.5 KiB) Downloaded 542 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#54 Post by Master_wrong »

i have research about the trend of supercomputer using GPU, and here is what i found...

some information...
http://rcc.uga.edu/equipment/subpages/gpu_computing/

driver...
http://developer.nvidia.com/object/cuda ... html#Linux
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
Aitch
Posts: 6518
Joined: Wed 04 Apr 2007, 15:57
Location: Chatham, Kent, UK

#55 Post by Aitch »

How about a Puppy users p2p cluster?

http://gpu.sourceforge.net/index.php

http://sourceforge.net/projects/gpu/

Harness

http://www.csm.ornl.gov/harness/

Condor [previously listed?]

http://www.cs.wisc.edu/condor/

Using Network mapper

http://sourceforge.net/project/screensh ... p_id=58134

High speed compile environment, maybe? Looks fun!

Linux HA [high availability cluster] Users Guide

http://www.linux-ha.org/doc/users-guide ... guide.html

Aitch :)

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#56 Post by Master_wrong »

from
Linux HA [high availability cluster] Users Guide
This utility will copy the configuration files over to any node listed in /etc/ha.d/ha.cf using scp.
scp, that's what i use to set-up my puppy-nodes and server 8)

btw i have successfully test automatic setting of all node and server when using pxe...
it means puppy-cluster will only need 1 cd (and nodes with 1gb ram) that is it.
just boot server, boot all nodes, then after all computer boot up properly, all setting are set automagically, no need to input ip setting at server anymore but i open geany anyway just in case...

so I have accomplish my goal for the project

My goal for the project is
1. make puppy cluster computer ---> Done
2. setting all cluster from server ----> Done
3. automatic setting ----> Done
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
Aitch
Posts: 6518
Joined: Wed 04 Apr 2007, 15:57
Location: Chatham, Kent, UK

#57 Post by Aitch »

You forgot the 'crucial ending line'....

Testers needed! :D

As soon as I can get some time/space I intend to test it - thanks!

Have you got a working software list, or do I need to learn C++? :? :wink:

Aitch :)

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#58 Post by Master_wrong »

:D :D :D
Have you got a working software list, or do I need to learn C++? Confused Wink
indeed, john the ripper is working, netcrack seem working...
btw you do know that mpir.c from svat can be used to run normal program into mpi-cluster environment ?
You forgot the 'crucial ending line'....

Testers needed! Very Happy
sure aitch...

just make sure you create the humongous initrd first to boot pxe...

that means copy this script, make it autoload (also the unfs server if needed), remove the old autoconfig, remaster..., create humongous-initrd from remastered-iso...

then reboot with the new iso, set pxe into normal or debug mode,
then boot all other pc with pxe enabled when server ask
"Press ok if THIS PC IS SERVER..."

after all pc boot properly, click ok, see list of ip in geany, after you close geany, the cluster is ready to run mpich2.

attached is my test pic

Have a nice day

edit... I forgot to add dhcpcd -d
i use seperate file at startup which run dhcpcd -d

#!/bin/sh
tail -n +4 $0 > /tmp/test1.txt
exec rxvt -e sh /tmp/test1.txt

#------------------begin ssh setting
echo "ALL: ALL" > /etc/hosts.allow
echo " " > /etc/hosts.deny
if [ `egrep sshd /etc/passwd|wc -l` -eq 0 ]
then
echo "sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:sbin/nologin" >> /etc/passwd
fi
mkdir -p /var/empty/sshd
/usr/sbin/sshd
#-----------------end ssh setting

#begin check connection to network and show ip
no_network=""
ipp1=`/sbin/ifconfig eth0 | sed '/inet\ /!d;s/.*r://g;s/\ .*//g'`
until [[ $ipp1 != $no_network ]];do ipp1=`/sbin/ifconfig eth0 | sed '/inet\ /!d;s/.*r://g;s/\ .*//g'` ;sleep 3;echo;echo "Not connected to network ?";echo "please connect this computer to network"
done
echo $ipp1
#end

#check hostname master if available and run as server or client
ipp=$ipp1
ip=$ipp1"."
ipp2=`echo ${ip%.*.}`
ip_master=$ipp2".115"

echo "server ip= "$ip_master

m=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $ip_master "ls"`

if [ "$m" = "" ] ; then
echo "setting pc as server"
ifconfig eth0 $ip_master netmask 255.255.255.0 up
ifconfig eth0
#no master node--->be master
hostname puppy-1
rm ~/my-applications/clusterdata/ip-list
rm ~/my-applications/clusterdata/etc/hosts
rm /etc/hosts
echo "$ip_master puppy-1" >>~/my-applications/clusterdata/etc/hosts
echo "$ip_master puppy-1" >>/etc/hosts
echo "$ip_master" >~/my-applications/clusterdata/ip-list
/usr/sbin/netboot-server-jb.sh
echo "this computer is now server"

Xdialog --yesno "Press ok if THIS PC IS SERVER and make sure all pc-node are ready" 0 0 && server="confirm" ;echo $server "SERVER commencing..."
#--------------begin server
if [ "$server" == "confirm" ]; then
geany ~/my-applications/clusterdata/ip-list
xmessage "Please create list of all ip that you have,
put server on top of the list,
press ok after you save the list"
#---begin probing
n=0
rm ~/my-applications/clusterdata/home/mpd.hosts
rm ~/my-applications/clusterdata/etc/hosts

#ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no localhost -v
echo 127.0.0.1 localhost>>~/my-applications/clusterdata/etc/hosts
for i in `cut --delimiter=: -f1 ~/my-applications/clusterdata/ip-list`;do process=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $i "cat /proc/cpuinfo|grep ^processor|wc -l"`;let n="$n+1";echo $n;echo $i:$process>>~/my-applications/clusterdata/home/mpd.hosts;echo $i puppy-$n>>~/my-applications/clusterdata/etc/hosts;ssh $i hostname puppy-$n ;echo "computer:" $i "has:"$process " processor" ;done
#end probing

# begin setting server
cp -prv ~/my-applications/clusterdata/etc/* /etc/
cp -prv ~/my-applications/clusterdata/home/* ~
cp -prv ~/my-applications/clusterdata/home/.mpd.conf ~
# end setting server

#begin setting cluster
# for i in `cut --delimiter=: -f1 ~/mpd.hosts`;do ssh $i mkdir -p .ssh; cat ~/.ssh/id_rsa.pub |ssh $i "cat>>~/.ssh/authorized_keys";done
for i in `cut --delimiter=: -f1 ~/mpd.hosts`;do scp ~/.mpd.conf $i:~;scp ~/mpd.hosts $i:~;scp /etc/mpd.conf $i:/etc;scp /etc/hosts $i:/etc;done
fi
#end setting cluster
###set client nfs
for i in `cut --delimiter=: -f1 ~/my-applications/clusterdata/ip-list`;do process=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $i "mkdir /mnt/try"; ssh $i "mount -o port=4711,mountport=4711,mountvers=3,nfsvers=3,nolock,tcp $ip_master:/root /mnt/try
"`;done
####NFS



else
#master exist---->send ip data to master
echo "server exist"
ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $ip_master "echo $ipp>>~/my-applications/clusterdata/ip-list"

echo "this pc is ready as node"
echo $ipp
hostname
fi

cd ~/my-applications
rox
xmessage " this pc is ready to run cluster "
cd ~
urxvt
Attachments
test-server1.png
(16.74 KiB) Downloaded 1122 times
test-server2.png
(14.95 KiB) Downloaded 439 times
test-node.png
(5.06 KiB) Downloaded 1069 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

gcmartin

Excellent

#59 Post by gcmartin »

Master_wrong wrote:from Linux HA [high availability cluster] Users Guide .... so I have accomplish my goal for the project ... My goal for the project is ... ----> Done
Thanks for clarifying everything you've done thru-out this Thread. I have followed you work religiously. Very clear objectives, approaches, conclusion.

I have one complex request.
  • Would you setup your LiveCd to automatically start the PXE environment when it boots without user intervention.
  • Would you post the LiveCD/LiveDVD that would boot the Cluster Master starting the PXE environment with those files necessary for the cluster nodes to PXE from, please.
This would make an automatic (another word for dummies like me) cluster where all I do is boot "Live" and start nodes, wthout ever typing or clicking on anything, to begin and run my cluster.

You may want to start a new Thread announcing your Multi PC "ClusterPUP".
See my PM to you.

Again, your work here is phenomenal.
Hope this helps

User avatar
Aitch
Posts: 6518
Joined: Wed 04 Apr 2007, 15:57
Location: Chatham, Kent, UK

#60 Post by Aitch »

gcmartin wrote:You may want to start a new Thread announcing your Multi PC "ClusterPUP".
[See my PM to you.]

Again, your work here is phenomenal
2nd that, with big thumbs up :D [even though I haven't implemented it yet]

Anyone doing video/high intensity graphics/blender will benefit from this, however, can compiling be speeded up on a cluster, it is common to hear it taking 10/12 hrs?

Aitch :)

Post Reply