Puppy 245mpi cluster made easy
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
Thank you gcmartin,
your effort in pxe documentation also help this project somehow.
btw here is some article about cluster:
http://www.clustermonkey.net//content/view/89/32/
the site also has ton of info about mpi
http://www.clustermonkey.net//content/view/14/28/
your effort in pxe documentation also help this project somehow.
btw here is some article about cluster:
http://www.clustermonkey.net//content/view/89/32/
the site also has ton of info about mpi
http://www.clustermonkey.net//content/view/14/28/
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
Wow, Master_wrong....you HAVE been busy, and got away from this thread...
http://www.murga-linux.com/puppy/viewtopic.php?t=34647
Since my initial interest, I've had a change in hardware direction, and now don't use much of my old scsi server boxes, as they use too much power, and make too much noise
However, I still think clustering is under-rated, especially in the puppyworld, with so much old hardware available
A couple or 3/more of useful links
http://www.linuxclusters.com/books/clus ... index.html
http://freshmeat.net/articles/linux-clustering-software
http://www.jumpingbean.co.za/linux-clus ... ailability
http://linuxreviews.org/howtos/virtual_ ... ter_howto/
http://www.ultramonkey.org/
http://www.gdargaud.net/Hack/ClusterNotes.html#Software
http://www.cs.wisc.edu/condor/
Good luck guys....some spaghetti examples
http://www.calvin.edu/~adams/research/m ... f/related/
Aitch
http://www.murga-linux.com/puppy/viewtopic.php?t=34647
Since my initial interest, I've had a change in hardware direction, and now don't use much of my old scsi server boxes, as they use too much power, and make too much noise
However, I still think clustering is under-rated, especially in the puppyworld, with so much old hardware available
A couple or 3/more of useful links
http://www.linuxclusters.com/books/clus ... index.html
http://freshmeat.net/articles/linux-clustering-software
http://www.jumpingbean.co.za/linux-clus ... ailability
http://linuxreviews.org/howtos/virtual_ ... ter_howto/
http://www.ultramonkey.org/
http://www.gdargaud.net/Hack/ClusterNotes.html#Software
http://www.cs.wisc.edu/condor/
Good luck guys....some spaghetti examples
http://www.calvin.edu/~adams/research/m ... f/related/
Aitch
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
Hi Aitch, welcome aboard...
yes it quite complicated...Wow, Master_wrong....you HAVE been busy, and got away from this thread...
but it was fun, still one thing is missing... howto incorporate pxe into it, yes i can do it already but how to make it more simple ?
I'm thinking about IBS atm, eh no no not irritable bowel syndrome
here:
http://www.stsx.org/projects/ibs/ibs-manual.html
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
Try iPXE
http://ipxe.org/
I just saw it demo'd at a linux fair in London....it looks cool, but don't ask me how to combine it with what you've done...that's quite an achievement, already
btw, here's a tempting project to build to run it on
http://www.mini-itx.com/projects/cluster/?p
Aitch
http://ipxe.org/
I just saw it demo'd at a linux fair in London....it looks cool, but don't ask me how to combine it with what you've done...that's quite an achievement, already
btw, here's a tempting project to build to run it on
http://www.mini-itx.com/projects/cluster/?p
Aitch
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
@aitch
1. its used to replace existing pxe rom on network card ----> client side
2. chainload into ipxe ----> which mean pxe chainload into ipxe ---->client side
what we need is server side... such as IBS ----> Instant Boot Server
once we have the server surely we can use ipxe... maybe to create cloud puppy supercomputer ?
that would be cool
Btw speaking of pxe....
I just realize that when remastering, when program ask about separate zl245332.... you have to answer "no"...
all you need is correct version of python...
for example you want version 1.1.1 of mpich2, you will need python 2.6 at least...
from above statement we know thatYou can use iPXE to replace the existing PXE ROM on your network card, or you can chainload into iPXE to obtain the features of iPXE without the hassle of reflashing.
1. its used to replace existing pxe rom on network card ----> client side
2. chainload into ipxe ----> which mean pxe chainload into ipxe ---->client side
what we need is server side... such as IBS ----> Instant Boot Server
once we have the server surely we can use ipxe... maybe to create cloud puppy supercomputer ?
that would be cool
Btw speaking of pxe....
I just realize that when remastering, when program ask about separate zl245332.... you have to answer "no"...
thanks aitch, the best part is you can actually install other newer version of mpich and use my script because the setting is the samethat's quite an achievement, already Very Happy
all you need is correct version of python...
for example you want version 1.1.1 of mpich2, you will need python 2.6 at least...
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
Master_wrong
Ah yes, I see
I realize this also plugs into Sunburnt's previous posts, here
http://www.murga-linux.com/puppy/viewtopic.php?t=5797
and
http://www.murga-linux.com/puppy/viewtopic.php?t=9198
which is back in 2006....
Haven't seen Sunburnt around for a while though....but this
Maybe time to think about some old thin client hardware on ebay....?
'....cloud puppy supercomputer' ?
mmm, please
Aitch
Ah yes, I see
I realize this also plugs into Sunburnt's previous posts, here
http://www.murga-linux.com/puppy/viewtopic.php?t=5797
and
http://www.murga-linux.com/puppy/viewtopic.php?t=9198
which is back in 2006....
Haven't seen Sunburnt around for a while though....but this
seems to have finally become a distinct possibleSunburnt wrote:I think a Puppy server - client setup on a CD is what John Murga has in mind.
Keep an eye out for possable [sic] developments.
Maybe time to think about some old thin client hardware on ebay....?
'....cloud puppy supercomputer' ?
mmm, please
Aitch
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
here is the IBS, i try but cannot understand how to create chroot dir...
seems to have finally become a distinct possible
supposedly this can create pxe server with nfs and dhcp all in one.
- Attachments
-
- ibs.zip
- (173.5 KiB) Downloaded 543 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
i have research about the trend of supercomputer using GPU, and here is what i found...
some information...
http://rcc.uga.edu/equipment/subpages/gpu_computing/
driver...
http://developer.nvidia.com/object/cuda ... html#Linux
some information...
http://rcc.uga.edu/equipment/subpages/gpu_computing/
driver...
http://developer.nvidia.com/object/cuda ... html#Linux
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
How about a Puppy users p2p cluster?
http://gpu.sourceforge.net/index.php
http://sourceforge.net/projects/gpu/
Harness
http://www.csm.ornl.gov/harness/
Condor [previously listed?]
http://www.cs.wisc.edu/condor/
Using Network mapper
http://sourceforge.net/project/screensh ... p_id=58134
High speed compile environment, maybe? Looks fun!
Linux HA [high availability cluster] Users Guide
http://www.linux-ha.org/doc/users-guide ... guide.html
Aitch
http://gpu.sourceforge.net/index.php
http://sourceforge.net/projects/gpu/
Harness
http://www.csm.ornl.gov/harness/
Condor [previously listed?]
http://www.cs.wisc.edu/condor/
Using Network mapper
http://sourceforge.net/project/screensh ... p_id=58134
High speed compile environment, maybe? Looks fun!
Linux HA [high availability cluster] Users Guide
http://www.linux-ha.org/doc/users-guide ... guide.html
Aitch
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
from
Linux HA [high availability cluster] Users Guide
btw i have successfully test automatic setting of all node and server when using pxe...
it means puppy-cluster will only need 1 cd (and nodes with 1gb ram) that is it.
just boot server, boot all nodes, then after all computer boot up properly, all setting are set automagically, no need to input ip setting at server anymore but i open geany anyway just in case...
so I have accomplish my goal for the project
My goal for the project is
1. make puppy cluster computer ---> Done
2. setting all cluster from server ----> Done
3. automatic setting ----> Done
Linux HA [high availability cluster] Users Guide
scp, that's what i use to set-up my puppy-nodes and serverThis utility will copy the configuration files over to any node listed in /etc/ha.d/ha.cf using scp.
btw i have successfully test automatic setting of all node and server when using pxe...
it means puppy-cluster will only need 1 cd (and nodes with 1gb ram) that is it.
just boot server, boot all nodes, then after all computer boot up properly, all setting are set automagically, no need to input ip setting at server anymore but i open geany anyway just in case...
so I have accomplish my goal for the project
My goal for the project is
1. make puppy cluster computer ---> Done
2. setting all cluster from server ----> Done
3. automatic setting ----> Done
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
indeed, john the ripper is working, netcrack seem working...Have you got a working software list, or do I need to learn C++? Confused Wink
btw you do know that mpir.c from svat can be used to run normal program into mpi-cluster environment ?
sure aitch...You forgot the 'crucial ending line'....
Testers needed! Very Happy
just make sure you create the humongous initrd first to boot pxe...
that means copy this script, make it autoload (also the unfs server if needed), remove the old autoconfig, remaster..., create humongous-initrd from remastered-iso...
then reboot with the new iso, set pxe into normal or debug mode,
then boot all other pc with pxe enabled when server ask
"Press ok if THIS PC IS SERVER..."
after all pc boot properly, click ok, see list of ip in geany, after you close geany, the cluster is ready to run mpich2.
attached is my test pic
Have a nice day
edit... I forgot to add dhcpcd -d
i use seperate file at startup which run dhcpcd -d
#!/bin/sh
tail -n +4 $0 > /tmp/test1.txt
exec rxvt -e sh /tmp/test1.txt
#------------------begin ssh setting
echo "ALL: ALL" > /etc/hosts.allow
echo " " > /etc/hosts.deny
if [ `egrep sshd /etc/passwd|wc -l` -eq 0 ]
then
echo "sshd74:74:Privilege-separated SSH:/var/empty/sshd:sbin/nologin" >> /etc/passwd
fi
mkdir -p /var/empty/sshd
/usr/sbin/sshd
#-----------------end ssh setting
#begin check connection to network and show ip
no_network=""
ipp1=`/sbin/ifconfig eth0 | sed '/inet\ /!d;s/.*r://g;s/\ .*//g'`
until [[ $ipp1 != $no_network ]];do ipp1=`/sbin/ifconfig eth0 | sed '/inet\ /!d;s/.*r://g;s/\ .*//g'` ;sleep 3;echo;echo "Not connected to network ?";echo "please connect this computer to network"
done
echo $ipp1
#end
#check hostname master if available and run as server or client
ipp=$ipp1
ip=$ipp1"."
ipp2=`echo ${ip%.*.}`
ip_master=$ipp2".115"
echo "server ip= "$ip_master
m=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $ip_master "ls"`
if [ "$m" = "" ] ; then
echo "setting pc as server"
ifconfig eth0 $ip_master netmask 255.255.255.0 up
ifconfig eth0
#no master node--->be master
hostname puppy-1
rm ~/my-applications/clusterdata/ip-list
rm ~/my-applications/clusterdata/etc/hosts
rm /etc/hosts
echo "$ip_master puppy-1" >>~/my-applications/clusterdata/etc/hosts
echo "$ip_master puppy-1" >>/etc/hosts
echo "$ip_master" >~/my-applications/clusterdata/ip-list
/usr/sbin/netboot-server-jb.sh
echo "this computer is now server"
Xdialog --yesno "Press ok if THIS PC IS SERVER and make sure all pc-node are ready" 0 0 && server="confirm" ;echo $server "SERVER commencing..."
#--------------begin server
if [ "$server" == "confirm" ]; then
geany ~/my-applications/clusterdata/ip-list
xmessage "Please create list of all ip that you have,
put server on top of the list,
press ok after you save the list"
#---begin probing
n=0
rm ~/my-applications/clusterdata/home/mpd.hosts
rm ~/my-applications/clusterdata/etc/hosts
#ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no localhost -v
echo 127.0.0.1 localhost>>~/my-applications/clusterdata/etc/hosts
for i in `cut --delimiter=: -f1 ~/my-applications/clusterdata/ip-list`;do process=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $i "cat /proc/cpuinfo|grep ^processor|wc -l"`;let n="$n+1";echo $n;echo $i:$process>>~/my-applications/clusterdata/home/mpd.hosts;echo $i puppy-$n>>~/my-applications/clusterdata/etc/hosts;ssh $i hostname puppy-$n ;echo "computer:" $i "has:"$process " processor" ;done
#end probing
# begin setting server
cp -prv ~/my-applications/clusterdata/etc/* /etc/
cp -prv ~/my-applications/clusterdata/home/* ~
cp -prv ~/my-applications/clusterdata/home/.mpd.conf ~
# end setting server
#begin setting cluster
# for i in `cut --delimiter=: -f1 ~/mpd.hosts`;do ssh $i mkdir -p .ssh; cat ~/.ssh/id_rsa.pub |ssh $i "cat>>~/.ssh/authorized_keys";done
for i in `cut --delimiter=: -f1 ~/mpd.hosts`;do scp ~/.mpd.conf $i:~;scp ~/mpd.hosts $i:~;scp /etc/mpd.conf $i:/etc;scp /etc/hosts $i:/etc;done
fi
#end setting cluster
###set client nfs
for i in `cut --delimiter=: -f1 ~/my-applications/clusterdata/ip-list`;do process=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $i "mkdir /mnt/try"; ssh $i "mount -o port=4711,mountport=4711,mountvers=3,nfsvers=3,nolock,tcp $ip_master:/root /mnt/try
"`;done
####NFS
else
#master exist---->send ip data to master
echo "server exist"
ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $ip_master "echo $ipp>>~/my-applications/clusterdata/ip-list"
echo "this pc is ready as node"
echo $ipp
hostname
fi
cd ~/my-applications
rox
xmessage " this pc is ready to run cluster "
cd ~
urxvt
- Attachments
-
- test-server1.png
- (16.74 KiB) Downloaded 1122 times
-
- test-server2.png
- (14.95 KiB) Downloaded 439 times
-
- test-node.png
- (5.06 KiB) Downloaded 1069 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
Excellent
Thanks for clarifying everything you've done thru-out this Thread. I have followed you work religiously. Very clear objectives, approaches, conclusion.Master_wrong wrote:from Linux HA [high availability cluster] Users Guide .... so I have accomplish my goal for the project ... My goal for the project is ... ----> Done
I have one complex request.
- Would you setup your LiveCd to automatically start the PXE environment when it boots without user intervention.
- Would you post the LiveCD/LiveDVD that would boot the Cluster Master starting the PXE environment with those files necessary for the cluster nodes to PXE from, please.
You may want to start a new Thread announcing your Multi PC "ClusterPUP".
See my PM to you.
Again, your work here is phenomenal.
Hope this helps
2nd that, with big thumbs up [even though I haven't implemented it yet]gcmartin wrote:You may want to start a new Thread announcing your Multi PC "ClusterPUP".
[See my PM to you.]
Again, your work here is phenomenal
Anyone doing video/high intensity graphics/blender will benefit from this, however, can compiling be speeded up on a cluster, it is common to hear it taking 10/12 hrs?
Aitch
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
@gcmartin
what's important is that now it can boot headless pc. but you still need to click
@Aitch Smile
it seems good
this is called renderfarm i guess
Anyone willing to test ? [/quote]
well i can put netboot folder into root which will be saved during remaster, but the iso size will be enormously 600+MThanks for clarifying everything you've done thru-out this Thread. I have followed you work religiously. Very clear objectives, approaches, conclusion.
I have one complex request.
* Would you setup your LiveCd to automatically start the PXE environment when it boots without user intervention.
* Would you post the LiveCD/LiveDVD that would boot the Cluster Master starting the PXE environment with those files necessary for the cluster nodes to PXE from, please.
This would make an automatic (another word for dummies like me) cluster where all I do is boot "Live" and start nodes, wthout ever typing or clicking on anything, to begin and run my cluster.
You may want to start a new Thread announcing your Multi PC "ClusterPUP".
See my PM to you.
what's important is that now it can boot headless pc. but you still need to click
@Aitch Smile
here i test mpir...Anyone doing video/high intensity graphics/blender will benefit from this, however, can compiling be speeded up on a cluster, it is common to hear it taking 10/12 hrs?
it seems good
off course i didnt install blender and i dont know howto run dvdauthor...# mpirun -np 2 ~/mpir -s 101 -e 1000 'blender -b in.blend -s %start -e %end -a'
sh: blender: command not found
sh: blender: command not found
^C
# mpirun -np 2 ~/mpir -s 101 -e 1000 'dvdauthor -b in.blend -s %start -e %end -a'
DVDAuthor::dvdauthor, version 0.6.12-alpha.
Build options: gnugetopt magick iconv freetype
Send bugs to <dvdauthor-users@lists.sourceforge.net>
ERR: Must first specify -t, -m, or -x.
DVDAuthor::dvdauthor, version 0.6.12-alpha.
Build options: gnugetopt magick iconv freetype
Send bugs to <dvdauthor-users@lists.sourceforge.net>
^C
this is called renderfarm i guess
Anyone willing to test ? [/quote]
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
testing mpich2-132
I have test Mpich2-132 on 5.2.
.
i have install required fortran compile and install it also run few test.
it seem new mpich didnt use mpdboot anymore, but hydra and what it need is just list of ip...
so i might have to create new script again.
edit... false alarm i didn configure it properly
.
i have install required fortran compile and install it also run few test.
it seem new mpich didnt use mpdboot anymore, but hydra and what it need is just list of ip...
so i might have to create new script again.
edit... false alarm i didn configure it properly
- Attachments
-
- mpich2-132.jpg
- (20.45 KiB) Downloaded 1570 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
remote FS use in Cluster
Don't know if you are aware, but, SAMBA has been used om Cllusters for awhile. You may find this use a lot in cluster community. See here for starters. then you should join a SAMBA forum where a large community of user are available that will help.
See SAMBA cluster reference here and the Cluster support covers here.
Hope this helps
See SAMBA cluster reference here and the Cluster support covers here.
Hope this helps
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
Thank gcmartin, it's more like database solution for cluster and i bet the DB stands for Database...
but sure very useful for some cluster application.Requirements
CTDB relies on a clustered filesystem being available and shared on all nodes that participate in the CTDB cluster. This filesystem must be mounted and available on all nodes in the CTDB cluster.
On top of this cluster filesystem, CTDB then provides clustered HA features so that data from the clustered filesystem can be exported through multiple nodes in the CTDB cluster using various services. Currently included with CTDB are the necessary hooks for Samba, NFS and ftp exports. Support for new service types can easily be added.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
CTDB and ClusterPup
If you are familiar with sharing files from one Windows PC with the other PCs on the LAN, then this allows ClusterPUP to become a PC that can also share on the LAN, same as all Windows PCs.
Thus, ClusterPUP can not only "mount" Windows share resources, but with CTDB, it can make available folders and files for LAN PCs to read and write to for the cluster.. Now, as a simple example, I can load data, for cluster management/applications, directly from somewhere else on my LAN, AND, I can run applications within my cluster where the resulting data can be viewed, used, manipulated, or managed by things on my LAN. All this done with only lifting a finger!
Think of it as a "secure" data sharing gateway for your cluster.See this and PDF reference at bottom
and this wiki
Hope this helps
Thus, ClusterPUP can not only "mount" Windows share resources, but with CTDB, it can make available folders and files for LAN PCs to read and write to for the cluster.. Now, as a simple example, I can load data, for cluster management/applications, directly from somewhere else on my LAN, AND, I can run applications within my cluster where the resulting data can be viewed, used, manipulated, or managed by things on my LAN. All this done with only lifting a finger!
Think of it as a "secure" data sharing gateway for your cluster.See this and PDF reference at bottom
and this wiki
Hope this helps