Puppy 245mpi cluster made easy
-
- Posts: 17
- Joined: Tue 18 Jan 2011, 02:07
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
i spend last night testing like couple dozens of program....
it end up only couple of program that compiled successfully
here is 2 interesting program that might be useful,
1. create /etc/exports file: contain=====> /root (ro) --->for read only
open terminal at unfsd directrory
2. ./configure
3. make
4. make install
5. ./unfsd -s -d -p -n 4711 -m 4711 -e /etc/exports
6. at client : mkdir /mnt/try
7. copy unfsd,./configure,make,make install
8. mount -o port=4711,mountport=4711,mountvers=3,nfsvers=3,nolock,tcp 192.168.2.130:/root /mnt/try
it end up only couple of program that compiled successfully
here is 2 interesting program that might be useful,
too bad i cannot get anything to work, i simply not smart enough to create config file, if anyone capable please let me know1. tentakel :
Tentakel is a program for executing the same command on many hosts in parallel
using various remote methods. It can make use of several sets of hosts that are
defined in a configuration file as groups.
It also supports an interactive mode that can be used for repeated commands.
The author uses tentakel to simultaneously install software on many
workstations or ask and set parameters on a linux compute cluster. With the
power of format strings tentakel can also be used for monitoring purposes.
Supported remote methods are ssh(1) and rsh(1). Both must be configured to
allow for password-less logins. Password-protected keyfiles for ssh can be
used with ssh-agent(1).
A plugin mechanism allows users to implement their own remote methods in
addition to the builtin ones.
For more information on available options please refer to the manpage
tentakel(1).
The project homepage is: http://tentakel.biskalar.de/
all you need to do
2. unfs3-0.9.22.tar.gz :Great NFS for puppy, it capable of export and mount /root
INTRODUCTION
============
UNFS3 is a user-space implementation of the NFSv3 server
specification.
UNFS3 supports all NFSv3 procedures with the exception of the
READDIRPLUS procedure. It tries to provide as much information
to NFS clients as possible, within the limits possible from
user-space.
See the unfsd(8) manpage for restrictions imposed on NFS
operations (section RESTRICTIONS) and for possible races
with local file system activity (section BUGS).
It is not possible to export to netgroups or wildcard hostnames
via /etc/exports, all other addressing methods should work. The
following options are recognized in the exports file: ro, rw,
root_squash, all_squash, no_root_squash, no_all_squash. If
other options are present, they are ignored.
Cluster extensions compatible to the older ClusterNFS project
are supported when the source is configured with --enable-cluster.
1. create /etc/exports file: contain=====> /root (ro) --->for read only
open terminal at unfsd directrory
2. ./configure
3. make
4. make install
5. ./unfsd -s -d -p -n 4711 -m 4711 -e /etc/exports
6. at client : mkdir /mnt/try
7. copy unfsd,./configure,make,make install
8. mount -o port=4711,mountport=4711,mountvers=3,nfsvers=3,nolock,tcp 192.168.2.130:/root /mnt/try
- Attachments
-
- unfs3-test.png
- (25.94 KiB) Downloaded 1110 times
-
- tentakel.zip
- (42.15 KiB) Downloaded 449 times
-
- unfs3-0.9.22.tar.gz
- (164.5 KiB) Downloaded 473 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
made pet of unfs3 at http://www.murga-linux.com/puppy/viewtopic.php?t=64491
btw pxe booting work using
http://www.murga-linux.com/puppy/viewtopic.php?t=64107
also other program tested
btw pxe booting work using
http://www.murga-linux.com/puppy/viewtopic.php?t=64107
also other program tested
pconsole does not work through daemons. The pconsole.sh script opens up a
number of connections to each node of your cluster of workstations. Then,
the pconsole binary attaches to the tty devices that these windows are
using, and it copies the input you type to all open connections......
.......
By default, pconsole tries to use SSH to make connections. If you do not have
'ssh', it will try to use 'telnet'.
- Attachments
-
- pconsole-test.png
- (32.67 KiB) Downloaded 488 times
Last edited by Master_wrong on Sun 06 Feb 2011, 06:12, edited 2 times in total.
What is a cluster
@Master _Wrong has done all of us a very valuable service. I am showing this picture here because over the past couple months, many who come here have an inaccurate perception of a cluster.
This picture is worth a thousand words, as, it shows where to place the monitor, keyboard, and mouse for the cluster. In a true cluster, the nodes/workers/slave PCs DO NOT HAVE A MONITOR, KEYBOARD, OR MOUSE. But they all work together on behalf of the workload within the cluster.
Hope this helps
This picture is worth a thousand words, as, it shows where to place the monitor, keyboard, and mouse for the cluster. In a true cluster, the nodes/workers/slave PCs DO NOT HAVE A MONITOR, KEYBOARD, OR MOUSE. But they all work together on behalf of the workload within the cluster.
Hope this helps
- Attachments
-
- Beowulf.png
- (86.79 KiB) Downloaded 521 times
Last edited by gcmartin on Sat 05 Feb 2011, 18:28, edited 1 time in total.
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
Thank you gcmartin,
your effort in pxe documentation also help this project somehow.
btw here is some article about cluster:
http://www.clustermonkey.net//content/view/89/32/
the site also has ton of info about mpi
http://www.clustermonkey.net//content/view/14/28/
your effort in pxe documentation also help this project somehow.
btw here is some article about cluster:
http://www.clustermonkey.net//content/view/89/32/
the site also has ton of info about mpi
http://www.clustermonkey.net//content/view/14/28/
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
Wow, Master_wrong....you HAVE been busy, and got away from this thread...
http://www.murga-linux.com/puppy/viewtopic.php?t=34647
Since my initial interest, I've had a change in hardware direction, and now don't use much of my old scsi server boxes, as they use too much power, and make too much noise
However, I still think clustering is under-rated, especially in the puppyworld, with so much old hardware available
A couple or 3/more of useful links
http://www.linuxclusters.com/books/clus ... index.html
http://freshmeat.net/articles/linux-clustering-software
http://www.jumpingbean.co.za/linux-clus ... ailability
http://linuxreviews.org/howtos/virtual_ ... ter_howto/
http://www.ultramonkey.org/
http://www.gdargaud.net/Hack/ClusterNotes.html#Software
http://www.cs.wisc.edu/condor/
Good luck guys....some spaghetti examples
http://www.calvin.edu/~adams/research/m ... f/related/
Aitch
http://www.murga-linux.com/puppy/viewtopic.php?t=34647
Since my initial interest, I've had a change in hardware direction, and now don't use much of my old scsi server boxes, as they use too much power, and make too much noise
However, I still think clustering is under-rated, especially in the puppyworld, with so much old hardware available
A couple or 3/more of useful links
http://www.linuxclusters.com/books/clus ... index.html
http://freshmeat.net/articles/linux-clustering-software
http://www.jumpingbean.co.za/linux-clus ... ailability
http://linuxreviews.org/howtos/virtual_ ... ter_howto/
http://www.ultramonkey.org/
http://www.gdargaud.net/Hack/ClusterNotes.html#Software
http://www.cs.wisc.edu/condor/
Good luck guys....some spaghetti examples
http://www.calvin.edu/~adams/research/m ... f/related/
Aitch
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
Hi Aitch, welcome aboard...
yes it quite complicated...Wow, Master_wrong....you HAVE been busy, and got away from this thread...
but it was fun, still one thing is missing... howto incorporate pxe into it, yes i can do it already but how to make it more simple ?
I'm thinking about IBS atm, eh no no not irritable bowel syndrome
here:
http://www.stsx.org/projects/ibs/ibs-manual.html
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
Try iPXE
http://ipxe.org/
I just saw it demo'd at a linux fair in London....it looks cool, but don't ask me how to combine it with what you've done...that's quite an achievement, already
btw, here's a tempting project to build to run it on
http://www.mini-itx.com/projects/cluster/?p
Aitch
http://ipxe.org/
I just saw it demo'd at a linux fair in London....it looks cool, but don't ask me how to combine it with what you've done...that's quite an achievement, already
btw, here's a tempting project to build to run it on
http://www.mini-itx.com/projects/cluster/?p
Aitch
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
@aitch
1. its used to replace existing pxe rom on network card ----> client side
2. chainload into ipxe ----> which mean pxe chainload into ipxe ---->client side
what we need is server side... such as IBS ----> Instant Boot Server
once we have the server surely we can use ipxe... maybe to create cloud puppy supercomputer ?
that would be cool
Btw speaking of pxe....
I just realize that when remastering, when program ask about separate zl245332.... you have to answer "no"...
all you need is correct version of python...
for example you want version 1.1.1 of mpich2, you will need python 2.6 at least...
from above statement we know thatYou can use iPXE to replace the existing PXE ROM on your network card, or you can chainload into iPXE to obtain the features of iPXE without the hassle of reflashing.
1. its used to replace existing pxe rom on network card ----> client side
2. chainload into ipxe ----> which mean pxe chainload into ipxe ---->client side
what we need is server side... such as IBS ----> Instant Boot Server
once we have the server surely we can use ipxe... maybe to create cloud puppy supercomputer ?
that would be cool
Btw speaking of pxe....
I just realize that when remastering, when program ask about separate zl245332.... you have to answer "no"...
thanks aitch, the best part is you can actually install other newer version of mpich and use my script because the setting is the samethat's quite an achievement, already Very Happy
all you need is correct version of python...
for example you want version 1.1.1 of mpich2, you will need python 2.6 at least...
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
Master_wrong
Ah yes, I see
I realize this also plugs into Sunburnt's previous posts, here
http://www.murga-linux.com/puppy/viewtopic.php?t=5797
and
http://www.murga-linux.com/puppy/viewtopic.php?t=9198
which is back in 2006....
Haven't seen Sunburnt around for a while though....but this
Maybe time to think about some old thin client hardware on ebay....?
'....cloud puppy supercomputer' ?
mmm, please
Aitch
Ah yes, I see
I realize this also plugs into Sunburnt's previous posts, here
http://www.murga-linux.com/puppy/viewtopic.php?t=5797
and
http://www.murga-linux.com/puppy/viewtopic.php?t=9198
which is back in 2006....
Haven't seen Sunburnt around for a while though....but this
seems to have finally become a distinct possibleSunburnt wrote:I think a Puppy server - client setup on a CD is what John Murga has in mind.
Keep an eye out for possable [sic] developments.
Maybe time to think about some old thin client hardware on ebay....?
'....cloud puppy supercomputer' ?
mmm, please
Aitch
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
here is the IBS, i try but cannot understand how to create chroot dir...
seems to have finally become a distinct possible
supposedly this can create pxe server with nfs and dhcp all in one.
- Attachments
-
- ibs.zip
- (173.5 KiB) Downloaded 542 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
i have research about the trend of supercomputer using GPU, and here is what i found...
some information...
http://rcc.uga.edu/equipment/subpages/gpu_computing/
driver...
http://developer.nvidia.com/object/cuda ... html#Linux
some information...
http://rcc.uga.edu/equipment/subpages/gpu_computing/
driver...
http://developer.nvidia.com/object/cuda ... html#Linux
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
How about a Puppy users p2p cluster?
http://gpu.sourceforge.net/index.php
http://sourceforge.net/projects/gpu/
Harness
http://www.csm.ornl.gov/harness/
Condor [previously listed?]
http://www.cs.wisc.edu/condor/
Using Network mapper
http://sourceforge.net/project/screensh ... p_id=58134
High speed compile environment, maybe? Looks fun!
Linux HA [high availability cluster] Users Guide
http://www.linux-ha.org/doc/users-guide ... guide.html
Aitch
http://gpu.sourceforge.net/index.php
http://sourceforge.net/projects/gpu/
Harness
http://www.csm.ornl.gov/harness/
Condor [previously listed?]
http://www.cs.wisc.edu/condor/
Using Network mapper
http://sourceforge.net/project/screensh ... p_id=58134
High speed compile environment, maybe? Looks fun!
Linux HA [high availability cluster] Users Guide
http://www.linux-ha.org/doc/users-guide ... guide.html
Aitch
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
from
Linux HA [high availability cluster] Users Guide
btw i have successfully test automatic setting of all node and server when using pxe...
it means puppy-cluster will only need 1 cd (and nodes with 1gb ram) that is it.
just boot server, boot all nodes, then after all computer boot up properly, all setting are set automagically, no need to input ip setting at server anymore but i open geany anyway just in case...
so I have accomplish my goal for the project
My goal for the project is
1. make puppy cluster computer ---> Done
2. setting all cluster from server ----> Done
3. automatic setting ----> Done
Linux HA [high availability cluster] Users Guide
scp, that's what i use to set-up my puppy-nodes and serverThis utility will copy the configuration files over to any node listed in /etc/ha.d/ha.cf using scp.
btw i have successfully test automatic setting of all node and server when using pxe...
it means puppy-cluster will only need 1 cd (and nodes with 1gb ram) that is it.
just boot server, boot all nodes, then after all computer boot up properly, all setting are set automagically, no need to input ip setting at server anymore but i open geany anyway just in case...
so I have accomplish my goal for the project
My goal for the project is
1. make puppy cluster computer ---> Done
2. setting all cluster from server ----> Done
3. automatic setting ----> Done
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
indeed, john the ripper is working, netcrack seem working...Have you got a working software list, or do I need to learn C++? Confused Wink
btw you do know that mpir.c from svat can be used to run normal program into mpi-cluster environment ?
sure aitch...You forgot the 'crucial ending line'....
Testers needed! Very Happy
just make sure you create the humongous initrd first to boot pxe...
that means copy this script, make it autoload (also the unfs server if needed), remove the old autoconfig, remaster..., create humongous-initrd from remastered-iso...
then reboot with the new iso, set pxe into normal or debug mode,
then boot all other pc with pxe enabled when server ask
"Press ok if THIS PC IS SERVER..."
after all pc boot properly, click ok, see list of ip in geany, after you close geany, the cluster is ready to run mpich2.
attached is my test pic
Have a nice day
edit... I forgot to add dhcpcd -d
i use seperate file at startup which run dhcpcd -d
#!/bin/sh
tail -n +4 $0 > /tmp/test1.txt
exec rxvt -e sh /tmp/test1.txt
#------------------begin ssh setting
echo "ALL: ALL" > /etc/hosts.allow
echo " " > /etc/hosts.deny
if [ `egrep sshd /etc/passwd|wc -l` -eq 0 ]
then
echo "sshd74:74:Privilege-separated SSH:/var/empty/sshd:sbin/nologin" >> /etc/passwd
fi
mkdir -p /var/empty/sshd
/usr/sbin/sshd
#-----------------end ssh setting
#begin check connection to network and show ip
no_network=""
ipp1=`/sbin/ifconfig eth0 | sed '/inet\ /!d;s/.*r://g;s/\ .*//g'`
until [[ $ipp1 != $no_network ]];do ipp1=`/sbin/ifconfig eth0 | sed '/inet\ /!d;s/.*r://g;s/\ .*//g'` ;sleep 3;echo;echo "Not connected to network ?";echo "please connect this computer to network"
done
echo $ipp1
#end
#check hostname master if available and run as server or client
ipp=$ipp1
ip=$ipp1"."
ipp2=`echo ${ip%.*.}`
ip_master=$ipp2".115"
echo "server ip= "$ip_master
m=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $ip_master "ls"`
if [ "$m" = "" ] ; then
echo "setting pc as server"
ifconfig eth0 $ip_master netmask 255.255.255.0 up
ifconfig eth0
#no master node--->be master
hostname puppy-1
rm ~/my-applications/clusterdata/ip-list
rm ~/my-applications/clusterdata/etc/hosts
rm /etc/hosts
echo "$ip_master puppy-1" >>~/my-applications/clusterdata/etc/hosts
echo "$ip_master puppy-1" >>/etc/hosts
echo "$ip_master" >~/my-applications/clusterdata/ip-list
/usr/sbin/netboot-server-jb.sh
echo "this computer is now server"
Xdialog --yesno "Press ok if THIS PC IS SERVER and make sure all pc-node are ready" 0 0 && server="confirm" ;echo $server "SERVER commencing..."
#--------------begin server
if [ "$server" == "confirm" ]; then
geany ~/my-applications/clusterdata/ip-list
xmessage "Please create list of all ip that you have,
put server on top of the list,
press ok after you save the list"
#---begin probing
n=0
rm ~/my-applications/clusterdata/home/mpd.hosts
rm ~/my-applications/clusterdata/etc/hosts
#ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no localhost -v
echo 127.0.0.1 localhost>>~/my-applications/clusterdata/etc/hosts
for i in `cut --delimiter=: -f1 ~/my-applications/clusterdata/ip-list`;do process=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $i "cat /proc/cpuinfo|grep ^processor|wc -l"`;let n="$n+1";echo $n;echo $i:$process>>~/my-applications/clusterdata/home/mpd.hosts;echo $i puppy-$n>>~/my-applications/clusterdata/etc/hosts;ssh $i hostname puppy-$n ;echo "computer:" $i "has:"$process " processor" ;done
#end probing
# begin setting server
cp -prv ~/my-applications/clusterdata/etc/* /etc/
cp -prv ~/my-applications/clusterdata/home/* ~
cp -prv ~/my-applications/clusterdata/home/.mpd.conf ~
# end setting server
#begin setting cluster
# for i in `cut --delimiter=: -f1 ~/mpd.hosts`;do ssh $i mkdir -p .ssh; cat ~/.ssh/id_rsa.pub |ssh $i "cat>>~/.ssh/authorized_keys";done
for i in `cut --delimiter=: -f1 ~/mpd.hosts`;do scp ~/.mpd.conf $i:~;scp ~/mpd.hosts $i:~;scp /etc/mpd.conf $i:/etc;scp /etc/hosts $i:/etc;done
fi
#end setting cluster
###set client nfs
for i in `cut --delimiter=: -f1 ~/my-applications/clusterdata/ip-list`;do process=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $i "mkdir /mnt/try"; ssh $i "mount -o port=4711,mountport=4711,mountvers=3,nfsvers=3,nolock,tcp $ip_master:/root /mnt/try
"`;done
####NFS
else
#master exist---->send ip data to master
echo "server exist"
ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $ip_master "echo $ipp>>~/my-applications/clusterdata/ip-list"
echo "this pc is ready as node"
echo $ipp
hostname
fi
cd ~/my-applications
rox
xmessage " this pc is ready to run cluster "
cd ~
urxvt
- Attachments
-
- test-server1.png
- (16.74 KiB) Downloaded 1122 times
-
- test-server2.png
- (14.95 KiB) Downloaded 439 times
-
- test-node.png
- (5.06 KiB) Downloaded 1069 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
Excellent
Thanks for clarifying everything you've done thru-out this Thread. I have followed you work religiously. Very clear objectives, approaches, conclusion.Master_wrong wrote:from Linux HA [high availability cluster] Users Guide .... so I have accomplish my goal for the project ... My goal for the project is ... ----> Done
I have one complex request.
- Would you setup your LiveCd to automatically start the PXE environment when it boots without user intervention.
- Would you post the LiveCD/LiveDVD that would boot the Cluster Master starting the PXE environment with those files necessary for the cluster nodes to PXE from, please.
You may want to start a new Thread announcing your Multi PC "ClusterPUP".
See my PM to you.
Again, your work here is phenomenal.
Hope this helps
2nd that, with big thumbs up [even though I haven't implemented it yet]gcmartin wrote:You may want to start a new Thread announcing your Multi PC "ClusterPUP".
[See my PM to you.]
Again, your work here is phenomenal
Anyone doing video/high intensity graphics/blender will benefit from this, however, can compiling be speeded up on a cluster, it is common to hear it taking 10/12 hrs?
Aitch