Puppy 245mpi cluster made easy

A home for all kinds of Puppy related projects
Message
Author
gcmartin

#46 Post by gcmartin »

.

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#47 Post by Master_wrong »

Thank you gcmartin,

your effort in pxe documentation also help this project somehow.

btw here is some article about cluster:
http://www.clustermonkey.net//content/view/89/32/

the site also has ton of info about mpi
http://www.clustermonkey.net//content/view/14/28/
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
Aitch
Posts: 6518
Joined: Wed 04 Apr 2007, 15:57
Location: Chatham, Kent, UK

#48 Post by Aitch »

Wow, Master_wrong....you HAVE been busy, and got away from this thread...

http://www.murga-linux.com/puppy/viewtopic.php?t=34647

Since my initial interest, I've had a change in hardware direction, and now don't use much of my old scsi server boxes, as they use too much power, and make too much noise :wink:

However, I still think clustering is under-rated, especially in the puppyworld, with so much old hardware available

A couple or 3/more of useful links

http://www.linuxclusters.com/books/clus ... index.html

http://freshmeat.net/articles/linux-clustering-software

http://www.jumpingbean.co.za/linux-clus ... ailability

http://linuxreviews.org/howtos/virtual_ ... ter_howto/

http://www.ultramonkey.org/

http://www.gdargaud.net/Hack/ClusterNotes.html#Software

http://www.cs.wisc.edu/condor/

Good luck guys....some spaghetti examples

http://www.calvin.edu/~adams/research/m ... f/related/

Aitch :)

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#49 Post by Master_wrong »

:D
Hi Aitch, welcome aboard...
Wow, Master_wrong....you HAVE been busy, and got away from this thread...
yes it quite complicated...
but it was fun, still one thing is missing... howto incorporate pxe into it, yes i can do it already but how to make it more simple ?

I'm thinking about IBS atm, eh no no not irritable bowel syndrome :lol:

here:
http://www.stsx.org/projects/ibs/ibs-manual.html
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
Aitch
Posts: 6518
Joined: Wed 04 Apr 2007, 15:57
Location: Chatham, Kent, UK

#50 Post by Aitch »

Try iPXE

http://ipxe.org/

I just saw it demo'd at a linux fair in London....it looks cool, but don't ask me how to combine it with what you've done...that's quite an achievement, already :D

btw, here's a tempting project to build to run it on

http://www.mini-itx.com/projects/cluster/?p

Aitch :)

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#51 Post by Master_wrong »

@aitch
You can use iPXE to replace the existing PXE ROM on your network card, or you can chainload into iPXE to obtain the features of iPXE without the hassle of reflashing.
from above statement we know that
1. its used to replace existing pxe rom on network card ----> client side
2. chainload into ipxe ----> which mean pxe chainload into ipxe ---->client side

what we need is server side... such as IBS ----> Instant Boot Server

once we have the server surely we can use ipxe... maybe to create cloud puppy supercomputer ?
that would be cool 8)

Btw speaking of pxe....

I just realize that when remastering, when program ask about separate zl245332.... you have to answer "no"...
that's quite an achievement, already Very Happy
thanks aitch, the best part is you can actually install other newer version of mpich and use my script because the setting is the same :idea:
all you need is correct version of python...

for example you want version 1.1.1 of mpich2, you will need python 2.6 at least...
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
Aitch
Posts: 6518
Joined: Wed 04 Apr 2007, 15:57
Location: Chatham, Kent, UK

#52 Post by Aitch »

Master_wrong

Ah yes, I see

I realize this also plugs into Sunburnt's previous posts, here

http://www.murga-linux.com/puppy/viewtopic.php?t=5797

and

http://www.murga-linux.com/puppy/viewtopic.php?t=9198

which is back in 2006....

Haven't seen Sunburnt around for a while though....but this
Sunburnt wrote:I think a Puppy server - client setup on a CD is what John Murga has in mind.
Keep an eye out for possable [sic] developments.
seems to have finally become a distinct possible

Maybe time to think about some old thin client hardware on ebay....?

'....cloud puppy supercomputer' ?

mmm, please :lol:

Aitch :)

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#53 Post by Master_wrong »


seems to have finally become a distinct possible
here is the IBS, i try but cannot understand how to create chroot dir...
supposedly this can create pxe server with nfs and dhcp all in one.
Attachments
ibs.zip
(173.5 KiB) Downloaded 542 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#54 Post by Master_wrong »

i have research about the trend of supercomputer using GPU, and here is what i found...

some information...
http://rcc.uga.edu/equipment/subpages/gpu_computing/

driver...
http://developer.nvidia.com/object/cuda ... html#Linux
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
Aitch
Posts: 6518
Joined: Wed 04 Apr 2007, 15:57
Location: Chatham, Kent, UK

#55 Post by Aitch »

How about a Puppy users p2p cluster?

http://gpu.sourceforge.net/index.php

http://sourceforge.net/projects/gpu/

Harness

http://www.csm.ornl.gov/harness/

Condor [previously listed?]

http://www.cs.wisc.edu/condor/

Using Network mapper

http://sourceforge.net/project/screensh ... p_id=58134

High speed compile environment, maybe? Looks fun!

Linux HA [high availability cluster] Users Guide

http://www.linux-ha.org/doc/users-guide ... guide.html

Aitch :)

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#56 Post by Master_wrong »

from
Linux HA [high availability cluster] Users Guide
This utility will copy the configuration files over to any node listed in /etc/ha.d/ha.cf using scp.
scp, that's what i use to set-up my puppy-nodes and server 8)

btw i have successfully test automatic setting of all node and server when using pxe...
it means puppy-cluster will only need 1 cd (and nodes with 1gb ram) that is it.
just boot server, boot all nodes, then after all computer boot up properly, all setting are set automagically, no need to input ip setting at server anymore but i open geany anyway just in case...

so I have accomplish my goal for the project

My goal for the project is
1. make puppy cluster computer ---> Done
2. setting all cluster from server ----> Done
3. automatic setting ----> Done
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
Aitch
Posts: 6518
Joined: Wed 04 Apr 2007, 15:57
Location: Chatham, Kent, UK

#57 Post by Aitch »

You forgot the 'crucial ending line'....

Testers needed! :D

As soon as I can get some time/space I intend to test it - thanks!

Have you got a working software list, or do I need to learn C++? :? :wink:

Aitch :)

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#58 Post by Master_wrong »

:D :D :D
Have you got a working software list, or do I need to learn C++? Confused Wink
indeed, john the ripper is working, netcrack seem working...
btw you do know that mpir.c from svat can be used to run normal program into mpi-cluster environment ?
You forgot the 'crucial ending line'....

Testers needed! Very Happy
sure aitch...

just make sure you create the humongous initrd first to boot pxe...

that means copy this script, make it autoload (also the unfs server if needed), remove the old autoconfig, remaster..., create humongous-initrd from remastered-iso...

then reboot with the new iso, set pxe into normal or debug mode,
then boot all other pc with pxe enabled when server ask
"Press ok if THIS PC IS SERVER..."

after all pc boot properly, click ok, see list of ip in geany, after you close geany, the cluster is ready to run mpich2.

attached is my test pic

Have a nice day

edit... I forgot to add dhcpcd -d
i use seperate file at startup which run dhcpcd -d

#!/bin/sh
tail -n +4 $0 > /tmp/test1.txt
exec rxvt -e sh /tmp/test1.txt

#------------------begin ssh setting
echo "ALL: ALL" > /etc/hosts.allow
echo " " > /etc/hosts.deny
if [ `egrep sshd /etc/passwd|wc -l` -eq 0 ]
then
echo "sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:sbin/nologin" >> /etc/passwd
fi
mkdir -p /var/empty/sshd
/usr/sbin/sshd
#-----------------end ssh setting

#begin check connection to network and show ip
no_network=""
ipp1=`/sbin/ifconfig eth0 | sed '/inet\ /!d;s/.*r://g;s/\ .*//g'`
until [[ $ipp1 != $no_network ]];do ipp1=`/sbin/ifconfig eth0 | sed '/inet\ /!d;s/.*r://g;s/\ .*//g'` ;sleep 3;echo;echo "Not connected to network ?";echo "please connect this computer to network"
done
echo $ipp1
#end

#check hostname master if available and run as server or client
ipp=$ipp1
ip=$ipp1"."
ipp2=`echo ${ip%.*.}`
ip_master=$ipp2".115"

echo "server ip= "$ip_master

m=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $ip_master "ls"`

if [ "$m" = "" ] ; then
echo "setting pc as server"
ifconfig eth0 $ip_master netmask 255.255.255.0 up
ifconfig eth0
#no master node--->be master
hostname puppy-1
rm ~/my-applications/clusterdata/ip-list
rm ~/my-applications/clusterdata/etc/hosts
rm /etc/hosts
echo "$ip_master puppy-1" >>~/my-applications/clusterdata/etc/hosts
echo "$ip_master puppy-1" >>/etc/hosts
echo "$ip_master" >~/my-applications/clusterdata/ip-list
/usr/sbin/netboot-server-jb.sh
echo "this computer is now server"

Xdialog --yesno "Press ok if THIS PC IS SERVER and make sure all pc-node are ready" 0 0 && server="confirm" ;echo $server "SERVER commencing..."
#--------------begin server
if [ "$server" == "confirm" ]; then
geany ~/my-applications/clusterdata/ip-list
xmessage "Please create list of all ip that you have,
put server on top of the list,
press ok after you save the list"
#---begin probing
n=0
rm ~/my-applications/clusterdata/home/mpd.hosts
rm ~/my-applications/clusterdata/etc/hosts

#ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no localhost -v
echo 127.0.0.1 localhost>>~/my-applications/clusterdata/etc/hosts
for i in `cut --delimiter=: -f1 ~/my-applications/clusterdata/ip-list`;do process=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $i "cat /proc/cpuinfo|grep ^processor|wc -l"`;let n="$n+1";echo $n;echo $i:$process>>~/my-applications/clusterdata/home/mpd.hosts;echo $i puppy-$n>>~/my-applications/clusterdata/etc/hosts;ssh $i hostname puppy-$n ;echo "computer:" $i "has:"$process " processor" ;done
#end probing

# begin setting server
cp -prv ~/my-applications/clusterdata/etc/* /etc/
cp -prv ~/my-applications/clusterdata/home/* ~
cp -prv ~/my-applications/clusterdata/home/.mpd.conf ~
# end setting server

#begin setting cluster
# for i in `cut --delimiter=: -f1 ~/mpd.hosts`;do ssh $i mkdir -p .ssh; cat ~/.ssh/id_rsa.pub |ssh $i "cat>>~/.ssh/authorized_keys";done
for i in `cut --delimiter=: -f1 ~/mpd.hosts`;do scp ~/.mpd.conf $i:~;scp ~/mpd.hosts $i:~;scp /etc/mpd.conf $i:/etc;scp /etc/hosts $i:/etc;done
fi
#end setting cluster
###set client nfs
for i in `cut --delimiter=: -f1 ~/my-applications/clusterdata/ip-list`;do process=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $i "mkdir /mnt/try"; ssh $i "mount -o port=4711,mountport=4711,mountvers=3,nfsvers=3,nolock,tcp $ip_master:/root /mnt/try
"`;done
####NFS



else
#master exist---->send ip data to master
echo "server exist"
ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $ip_master "echo $ipp>>~/my-applications/clusterdata/ip-list"

echo "this pc is ready as node"
echo $ipp
hostname
fi

cd ~/my-applications
rox
xmessage " this pc is ready to run cluster "
cd ~
urxvt
Attachments
test-server1.png
(16.74 KiB) Downloaded 1122 times
test-server2.png
(14.95 KiB) Downloaded 439 times
test-node.png
(5.06 KiB) Downloaded 1069 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

gcmartin

Excellent

#59 Post by gcmartin »

Master_wrong wrote:from Linux HA [high availability cluster] Users Guide .... so I have accomplish my goal for the project ... My goal for the project is ... ----> Done
Thanks for clarifying everything you've done thru-out this Thread. I have followed you work religiously. Very clear objectives, approaches, conclusion.

I have one complex request.
  • Would you setup your LiveCd to automatically start the PXE environment when it boots without user intervention.
  • Would you post the LiveCD/LiveDVD that would boot the Cluster Master starting the PXE environment with those files necessary for the cluster nodes to PXE from, please.
This would make an automatic (another word for dummies like me) cluster where all I do is boot "Live" and start nodes, wthout ever typing or clicking on anything, to begin and run my cluster.

You may want to start a new Thread announcing your Multi PC "ClusterPUP".
See my PM to you.

Again, your work here is phenomenal.
Hope this helps

User avatar
Aitch
Posts: 6518
Joined: Wed 04 Apr 2007, 15:57
Location: Chatham, Kent, UK

#60 Post by Aitch »

gcmartin wrote:You may want to start a new Thread announcing your Multi PC "ClusterPUP".
[See my PM to you.]

Again, your work here is phenomenal
2nd that, with big thumbs up :D [even though I haven't implemented it yet]

Anyone doing video/high intensity graphics/blender will benefit from this, however, can compiling be speeded up on a cluster, it is common to hear it taking 10/12 hrs?

Aitch :)

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#61 Post by Master_wrong »

@gcmartin
Thanks for clarifying everything you've done thru-out this Thread. I have followed you work religiously. Very clear objectives, approaches, conclusion.

I have one complex request.


* Would you setup your LiveCd to automatically start the PXE environment when it boots without user intervention.
* Would you post the LiveCD/LiveDVD that would boot the Cluster Master starting the PXE environment with those files necessary for the cluster nodes to PXE from, please.

This would make an automatic (another word for dummies like me) cluster where all I do is boot "Live" and start nodes, wthout ever typing or clicking on anything, to begin and run my cluster.

You may want to start a new Thread announcing your Multi PC "ClusterPUP".
See my PM to you.
well i can put netboot folder into root which will be saved during remaster, but the iso size will be enormously 600+M :?

what's important is that now it can boot headless pc. but you still need to click

@Aitch Smile
Anyone doing video/high intensity graphics/blender will benefit from this, however, can compiling be speeded up on a cluster, it is common to hear it taking 10/12 hrs?
here i test mpir...
it seems good
# mpirun -np 2 ~/mpir -s 101 -e 1000 'blender -b in.blend -s %start -e %end -a'
sh: blender: command not found
sh: blender: command not found
^C

# mpirun -np 2 ~/mpir -s 101 -e 1000 'dvdauthor -b in.blend -s %start -e %end -a'
DVDAuthor::dvdauthor, version 0.6.12-alpha.
Build options: gnugetopt magick iconv freetype
Send bugs to <dvdauthor-users@lists.sourceforge.net>

ERR: Must first specify -t, -m, or -x.
DVDAuthor::dvdauthor, version 0.6.12-alpha.
Build options: gnugetopt magick iconv freetype
Send bugs to <dvdauthor-users@lists.sourceforge.net>
^C
off course i didnt install blender and i dont know howto run dvdauthor... :lol:
this is called renderfarm i guess
Anyone willing to test ? :wink: [/quote]
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

testing mpich2-132

#62 Post by Master_wrong »

I have test Mpich2-132 on 5.2.
.
i have install required fortran compile and install it also run few test.

it seem new mpich didnt use mpdboot anymore, but hydra and what it need is just list of ip...
so i might have to create new script again.
edit... false alarm i didn configure it properly
Attachments
mpich2-132.jpg
(20.45 KiB) Downloaded 1570 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

gcmartin

remote FS use in Cluster

#63 Post by gcmartin »

Don't know if you are aware, but, SAMBA has been used om Cllusters for awhile. You may find this use a lot in cluster community. See here for starters. then you should join a SAMBA forum where a large community of user are available that will help.

See SAMBA cluster reference here and the Cluster support covers here.

Hope this helps

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#64 Post by Master_wrong »

Thank gcmartin, it's more like database solution for cluster and i bet the DB stands for Database...
Requirements
CTDB relies on a clustered filesystem being available and shared on all nodes that participate in the CTDB cluster. This filesystem must be mounted and available on all nodes in the CTDB cluster.

On top of this cluster filesystem, CTDB then provides clustered HA features so that data from the clustered filesystem can be exported through multiple nodes in the CTDB cluster using various services. Currently included with CTDB are the necessary hooks for Samba, NFS and ftp exports. Support for new service types can easily be added.
but sure very useful for some cluster application.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

gcmartin

CTDB and ClusterPup

#65 Post by gcmartin »

If you are familiar with sharing files from one Windows PC with the other PCs on the LAN, then this allows ClusterPUP to become a PC that can also share on the LAN, same as all Windows PCs.

Thus, ClusterPUP can not only "mount" Windows share resources, but with CTDB, it can make available folders and files for LAN PCs to read and write to for the cluster.. Now, as a simple example, I can load data, for cluster management/applications, directly from somewhere else on my LAN, AND, I can run applications within my cluster where the resulting data can be viewed, used, manipulated, or managed by things on my LAN. All this done with only lifting a finger!

Think of it as a "secure" data sharing gateway for your cluster.See this and PDF reference at bottom
and this wiki

Hope this helps

Post Reply