Puppy 245mpi cluster made easy

A home for all kinds of Puppy related projects
Message
Author
Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

Puppy 245mpi cluster made easy

#1 Post by Master_wrong »

-----------------
edit march-18-2011 -Howto tutorial
there is a complete installation manual available at "ClusterPUP - A Very Simple Cluster Implementation" -
https://docs.google.com/document/d/1jUK ... 4ML&pli=1#

-----------------------------

edit feb 27-2011 -newest puplet here----> with pxe, auto detect and setting

http://www.murga-linux.com/puppy/viewto ... 199#499199

--------------------



http://hotfile.com/dl/94559819/a465c13/ ... r.zip.html

this is my first puplet....
all it does is simplify setting of mpich cluster... and made it as easy as 123...

Edit-feb-1-2011:
attached below is new config-script
1. improve key-less ssh
2. fix counting processor
3. nfs script (need to install nfs_common.tar.gz from http://murga-linux.com/puppy/viewtopic.php?t=45090
both server and client will be able to access /mnt/my. server need to mount sda1)

no more installing stuff into your desktop pc , al you need to have is cd and no need to setting ssh password-less and mpich which complicated for most ppl.
bottom line... no more headache for trying cluster computer....

see picture
1. basic setting, time date, hostname, nothing to edit here, my program will automatically set the hostname. press ok
2. Network wizard, select your network setting accordingly
3. If connection to network succed, a question will pop up, leave them if you are not on server pc
4. If you are on server pc then click yes and enter the list ip of all pc, put server ip on top, save and exit.
5. First time ssh will ask question, answer yes
6. Run mpd, compile program c, then run it, what was wrong ?... forgot to distribute the compiled program... use cluster-ssh-fileshare.sh
7. Program work after compiled program are distributed across cluster pc

I you want to know more about clustering....
http://www.phy.duke.edu/~rgb/Beowulf/be ... index.html

also here is pdf to help using mpich2
http://www.fz-juelich.de/jsc/datapool/p ... nstall.pdf

Thank to gcmartin for the picture:
https://docs.google.com/drawings/edit?i ... l=en&pli=1

have a nice day
Attachments
config-script.tar.gz
(1.64 KiB) Downloaded 754 times
4.png
(190.3 KiB) Downloaded 903 times
6.png
(164.06 KiB) Downloaded 885 times
7.png
(162.6 KiB) Downloaded 924 times
Last edited by Master_wrong on Thu 28 Apr 2011, 01:42, edited 13 times in total.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
puppyiso
Posts: 594
Joined: Tue 13 Jan 2009, 02:27

#2 Post by puppyiso »

Can you provide more detailed use manual?

Most importantly how to configure the network and the internet?

CD only or can be installed to USB?

John

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#3 Post by Master_wrong »


Can you provide more detailed use manual?
done
Most importantly how to configure the network and the internet?
the configuration to passwordless ssh and mpich are automatic.
about internet... this puplet are not meant to connect to internet, the main purpose is for study cluster computer and research.
CD only or can be installed to USB?
yes you can install on cd and usb...
but remember then the size is 294M, so for pc with 512M or lower ram i recommend to use cd, its cheaper than usb.

if all pc has 1 G ram, you can use usb and boot all using just that 1 usb.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

My puppy has more flop than yours

#4 Post by Master_wrong »

I add interesting program....


flop.c ----> to count how many flop your cluster pc has to compare with those super computer.

now i can say that my puppy has more flop than yours :lol: :lol: :lol: :lol: :lol:
Attachments
flop.c.zip
(925 Bytes) Downloaded 552 times
flop.jpg
(96.18 KiB) Downloaded 5239 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

gcmartin

"Masterful" effort...Thanks

#5 Post by gcmartin »

I run a LiveCD environment. So, for clustering several PCs to accomplish workload under a single-headed "Master" is an accomplishment for certain types of workloads. I applaud your accomplishment as this is significant.

I'm new to PC clustering. I have couple of questions? (assume a 3 PC running cluster)
  • Is your work for 32bit or 64bit PCs?
  • Does your current work require all of the cluster machines to be same?
  • To dispatch work, how do the cluster machines know when to pitch-in (dynamic work allocation or static)?
  • Assuming that the master has 2 LAN cards (1 to cluster switch-other to normal LAN)Can the master run a full Puppy desktop configuration as it handles workload management to those in the cluster?
This appears, to me, to be a very significant breakthrough in the Puppy world.
Thanks in advance

gcmartin

Re: My puppy has more flop than yours

#6 Post by gcmartin »

@Master ...
Master_wrong wrote:I add interesting program....flop.c ---> to count how many flop your cluster pc has to compare with those super computer. ...
Can you help by sharing how to interpret the data shown? Trying to understand what I'm seeing in terms of process/job dispatch and what is in your cluster. Please explain.

Thanks in advance

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#7 Post by Master_wrong »

Hi Gcmartin,

:lol: :lol:

I was confused with my finding myself....

the first data with -np 1 -----> is using 1 processor only so there puppy-1 700 Mflop
2nd data -np 2 -----> 2 processor, because its dual core both are puppy-1 14000 Mflop, exactly double

so the funny stuff begun when other cluster join in

3rd data -np 3 -----> use 3 processor so it use puppy-1 and puppy-2 the drop to 590Mflop
4th data -np 4 -----> use 4 processor another drop to 400 Mflop, all processor on both computer are used

I assume because my destop(puppy-1) is newer than my laptop(puppy-2) this actually means that my laptop are too slow which hit the flop, but if we see the error then we see that clustering help minimize the rate of error.
I run a LiveCD environment. So, for clustering several PCs to accomplish workload under a single-headed "Master" is an accomplishment for certain types of workloads. I applaud your accomplishment as this is significant.
thank you
* Is your work for 32bit or 64bit PCs?
* Does your current work require all of the cluster machines to be same?
* To dispatch work, how do the cluster machines know when to pitch-in (dynamic work allocation or static)?
* Assuming that the master has 2 LAN cards (1 to cluster switch-other to normal LAN)Can the master run a full Puppy desktop configuration as it handles workload management to those in the cluster?
- i believe it was 32 bit.
- seen from my experiment then it is
- mpich will dispatch the workload among cluster, because mpich also installed in the cluster machine and has all program needed to run.
- i believe that if we have enough processor power to handle and we set the setting correctly, ie: we have dualcore so use 1 for puppy and use 1 for cluster work... i think it will work.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

gcmartin

Unexplained behavior

#8 Post by gcmartin »

Master...
I hope this makes sense. When I run an OS aware of multi-core processors, the hardware, under OS direction sometimes, is managing what one can consider to be "clustering".

When we move from local hardware clustering into "soft" clustering, we introduce a "new" bus that is used in communication of process controls, where it is significantly slower than the local multi-core bus. This should impact the behavior of some applications. So, application design is a factor depending upon what the objectives are to be.

In this case, we are seeing how "this particular program, only," behaves in your cluster. Another program with different design objectives in its design would be able to exploit you current cluster in a different fashion such that you would meet a different result. Further, there MAY be some cluster OS controls which, too, MAY affect /enhance program's performance in soft cluster.

Hope this helps.
P.S, My question on Puppy was to try to determine if you felt that a X-desktop console would contribute to the overall issues that would benefit cluster management, use, and performance.

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#9 Post by Master_wrong »

I try to compute how much flop does my laptop has....

it s just 200...
so imagine this pararel configuration while 2 other has 700 and my laptop has 200 so what happen is those faster processor just wait idle for the laptop to finish his work and dropping the flop count in the process.

it also makes senses too that
200x3 = 600 equal to the result of -np 3 , because workload is split by 3 processor
200x2 = 400 equal to the result of -np 4, because workload is split by 2 computer

btw my laptop seem just single processor, puppy see it as single processor and mpi treated it as single processor.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

whatupuppy
Posts: 17
Joined: Tue 18 Jan 2011, 02:07

#10 Post by whatupuppy »

Can you let me know your build process for puppy-cluster?

I'm waiting for a couple of computer and will test your puppy-cluster build.

Can not wait.

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#11 Post by Master_wrong »

Can you let me know your build process for puppy-cluster?
I already post all picture that needed to run the cluster.
all you need is a working network that connect those computer.

if you ask how i build.....
http://www.murga-linux.com/puppy/viewto ... 7&start=15

I'm waiting for a couple of computer and will test your puppy-cluster build.
wow you are building cluster ??? :lol:
were all the computer identical ? it would be best if they were identical.


btw found few great software too
http://freshmeat.net/search?page=12&q=c ... =&without=


some of them....
NetCrack is cluster software developed to distribute a hashing algorithm's cracking process work using a brute force attack.
offcourse i havent try any of them them
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
acrocosm
Posts: 44
Joined: Sun 12 Sep 2010, 17:31
Contact:

#12 Post by acrocosm »

Hmm so considering I have this properly set up and running on 3-4 machines, can I run blender to render a single image on all of them without having to configure anything else?

Or i need to run blender with the same scene on each of them and from the master machine I'll say: slave 1 render this layer, slave 2 render that layer and so on?

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#13 Post by Master_wrong »

you need a specially patched program.

but maybe i was wrong....


try.... http://svat.sourceforge.net/

they use blender...
The first example renders 2,000 frames with Blender on a 512 CPU computer. Notice how mpir uses a command-line template. You specify a frame range length with the '-g' option. Progress is echoed as a table.

# Start the multi-cpu rendering.
mpirun -np $NSLOTS mpir -S 15 -s 1 -e 2000 -g 15 'blender -b scene.blend -o /tmp/ -F PNG -x 1 -t 1 -s %start -e %end -a'
* mpir - This is a generic server/client MPI program to delegate independent tasks to many processors. This makes it easy to use graphics tools on MPI-centric hosts. See examples of its use are in Section 4, such as batch rendering or batch image post-processing.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
acrocosm
Posts: 44
Joined: Sun 12 Sep 2010, 17:31
Contact:

#14 Post by acrocosm »

Thanks for the link and of course thank you for your work!! I'll study that site and see what I can figure out.

Also I wanted to ask... is this vanilla luci 245 with a special sfs for clustering? Can I use the sfs on 5.2 or any other pupplet as long as the dependencies are satisfied?


Ultimately what I'd like to do is use the cannibalized spare machines I found on the streets or from friends to build a mini renderfarm for me and a friend. But I'm not so clear about distributed, grid, clustering and so on. I'd be obliged if you could clarify...

I'd like to run those machines diskless and if possible monitorless. Because I have 3-4 machines on the same room I don't see any reason in setting up a network where I run blender with the same scene and from one of those I tell machine2 render frames 1-100, machine3 render 101-200 etc. I can do that on each of them manually, they are next to another. Or I could use VNC to remotely control them and start the rendering.

What I'd like is all of them to act as one, shared memory, shared cpus. I don't expect shared graphics accelerators but it would be a nice touch. Is that possible? Is that a beowulf? Can I do that with this pupplet?

Sorry if I'm such a noob ^_^ and thank you again for your work. Using super fast and light puppy for clustering is such a brilliant idea!

User avatar
acrocosm
Posts: 44
Joined: Sun 12 Sep 2010, 17:31
Contact:

#15 Post by acrocosm »

Ok I'm setting this up on 2 machines to check if blender can see all CPUs. I got something odd...

Code: Select all

inet addr:127.0.0.1 Mask:255.0.0.0 inet addr:192.168.1.77 Bcast:192.168.1.255 Mask:255.255.255.0
confirm SERVER commencing...
1
computer: 192.168.1.77 has:4  processor
2
computer: 192.168.1.66 has:2  processor
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet addr:192.168.1.77  
`/root/my-applications/clusterdata/etc/hosts' -> `/etc/hosts'
`/root/my-applications/clusterdata/etc/mpd.conf' -> `/etc/mpd.conf'
`/root/my-applications/clusterdata/home/mpd.hosts' -> `/root/mpd.hosts'
`/root/my-applications/clusterdata/home/.mpd.conf' -> `/root/.mpd.conf'
.mpd.conf                                     100%   20     0.0KB/s   00:00    
mpd.hosts                                     100%   30     0.0KB/s   00:00    
mpd.conf                                      100%   20     0.0KB/s   00:00    
hosts                                         100%   62     0.1KB/s   00:00    
.mpd.conf                                     100%   20     0.0KB/s   00:00    
mpd.hosts                                     100%   30     0.0KB/s   00:00    
mpd.conf                                      100%   20     0.0KB/s   00:00    
hosts                                         100%   62     0.1KB/s   00:00
Check computer 2, it is reported as having 2 proccessors. That's not possible. This machine is a Thinkpad x40 with a 1.2 pentium M. Dual cores didn't even exist back then! I'm puzzled... o.O

whatupuppy
Posts: 17
Joined: Tue 18 Jan 2011, 02:07

#16 Post by whatupuppy »

Sorry been busy and very new to cluster computing;
and want to learn more.
Master_wrong wrote:
I already post all picture that needed to run the cluster.
all you need is a working network that connect those computer.

if you ask how i build.....
http://www.murga-linux.com/puppy/viewto ... 7&start=15
Thanks for the wonderful instructions with pictures.

Master_wrong wrote:
wow you are building cluster ??? :lol:
were all the computer identical ? it would be best if they were identical.
Yes, to play around with and to learn parallel programming.
So far I got two P4 2.4 - 1GB RAM up and running.
I think I will have two more ready from the spare parts.
Also, the current setup is using about 2.1Amp of electricity, so I'm moving over to laptops. Trying to borrow some of my friends laptop from his office.

Also, would it better to do a full install to the hard drive if I have less RAM.
Another thought is what if the server is the least powerful, would it make a difference if the worker bees are much better.
Example:
Server: P2 - 700MHZ with 512MB
Client1 : Quad 2.4 Ghz 8GB RAM
Client2: Quad 2.4 Ghz 4GB RAM
Client3: P4 2.4 - 1GB RAM
Client4: P4 2.4 - 1GB RAM

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#17 Post by Master_wrong »

@acrocosm
Thank you for the info,

please try
grep -c ^processor /proc/cpuinfo

compared to
grep -c processor /proc/cpuinfo
Also I wanted to ask... is this vanilla luci 245 with a special sfs for clustering? Can I use the sfs on 5.2 or any other pupplet as long as the dependencies are satisfied?
yes this is...
if you want to build from 52 you need to download the dev for 52 instead then install it, i use pet maker to install it btw. install the mpich, also all the necessary program, phyton etc
What I'd like is all of them to act as one, shared memory, shared cpus. I don't expect shared graphics accelerators but it would be a nice touch. Is that possible? Is that a beowulf? Can I do that with this pupplet? ? Is that a beowulf? Can I do that with this pupplet?
:lol: :lol: :lol:

you ask to much, frankly i just learn this too. thank, btw yes this can be classified as beowulf. http://www.linuxjournal.com/article/5690?page=0,0
let me know how the blender thing work ok ?

@whatupuppy
Yes, to play around with and to learn parallel programming.
So far I got two P4 2.4 - 1GB RAM up and running.
I think I will have two more ready from the spare parts.
Also, the current setup is using about 2.1Amp of electricity, so I'm moving over to laptops. Trying to borrow some of my friends laptop from his office.

Also, would it better to do a full install to the hard drive if I have less RAM.
Another thought is what if the server is the least powerful, would it make a difference if the worker bees are much better.
Example:
Server: P2 - 700MHZ with 512MB
Client1 : Quad 2.4 Ghz 8GB RAM
Client2: Quad 2.4 Ghz 4GB RAM
Client3: P4 2.4 - 1GB RAM
Client4: P4 2.4 - 1GB RAM
yes its fun isnt ?

if you use different pc speed then you need to do manual load balancing... which i didnt have any clue atm.
or else the faster comp will sit idle, no matter as server or as client.
Last edited by Master_wrong on Tue 25 Jan 2011, 16:01, edited 1 time in total.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#18 Post by Master_wrong »

for those read petur's pdf, here is the program for john the ripper

http://www.bindshell.net/tools/johntheripper

please use the version that i posted ( john-1.7.3.1-all-2-mpi8 ) because other version i try resulted in buffer overflow error.
Attachments
john-1.7.3.1-all-2-mpi8b.jpg
(47.08 KiB) Downloaded 1221 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

gcmartin

Information on Clustering

#19 Post by gcmartin »

This is NOT an effort to hi-jack this thread. But, I believe in Clustering, and I just want to share some information that will prove useful to everyone.
_______________________________________
@whatupuppy asks about processor speed of master system versus the worker systems.

Having measured this in the past of non-Intel configurations I share:
The Master MUST be the fastest (or as fast as you can afford). Reason: If workers are faster, job check-in and check-out will not occur until the Master frees up. This cause spin-looping at the workers trying to exchange information to the Master. Remember, the workers are also looking for work as well as performing work.
_______________________________________
@Everyone else. Here's a tip: Parallel FS can enormously reduce your clustering's effort in achieving ANY future results you will want to achieve. The issue is "you don't want an I/O management nightmare which results from those you are not guided properly on the systems layout." Its NOT a necessity, in the beginning, but if its ignore, you will be abandoning your cluster in the future because of the manual workload it will place on you.

There are couple proven implementation available.
_______________________________________
Lastly the faster the bus between the systems within the cluster, and the lest encumbered they are (for example only cluster machines on the switch and the LAN cards in Master and workers, the better the cluster's performance. Remember, the cluster, logically, intends to be a "big single computer" with its way to the LAN via the master. And, the "Big Single Computer", Your Cluster, is a series of separate from the normal LAN computers coupled together via a set of LAN cards and a switch to each other cluster machine.Think of this as the bus/channel between the cluster machines who need to constantly stay in touch with each other. Even though ethernet technology is employed, the information passing back and forth is NOT LAN related. It's cluster related and they speak their own language. The faster this bus, the better the communications within the cluster, thus, all cluster machines should be using same speed LAN cards (or the Master "can" use the fastest LAN card), for obvious reasons.

This will make your life easier as you try to address cluster performance issues.
_______________________________________
Someone, MAY want to post a picture here to show a view of a cluster on a network. It will make it apparent what a cluster "really" is. This should be the first thing someone sees when they come here tring to understand and implement a cluster.

Again, this is ONLY some information that may prove useful to some.I Hope this helps
@Master_wrong look for a PM soon

whatupuppy
Posts: 17
Joined: Tue 18 Jan 2011, 02:07

#20 Post by whatupuppy »

@Master_wrong & gcmartin

Thanks for the info and sharing your knowledge.
My friend said he can lend me 6 - 1 gigabit network cards and a 1 - 24 ports 10gb router. I think that should help with communications between the computers.

I will have by the weekend:
(4) P4 2.4 - 1GB RAM w 20GB HD
(1 ) IBM R30 1.2 - 512MB RAM w 60GB HD
(1) Dell 8600 1.8 - 1GB RAM w 80GB HD

Will the IBM slow down the cluster?

On the topic of power consumption, my friend will trade me his laptops for the desktops. The desktop use about 1 amp each and the laptop only uses .40amp.

Also, what can I do to help?

Post Reply