Puppy 245mpi cluster made easy

A home for all kinds of Puppy related projects
Message
Author
whatupuppy
Posts: 17
Joined: Tue 18 Jan 2011, 02:07

#16 Post by whatupuppy »

Sorry been busy and very new to cluster computing;
and want to learn more.
Master_wrong wrote:
I already post all picture that needed to run the cluster.
all you need is a working network that connect those computer.

if you ask how i build.....
http://www.murga-linux.com/puppy/viewto ... 7&start=15
Thanks for the wonderful instructions with pictures.

Master_wrong wrote:
wow you are building cluster ??? :lol:
were all the computer identical ? it would be best if they were identical.
Yes, to play around with and to learn parallel programming.
So far I got two P4 2.4 - 1GB RAM up and running.
I think I will have two more ready from the spare parts.
Also, the current setup is using about 2.1Amp of electricity, so I'm moving over to laptops. Trying to borrow some of my friends laptop from his office.

Also, would it better to do a full install to the hard drive if I have less RAM.
Another thought is what if the server is the least powerful, would it make a difference if the worker bees are much better.
Example:
Server: P2 - 700MHZ with 512MB
Client1 : Quad 2.4 Ghz 8GB RAM
Client2: Quad 2.4 Ghz 4GB RAM
Client3: P4 2.4 - 1GB RAM
Client4: P4 2.4 - 1GB RAM

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#17 Post by Master_wrong »

@acrocosm
Thank you for the info,

please try
grep -c ^processor /proc/cpuinfo

compared to
grep -c processor /proc/cpuinfo
Also I wanted to ask... is this vanilla luci 245 with a special sfs for clustering? Can I use the sfs on 5.2 or any other pupplet as long as the dependencies are satisfied?
yes this is...
if you want to build from 52 you need to download the dev for 52 instead then install it, i use pet maker to install it btw. install the mpich, also all the necessary program, phyton etc
What I'd like is all of them to act as one, shared memory, shared cpus. I don't expect shared graphics accelerators but it would be a nice touch. Is that possible? Is that a beowulf? Can I do that with this pupplet? ? Is that a beowulf? Can I do that with this pupplet?
:lol: :lol: :lol:

you ask to much, frankly i just learn this too. thank, btw yes this can be classified as beowulf. http://www.linuxjournal.com/article/5690?page=0,0
let me know how the blender thing work ok ?

@whatupuppy
Yes, to play around with and to learn parallel programming.
So far I got two P4 2.4 - 1GB RAM up and running.
I think I will have two more ready from the spare parts.
Also, the current setup is using about 2.1Amp of electricity, so I'm moving over to laptops. Trying to borrow some of my friends laptop from his office.

Also, would it better to do a full install to the hard drive if I have less RAM.
Another thought is what if the server is the least powerful, would it make a difference if the worker bees are much better.
Example:
Server: P2 - 700MHZ with 512MB
Client1 : Quad 2.4 Ghz 8GB RAM
Client2: Quad 2.4 Ghz 4GB RAM
Client3: P4 2.4 - 1GB RAM
Client4: P4 2.4 - 1GB RAM
yes its fun isnt ?

if you use different pc speed then you need to do manual load balancing... which i didnt have any clue atm.
or else the faster comp will sit idle, no matter as server or as client.
Last edited by Master_wrong on Tue 25 Jan 2011, 16:01, edited 1 time in total.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#18 Post by Master_wrong »

for those read petur's pdf, here is the program for john the ripper

http://www.bindshell.net/tools/johntheripper

please use the version that i posted ( john-1.7.3.1-all-2-mpi8 ) because other version i try resulted in buffer overflow error.
Attachments
john-1.7.3.1-all-2-mpi8b.jpg
(47.08 KiB) Downloaded 1221 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

gcmartin

Information on Clustering

#19 Post by gcmartin »

This is NOT an effort to hi-jack this thread. But, I believe in Clustering, and I just want to share some information that will prove useful to everyone.
_______________________________________
@whatupuppy asks about processor speed of master system versus the worker systems.

Having measured this in the past of non-Intel configurations I share:
The Master MUST be the fastest (or as fast as you can afford). Reason: If workers are faster, job check-in and check-out will not occur until the Master frees up. This cause spin-looping at the workers trying to exchange information to the Master. Remember, the workers are also looking for work as well as performing work.
_______________________________________
@Everyone else. Here's a tip: Parallel FS can enormously reduce your clustering's effort in achieving ANY future results you will want to achieve. The issue is "you don't want an I/O management nightmare which results from those you are not guided properly on the systems layout." Its NOT a necessity, in the beginning, but if its ignore, you will be abandoning your cluster in the future because of the manual workload it will place on you.

There are couple proven implementation available.
_______________________________________
Lastly the faster the bus between the systems within the cluster, and the lest encumbered they are (for example only cluster machines on the switch and the LAN cards in Master and workers, the better the cluster's performance. Remember, the cluster, logically, intends to be a "big single computer" with its way to the LAN via the master. And, the "Big Single Computer", Your Cluster, is a series of separate from the normal LAN computers coupled together via a set of LAN cards and a switch to each other cluster machine.Think of this as the bus/channel between the cluster machines who need to constantly stay in touch with each other. Even though ethernet technology is employed, the information passing back and forth is NOT LAN related. It's cluster related and they speak their own language. The faster this bus, the better the communications within the cluster, thus, all cluster machines should be using same speed LAN cards (or the Master "can" use the fastest LAN card), for obvious reasons.

This will make your life easier as you try to address cluster performance issues.
_______________________________________
Someone, MAY want to post a picture here to show a view of a cluster on a network. It will make it apparent what a cluster "really" is. This should be the first thing someone sees when they come here tring to understand and implement a cluster.

Again, this is ONLY some information that may prove useful to some.I Hope this helps
@Master_wrong look for a PM soon

whatupuppy
Posts: 17
Joined: Tue 18 Jan 2011, 02:07

#20 Post by whatupuppy »

@Master_wrong & gcmartin

Thanks for the info and sharing your knowledge.
My friend said he can lend me 6 - 1 gigabit network cards and a 1 - 24 ports 10gb router. I think that should help with communications between the computers.

I will have by the weekend:
(4) P4 2.4 - 1GB RAM w 20GB HD
(1 ) IBM R30 1.2 - 512MB RAM w 60GB HD
(1) Dell 8600 1.8 - 1GB RAM w 80GB HD

Will the IBM slow down the cluster?

On the topic of power consumption, my friend will trade me his laptops for the desktops. The desktop use about 1 amp each and the laptop only uses .40amp.

Also, what can I do to help?

gcmartin

Slow Cluster Master

#21 Post by gcmartin »

whatupuppy wrote:Will the IBM slow down the cluster?
Short answer, YES.

Explanation you can measure yourself: Your workers will NEVER achieve 100% utilitzation on HPC problems and large workloads. Your Master will bottleneck in high workload environments.
whatupuppy wrote:Also, what can I do to help?
Why not still use the IBM, but make it a worker. His SCSI drives should be a benefit to certain types of workloads. Its a Win-Win!
2 Questions:
  1. I'm a big IBM fan, but for this requirement, why do you want to make the IBM your master?
  2. What kinds of work will you have your cluster doing?
whatupuppy wrote: My friend said he can lend me 6 - 1 gigabit network cards and a 1 - 24 ports 10gb router. I think that should help with communications between the computers.
This means you are already looking at a very good solution. The faster those cards, the better your cluster can move work and talk to each other.

Also, in your future, consider a SAN with 6 cables to your Cluster heads using 10GB cards to each cluster and you now would be entering the realm of "SuperHuman" HPCs. Couple with Parallel I/O filesystem, and you should be able to fly ... so to speak.

Hope this helps.
P.S. Don't forget my 2 questions, please. I have a personal application I'd like to run in your cluster once you're all set up. It should take about 1 cluster minute.
Last edited by gcmartin on Wed 26 Jan 2011, 07:43, edited 6 times in total.

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#22 Post by Master_wrong »

I will have by the weekend:
(4) P4 2.4 - 1GB RAM w 20GB HD
(1 ) IBM R30 1.2 - 512MB RAM w 60GB HD
(1) Dell 8600 1.8 - 1GB RAM w 80GB HD

Will the IBM slow down the cluster?
yes, but its nice to experiment, try the flop.c program
Also, what can I do to help?

you can try blender ? with program mpir which i download from http://svat.sourceforge.net/
btw try use mpicc to compile the program

or would you compute chemical analisis...
http://classic.chem.msu.su/gran/gamess/

otherwise you can post your experiment on youtube or something, promote puppy :lol: :lol: :lol:

@gcmartin

thank you for the info... btw i also heard that if we really serious with cluster we also need to patch the kernel to maximize network speed... whatever it mean.
btw here is some info about parallel fs
http://blogs.sun.com/hstsao/entry/paral ... or_cluster
Attachments
mpir.c.zip
(5.89 KiB) Downloaded 283 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
acrocosm
Posts: 44
Joined: Sun 12 Sep 2010, 17:31
Contact:

#23 Post by acrocosm »

Master_wrong wrote: please try
grep -c ^processor /proc/cpuinfo
Gives 1
Master_wrong wrote: compared to
grep -c processor /proc/cpuinfo
Gives 2

What exactly does that mean?
gcmartin wrote:This is NOT an effort to hi-jack this thread.
How is helping and sharing knowledge hi-jacking? On the contrary... I'd consider myself with my limited knowledge and stupid questions more of a hi-jacker. In any case though I'd like to learn more and help if I can, it's an interesting project. So I'll stay around and occasionally bother you guys ^______^


Very interesting stuff. Personally I'm still just scratching the surface though. According to this shared RAM is out of the question. Is that right? Also apparently this won't actually make it act as one multiprocessor machine exactly, I'll still have to manually split the work somehow whether that's based on frames (explicitly for animation) or rows/blocks. hmmm, hmmm.

The machines I currently have available here are:
1 - Custom build Quadcore Q6600, 2GB DDR2 (master)
2 - X40 with 1.2 Pentium M, 1GB DDR (slave)
3 - Custom build Pentium 4 at 2.8 with HT, 1GB DDR (not joined yet)

There's a Pentium 4@2.4 and one more at 2.66, if I remember correctly, at my parents place, both with 512 DDR but those are in another city since my flat is too small to stock machines. If everything goes well I'll take them too.

Don't laugh, but I'm currently connecting them through wifi. I know it's bad performance wise, but I'm just testing and learning atm so it doesn't matter at all. That's why I don't care to add the other spare joined.

Anyway, maybe that's obvious for you guys but I run blender on the master off this 2-machine cluster just to see if it could see beyond the 4 cpus. It didn't but maybe it's the internal rendered that doesn't support mpi(?). I'll try yafaray or luxblend next and that svat thing, although it seems a bit scary ^_^

gcmartin

Personal view

#24 Post by gcmartin »

One way I view a cluster is as follows:
let's say I have 5 machines and one of them is the Master ("the entrance desk for the cluster")

I look at this as having 4 physically hungry dogs salivating, waiting, and looking for work. And you have this master whose job it is to throw the meat into the bin for the dogs to feed on. And, at the same time, the Master himself is salivating, looking to feed.

That's my idea of a cluster. Jobs go in and out thru the Master, while the dogs devour the feedings (work) and letting the Master know they're hungry again. Bigger the dog, less for the other guys.

IBM makes the baddest of the clusters. Univ. of Virginia has a Largest Mac Cluster in world. etc. There are clusters all over the world; large and small....just looks like another LAN machine.

A home cluster could let you do all sorts of things while on your home LAN. And there are Public Projects available that you can participate in which addresses all kinds of world problems from SETI to Space Station to Hubble to Water Shortages to Weather to monitoring community Biological Carbon Units activity. And can be used in Theory and Games. Wide Open!

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#25 Post by Master_wrong »

What exactly does that mean?
that mean that we have to change the script to correct the error...

here try this script...
what is changed :
- fix counting the processor for some older computer
- makes really-really automatic ssh (no more typing yes answer)

<script moved to first post>
Last edited by Master_wrong on Tue 01 Feb 2011, 11:04, edited 1 time in total.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
acrocosm
Posts: 44
Joined: Sun 12 Sep 2010, 17:31
Contact:

#26 Post by acrocosm »

Master_wrong wrote:
What exactly does that mean?
that mean that we have to change the script to correct the error...

here try this script...
what is changed :
- fix counting the processor for some older computer
- makes really-really automatic ssh (no more typing yes answer)
I'm afraid I still get the same results

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#27 Post by Master_wrong »

Sorry,
here try this one

this one, the counting process is

process=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $i "grep -c ^processor /proc/cpuinfo"`

which from your answer should do the trick

@gcmartin...

I have found the nfs...
i will test them tommorow
Attachments
acrocosm.sh.zip
(1.1 KiB) Downloaded 401 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

gcmartin

Practical use of cluster

#28 Post by gcmartin »

One problem 99% of all people will have with Clusters: "Give me a practical application that I can identify with". They cannot understand how to apply it.

Here's an application that should make it easy to understand. Here ONE method of applying it done by a vendor. I am NOT pushing their product; I am showing an example of what we should be looking and thinking here in Puppyland. Review this This is a very practical use.

This should now make it easier to understand and begin applying Clusters. Also, this is the basis of what Citrix does thanks to their modeling after IBM did this in the 90's.

Now what applications can we replicate or do in Puppyland Clusters...hmmm?
P.S. It occurs that if we could replicate something like this in a Puppy Cluster, we could run every PUP, every PET, every SFS, and everything in the cluster. Now, hmmmm...this is some real food for those Workers to feed on from the Master. Big Dogs versus Small Dogs rules will still apply, though. But, the work will be spread in a very interesting manner.
Last edited by gcmartin on Fri 28 Jan 2011, 20:58, edited 1 time in total.

User avatar
acrocosm
Posts: 44
Joined: Sun 12 Sep 2010, 17:31
Contact:

#29 Post by acrocosm »

Master_Wrong I'm sorry, I have to clarify...

The original scripts included on the iso would show 2 cpu for the x40 on the server. Both of your later auto-config scripts worked fine. The server would see just one cpu on the x40 as expected

However,

Code: Select all

grep -c processor /proc/cpuinfo
still gives 2 as a result.

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#30 Post by Master_wrong »

However,
Code:
grep -c processor /proc/cpuinfo
still gives 2 as a result.
so there is no problem. because grep -c processor /proc/cpuinfo incorrectly count processor, your /proc/cpuinfo might contain word processor twice which resulted in wrong count. this is used in the first script.

about nfs...
check

http://murga-linux.com/puppy/viewtopic.php?t=45090

its working but we can only mount /mnt/sda, we only need to install nfs_common.tar.gz
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#31 Post by jpeps »

acrocosm wrote:Master_Wrong I'm sorry, I have to clarify...

The original scripts included on the iso would show 2 cpu for the x40 on the server. Both of your later auto-config scripts worked fine. The server would see just one cpu on the x40 as expected

However,

Code: Select all

grep -c processor /proc/cpuinfo
still gives 2 as a result.
grep -c ^processor

gcmartin

Cluster applications

#32 Post by gcmartin »

IFF you are wondering how others have applied clusters, other than the one's I've mentioned earlier, look here for ways to use yours!
Hope this helps

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#33 Post by Master_wrong »

@gcmartin
P.S. It occurs that if we could replicate something like this in a Puppy Cluster, we could run every PUP, every PET, every SFS, and everything in the cluster. Now, hmmmm...this is some real food for those Workers to feed on from the Master. Big Dogs versus Small Dogs rules will still apply, though. But, the work will be spread in a very interesting manner.
I dont think that using cluster will benefit all program, such as office suite etc they dont need that much processing power...
i believe that those computer that waiting idle in screensaver mode would be more useful if used as cluster. but not all program need cluster processing power.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#34 Post by jpeps »

Master_wrong wrote:
I dont think that using cluster will benefit all program, such as office suite etc they dont need that much processing power
Seems like office suite, etc., would need unique software to make use of clustering (i.e, that can run parallel tasks); doesn't make much sense for basic apps.

gcmartin

Mistaken understanding

#35 Post by gcmartin »

Master_wrong wrote:
P.S. It occurs that if we could replicate something like this in a Puppy Cluster ... .
I dont think that using cluster will benefit all program, such as office suite etc they dont need that much processing power...
i believe that those computer that waiting idle in screensaver mode would be more useful if used as cluster. but not all program need cluster processing power.
Thanks Master_wrong (and @jpeps too). I agree with you that today's Office Suite is NOT an application for clustering (and I don't believe I suggested it either).

The example I provided was one where someone has found a way to dispatch VMs as work in a cluster. That's novel, if it works as they suggest and is a greater way to throw workers at workload. I only threw that our way in Puppyland to inspire ideas for what can be accomplished in a Puppy cluster.

Hope this helps

Post Reply