Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Mon 01 Sep 2014, 17:36
All times are UTC - 4
 Forum index » Advanced Topics » Puppy Projects
Puppy 245mpi cluster made easy
Moderators: Flash, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 2 of 5 [72 Posts]   Goto page: Previous 1, 2, 3, 4, 5 Next
Author Message
whatupuppy

Joined: 17 Jan 2011
Posts: 17

PostPosted: Mon 24 Jan 2011, 22:08    Post subject:  

Sorry been busy and very new to cluster computing;
and want to learn more.

[quote="Master_wrong"]
Quote:

I already post all picture that needed to run the cluster.
all you need is a working network that connect those computer.

if you ask how i build.....
http://www.murga-linux.com/puppy/viewtopic.php?t=63177&start=15


Thanks for the wonderful instructions with pictures.


[quote="Master_wrong"]
Quote:

wow you are building cluster ??? Laughing
were all the computer identical ? it would be best if they were identical.


Yes, to play around with and to learn parallel programming.
So far I got two P4 2.4 - 1GB RAM up and running.
I think I will have two more ready from the spare parts.
Also, the current setup is using about 2.1Amp of electricity, so I'm moving over to laptops. Trying to borrow some of my friends laptop from his office.

Also, would it better to do a full install to the hard drive if I have less RAM.
Another thought is what if the server is the least powerful, would it make a difference if the worker bees are much better.
Example:
Server: P2 - 700MHZ with 512MB
Client1 : Quad 2.4 Ghz 8GB RAM
Client2: Quad 2.4 Ghz 4GB RAM
Client3: P4 2.4 - 1GB RAM
Client4: P4 2.4 - 1GB RAM
Back to top
View user's profile Send private message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Tue 25 Jan 2011, 05:59    Post subject:  

@acrocosm
Thank you for the info,

please try
grep -c ^processor /proc/cpuinfo

compared to
grep -c processor /proc/cpuinfo

Quote:

Also I wanted to ask... is this vanilla luci 245 with a special sfs for clustering? Can I use the sfs on 5.2 or any other pupplet as long as the dependencies are satisfied?


yes this is...
if you want to build from 52 you need to download the dev for 52 instead then install it, i use pet maker to install it btw. install the mpich, also all the necessary program, phyton etc

Quote:

What I'd like is all of them to act as one, shared memory, shared cpus. I don't expect shared graphics accelerators but it would be a nice touch. Is that possible? Is that a beowulf? Can I do that with this pupplet? ? Is that a beowulf? Can I do that with this pupplet?


Laughing Laughing Laughing

you ask to much, frankly i just learn this too. thank, btw yes this can be classified as beowulf. http://www.linuxjournal.com/article/5690?page=0,0
let me know how the blender thing work ok ?

@whatupuppy
Quote:
Yes, to play around with and to learn parallel programming.
So far I got two P4 2.4 - 1GB RAM up and running.
I think I will have two more ready from the spare parts.
Also, the current setup is using about 2.1Amp of electricity, so I'm moving over to laptops. Trying to borrow some of my friends laptop from his office.

Also, would it better to do a full install to the hard drive if I have less RAM.
Another thought is what if the server is the least powerful, would it make a difference if the worker bees are much better.
Example:
Server: P2 - 700MHZ with 512MB
Client1 : Quad 2.4 Ghz 8GB RAM
Client2: Quad 2.4 Ghz 4GB RAM
Client3: P4 2.4 - 1GB RAM
Client4: P4 2.4 - 1GB RAM


yes its fun isnt ?

if you use different pc speed then you need to do manual load balancing... which i didnt have any clue atm.
or else the faster comp will sit idle, no matter as server or as client.

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199

Last edited by Master_wrong on Tue 25 Jan 2011, 12:01; edited 1 time in total
Back to top
View user's profile Send private message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Tue 25 Jan 2011, 07:02    Post subject:  

for those read petur's pdf, here is the program for john the ripper

http://www.bindshell.net/tools/johntheripper

please use the version that i posted ( john-1.7.3.1-all-2-mpi8 ) because other version i try resulted in buffer overflow error.
john-1.7.3.1-all-2-mpi8b.jpg
 Description   
 Filesize   47.08 KB
 Viewed   1165 Time(s)

john-1.7.3.1-all-2-mpi8b.jpg


_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
gcmartin

Joined: 14 Oct 2005
Posts: 4238
Location: Earth

PostPosted: Tue 25 Jan 2011, 16:58    Post subject: Information on Clustering  

This is NOT an effort to hi-jack this thread. But, I believe in Clustering, and I just want to share some information that will prove useful to everyone.
_______________________________________
@whatupuppy asks about processor speed of master system versus the worker systems.

Having measured this in the past of non-Intel configurations I share:
The Master MUST be the fastest (or as fast as you can afford). Reason: If workers are faster, job check-in and check-out will not occur until the Master frees up. This cause spin-looping at the workers trying to exchange information to the Master. Remember, the workers are also looking for work as well as performing work.
_______________________________________
@Everyone else. Here's a tip: Parallel FS can enormously reduce your clustering's effort in achieving ANY future results you will want to achieve. The issue is "you don't want an I/O management nightmare which results from those you are not guided properly on the systems layout." Its NOT a necessity, in the beginning, but if its ignore, you will be abandoning your cluster in the future because of the manual workload it will place on you.

There are couple proven implementation available.
_______________________________________
Lastly the faster the bus between the systems within the cluster, and the lest encumbered they are (for example only cluster machines on the switch and the LAN cards in Master and workers, the better the cluster's performance. Remember, the cluster, logically, intends to be a "big single computer" with its way to the LAN via the master. And, the "Big Single Computer", Your Cluster, is a series of separate from the normal LAN computers coupled together via a set of LAN cards and a switch to each other cluster machine.Think of this as the bus/channel between the cluster machines who need to constantly stay in touch with each other. Even though ethernet technology is employed, the information passing back and forth is NOT LAN related. It's cluster related and they speak their own language. The faster this bus, the better the communications within the cluster, thus, all cluster machines should be using same speed LAN cards (or the Master "can" use the fastest LAN card), for obvious reasons.

This will make your life easier as you try to address cluster performance issues.
_______________________________________
Someone, MAY want to post a picture here to show a view of a cluster on a network. It will make it apparent what a cluster "really" is. This should be the first thing someone sees when they come here tring to understand and implement a cluster.

Again, this is ONLY some information that may prove useful to some.I Hope this helps
@Master_wrong look for a PM soon

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send private message 
whatupuppy

Joined: 17 Jan 2011
Posts: 17

PostPosted: Tue 25 Jan 2011, 23:18    Post subject:  

@Master_wrong & gcmartin

Thanks for the info and sharing your knowledge.
My friend said he can lend me 6 - 1 gigabit network cards and a 1 - 24 ports 10gb router. I think that should help with communications between the computers.

I will have by the weekend:
(4) P4 2.4 - 1GB RAM w 20GB HD
(1 ) IBM R30 1.2 - 512MB RAM w 60GB HD
(1) Dell 8600 1.8 - 1GB RAM w 80GB HD

Will the IBM slow down the cluster?

On the topic of power consumption, my friend will trade me his laptops for the desktops. The desktop use about 1 amp each and the laptop only uses .40amp.

Also, what can I do to help?
Back to top
View user's profile Send private message 
gcmartin

Joined: 14 Oct 2005
Posts: 4238
Location: Earth

PostPosted: Wed 26 Jan 2011, 03:18    Post subject: Slow Cluster Master  

whatupuppy wrote:
Will the IBM slow down the cluster?
Short answer, YES.

Explanation you can measure yourself: Your workers will NEVER achieve 100% utilitzation on HPC problems and large workloads. Your Master will bottleneck in high workload environments.
whatupuppy wrote:
Also, what can I do to help?
Why not still use the IBM, but make it a worker. His SCSI drives should be a benefit to certain types of workloads. Its a Win-Win!
2 Questions:
  1. I'm a big IBM fan, but for this requirement, why do you want to make the IBM your master?
  2. What kinds of work will you have your cluster doing?

whatupuppy wrote:

My friend said he can lend me 6 - 1 gigabit network cards and a 1 - 24 ports 10gb router. I think that should help with communications between the computers.
This means you are already looking at a very good solution. The faster those cards, the better your cluster can move work and talk to each other.

Also, in your future, consider a SAN with 6 cables to your Cluster heads using 10GB cards to each cluster and you now would be entering the realm of "SuperHuman" HPCs. Couple with Parallel I/O filesystem, and you should be able to fly ... so to speak.

Hope this helps.
P.S. Don't forget my 2 questions, please. I have a personal application I'd like to run in your cluster once you're all set up. It should take about 1 cluster minute.

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile

Last edited by gcmartin on Wed 26 Jan 2011, 03:43; edited 6 times in total
Back to top
View user's profile Send private message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Wed 26 Jan 2011, 03:21    Post subject:  

Quote:

I will have by the weekend:
(4) P4 2.4 - 1GB RAM w 20GB HD
(1 ) IBM R30 1.2 - 512MB RAM w 60GB HD
(1) Dell 8600 1.8 - 1GB RAM w 80GB HD

Will the IBM slow down the cluster?


yes, but its nice to experiment, try the flop.c program

Quote:

Also, what can I do to help?



you can try blender ? with program mpir which i download from http://svat.sourceforge.net/
btw try use mpicc to compile the program

or would you compute chemical analisis...
http://classic.chem.msu.su/gran/gamess/

otherwise you can post your experiment on youtube or something, promote puppy Laughing Laughing Laughing

@gcmartin

thank you for the info... btw i also heard that if we really serious with cluster we also need to patch the kernel to maximize network speed... whatever it mean.
btw here is some info about parallel fs
http://blogs.sun.com/hstsao/entry/parallel_fs_for_cluster
mpir.c.zip
Description 
zip

 Download 
Filename  mpir.c.zip 
Filesize  5.89 KB 
Downloaded  133 Time(s) 

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
acrocosm


Joined: 12 Sep 2010
Posts: 53

PostPosted: Wed 26 Jan 2011, 12:56    Post subject:  

Master_wrong wrote:

please try
grep -c ^processor /proc/cpuinfo

Gives 1

Master_wrong wrote:

compared to
grep -c processor /proc/cpuinfo

Gives 2

What exactly does that mean?

gcmartin wrote:
This is NOT an effort to hi-jack this thread.

How is helping and sharing knowledge hi-jacking? On the contrary... I'd consider myself with my limited knowledge and stupid questions more of a hi-jacker. In any case though I'd like to learn more and help if I can, it's an interesting project. So I'll stay around and occasionally bother you guys ^______^


Very interesting stuff. Personally I'm still just scratching the surface though. According to this shared RAM is out of the question. Is that right? Also apparently this won't actually make it act as one multiprocessor machine exactly, I'll still have to manually split the work somehow whether that's based on frames (explicitly for animation) or rows/blocks. hmmm, hmmm.

The machines I currently have available here are:
1 - Custom build Quadcore Q6600, 2GB DDR2 (master)
2 - X40 with 1.2 Pentium M, 1GB DDR (slave)
3 - Custom build Pentium 4 at 2.8 with HT, 1GB DDR (not joined yet)

There's a Pentium 4@2.4 and one more at 2.66, if I remember correctly, at my parents place, both with 512 DDR but those are in another city since my flat is too small to stock machines. If everything goes well I'll take them too.

Don't laugh, but I'm currently connecting them through wifi. I know it's bad performance wise, but I'm just testing and learning atm so it doesn't matter at all. That's why I don't care to add the other spare joined.

Anyway, maybe that's obvious for you guys but I run blender on the master off this 2-machine cluster just to see if it could see beyond the 4 cpus. It didn't but maybe it's the internal rendered that doesn't support mpi(?). I'll try yafaray or luxblend next and that svat thing, although it seems a bit scary ^_^
Back to top
View user's profile Send private message Visit poster's website 
gcmartin

Joined: 14 Oct 2005
Posts: 4238
Location: Earth

PostPosted: Wed 26 Jan 2011, 14:17    Post subject: Personal view  

One way I view a cluster is as follows:
let's say I have 5 machines and one of them is the Master ("the entrance desk for the cluster")

I look at this as having 4 physically hungry dogs salivating, waiting, and looking for work. And you have this master whose job it is to throw the meat into the bin for the dogs to feed on. And, at the same time, the Master himself is salivating, looking to feed.

That's my idea of a cluster. Jobs go in and out thru the Master, while the dogs devour the feedings (work) and letting the Master know they're hungry again. Bigger the dog, less for the other guys.

IBM makes the baddest of the clusters. Univ. of Virginia has a Largest Mac Cluster in world. etc. There are clusters all over the world; large and small....just looks like another LAN machine.

A home cluster could let you do all sorts of things while on your home LAN. And there are Public Projects available that you can participate in which addresses all kinds of world problems from SETI to Space Station to Hubble to Water Shortages to Weather to monitoring community Biological Carbon Units activity. And can be used in Theory and Games. Wide Open!

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send private message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Wed 26 Jan 2011, 22:48    Post subject:  

Quote:

What exactly does that mean?

that mean that we have to change the script to correct the error...

here try this script...
what is changed :
- fix counting the processor for some older computer
- makes really-really automatic ssh (no more typing yes answer)

<script moved to first post>

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199

Last edited by Master_wrong on Tue 01 Feb 2011, 07:04; edited 1 time in total
Back to top
View user's profile Send private message 
acrocosm


Joined: 12 Sep 2010
Posts: 53

PostPosted: Fri 28 Jan 2011, 11:51    Post subject:  

Master_wrong wrote:
Quote:

What exactly does that mean?

that mean that we have to change the script to correct the error...

here try this script...
what is changed :
- fix counting the processor for some older computer
- makes really-really automatic ssh (no more typing yes answer)


I'm afraid I still get the same results
Back to top
View user's profile Send private message Visit poster's website 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Fri 28 Jan 2011, 12:13    Post subject:  

Sorry,
here try this one

this one, the counting process is

process=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $i "grep -c ^processor /proc/cpuinfo"`

which from your answer should do the trick

@gcmartin...

I have found the nfs...
i will test them tommorow
acrocosm.sh.zip
Description 
zip

 Download 
Filename  acrocosm.sh.zip 
Filesize  1.1 KB 
Downloaded  245 Time(s) 

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
gcmartin

Joined: 14 Oct 2005
Posts: 4238
Location: Earth

PostPosted: Fri 28 Jan 2011, 14:37    Post subject: Practical use of cluster  

One problem 99% of all people will have with Clusters: "Give me a practical application that I can identify with". They cannot understand how to apply it.

Here's an application that should make it easy to understand. Here ONE method of applying it done by a vendor. I am NOT pushing their product; I am showing an example of what we should be looking and thinking here in Puppyland. Review this This is a very practical use.

This should now make it easier to understand and begin applying Clusters. Also, this is the basis of what Citrix does thanks to their modeling after IBM did this in the 90's.

Now what applications can we replicate or do in Puppyland Clusters...hmmm?
P.S. It occurs that if we could replicate something like this in a Puppy Cluster, we could run every PUP, every PET, every SFS, and everything in the cluster. Now, hmmmm...this is some real food for those Workers to feed on from the Master. Big Dogs versus Small Dogs rules will still apply, though. But, the work will be spread in a very interesting manner.

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile

Last edited by gcmartin on Fri 28 Jan 2011, 16:58; edited 1 time in total
Back to top
View user's profile Send private message 
acrocosm


Joined: 12 Sep 2010
Posts: 53

PostPosted: Fri 28 Jan 2011, 16:46    Post subject:  

Master_Wrong I'm sorry, I have to clarify...

The original scripts included on the iso would show 2 cpu for the x40 on the server. Both of your later auto-config scripts worked fine. The server would see just one cpu on the x40 as expected

However,
Code:
grep -c processor /proc/cpuinfo
still gives 2 as a result.
Back to top
View user's profile Send private message Visit poster's website 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Fri 28 Jan 2011, 22:51    Post subject:  

Quote:

However,
Code:
grep -c processor /proc/cpuinfo
still gives 2 as a result.


so there is no problem. because grep -c processor /proc/cpuinfo incorrectly count processor, your /proc/cpuinfo might contain word processor twice which resulted in wrong count. this is used in the first script.

about nfs...
check

http://murga-linux.com/puppy/viewtopic.php?t=45090

its working but we can only mount /mnt/sda, we only need to install nfs_common.tar.gz

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 2 of 5 [72 Posts]   Goto page: Previous 1, 2, 3, 4, 5 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Puppy Projects
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1221s ][ Queries: 13 (0.0100s) ][ GZIP on ]