Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Sun 20 Apr 2014, 20:32
All times are UTC - 4
 Forum index » Advanced Topics » Puppy Projects
Puppy 245mpi cluster made easy
Moderators: Flash, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 5 [72 Posts]   Goto page: 1, 2, 3, 4, 5 Next
Author Message
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Thu 06 Jan 2011, 21:51    Post subject:  Puppy 245mpi cluster made easy  

-----------------
edit march-18-2011 -Howto tutorial
there is a complete installation manual available at "ClusterPUP - A Very Simple Cluster Implementation" -
https://docs.google.com/document/d/1jUKsxfkjwmU69vABOY1slSk5E9BmsobGaJ3nZj4vrKk/edit?hl=en&authkey=CNDM24ML&pli=1#

-----------------------------

edit feb 27-2011 -newest puplet here----> with pxe, auto detect and setting

http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199

--------------------



http://hotfile.com/dl/94559819/a465c13/puppy-cluster.zip.html

this is my first puplet....
all it does is simplify setting of mpich cluster... and made it as easy as 123...

Edit-feb-1-2011:
attached below is new config-script
1. improve key-less ssh
2. fix counting processor
3. nfs script (need to install nfs_common.tar.gz from http://murga-linux.com/puppy/viewtopic.php?t=45090
both server and client will be able to access /mnt/my. server need to mount sda1)

no more installing stuff into your desktop pc , al you need to have is cd and no need to setting ssh password-less and mpich which complicated for most ppl.
bottom line... no more headache for trying cluster computer....

see picture
1. basic setting, time date, hostname, nothing to edit here, my program will automatically set the hostname. press ok
2. Network wizard, select your network setting accordingly
3. If connection to network succed, a question will pop up, leave them if you are not on server pc
4. If you are on server pc then click yes and enter the list ip of all pc, put server ip on top, save and exit.
5. First time ssh will ask question, answer yes
6. Run mpd, compile program c, then run it, what was wrong ?... forgot to distribute the compiled program... use cluster-ssh-fileshare.sh
7. Program work after compiled program are distributed across cluster pc

I you want to know more about clustering....
http://www.phy.duke.edu/~rgb/Beowulf/beowulf_book/beowulf_book/index.html

also here is pdf to help using mpich2
http://www.fz-juelich.de/jsc/datapool/page/4224//install.pdf

Thank to gcmartin for the picture:
https://docs.google.com/drawings/edit?id=16jMd2gmt5YmmOlLWmb1q7fhufv4m0qKoKCZQJ348J54&hl=en&pli=1

have a nice day
config-script.tar.gz
Description 
gz

 Download 
Filename  config-script.tar.gz 
Filesize  1.64 KB 
Downloaded  391 Time(s) 
1.png
Description 
png

 Download 
Filename  1.png 
Filesize  304.08 KB 
Downloaded  683 Time(s) 
2.png
Description 
png

 Download 
Filename  2.png 
Filesize  322.35 KB 
Downloaded  540 Time(s) 
3.png
Description 
png

 Download 
Filename  3.png 
Filesize  310.7 KB 
Downloaded  486 Time(s) 
4.png
Description 
png

 Download 
Filename  4.png 
Filesize  190.3 KB 
Downloaded  499 Time(s) 
5.png
Description 
png

 Download 
Filename  5.png 
Filesize  304.5 KB 
Downloaded  491 Time(s) 
6.png
Description 
png

 Download 
Filename  6.png 
Filesize  164.06 KB 
Downloaded  480 Time(s) 
7.png
Description 
png

 Download 
Filename  7.png 
Filesize  162.6 KB 
Downloaded  530 Time(s) 

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199

Last edited by Master_wrong on Wed 27 Apr 2011, 21:42; edited 13 times in total
Back to top
View user's profile Send private message 
puppyiso


Joined: 12 Jan 2009
Posts: 595

PostPosted: Thu 06 Jan 2011, 22:08    Post subject:  

Can you provide more detailed use manual?

Most importantly how to configure the network and the internet?

CD only or can be installed to USB?

John
Back to top
View user's profile Send private message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Fri 07 Jan 2011, 06:43    Post subject:  

Quote:

Can you provide more detailed use manual?

done

Quote:

Most importantly how to configure the network and the internet?

the configuration to passwordless ssh and mpich are automatic.
about internet... this puplet are not meant to connect to internet, the main purpose is for study cluster computer and research.

Quote:
CD only or can be installed to USB?

yes you can install on cd and usb...
but remember then the size is 294M, so for pc with 512M or lower ram i recommend to use cd, its cheaper than usb.

if all pc has 1 G ram, you can use usb and boot all using just that 1 usb.

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Wed 12 Jan 2011, 11:43    Post subject: My puppy has more flop than yours  

I add interesting program....


flop.c ----> to count how many flop your cluster pc has to compare with those super computer.

now i can say that my puppy has more flop than yours Laughing Laughing Laughing Laughing Laughing
flop.c.zip
Description 
zip

 Download 
Filename  flop.c.zip 
Filesize  925 Bytes 
Downloaded  217 Time(s) 
flop.jpg
 Description   
 Filesize   96.18 KB
 Viewed   4595 Time(s)

flop.jpg


_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
gcmartin

Joined: 14 Oct 2005
Posts: 3634
Location: Earth

PostPosted: Wed 12 Jan 2011, 13:40    Post subject: "Masterful" effort...Thanks  

I run a LiveCD environment. So, for clustering several PCs to accomplish workload under a single-headed "Master" is an accomplishment for certain types of workloads. I applaud your accomplishment as this is significant.

I'm new to PC clustering. I have couple of questions? (assume a 3 PC running cluster)
  • Is your work for 32bit or 64bit PCs?
  • Does your current work require all of the cluster machines to be same?
  • To dispatch work, how do the cluster machines know when to pitch-in (dynamic work allocation or static)?
  • Assuming that the master has 2 LAN cards (1 to cluster switch-other to normal LAN)Can the master run a full Puppy desktop configuration as it handles workload management to those in the cluster?
This appears, to me, to be a very significant breakthrough in the Puppy world.
Thanks in advance

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send private message 
gcmartin

Joined: 14 Oct 2005
Posts: 3634
Location: Earth

PostPosted: Thu 13 Jan 2011, 00:23    Post subject: Re: My puppy has more flop than yours  

@Master ...
Master_wrong wrote:
I add interesting program....flop.c ---> to count how many flop your cluster pc has to compare with those super computer. ...
Can you help by sharing how to interpret the data shown? Trying to understand what I'm seeing in terms of process/job dispatch and what is in your cluster. Please explain.

Thanks in advance

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send private message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Thu 13 Jan 2011, 02:06    Post subject:  

Hi Gcmartin,

Laughing Laughing

I was confused with my finding myself....

the first data with -np 1 -----> is using 1 processor only so there puppy-1 700 Mflop
2nd data -np 2 -----> 2 processor, because its dual core both are puppy-1 14000 Mflop, exactly double

so the funny stuff begun when other cluster join in

3rd data -np 3 -----> use 3 processor so it use puppy-1 and puppy-2 the drop to 590Mflop
4th data -np 4 -----> use 4 processor another drop to 400 Mflop, all processor on both computer are used

I assume because my destop(puppy-1) is newer than my laptop(puppy-2) this actually means that my laptop are too slow which hit the flop, but if we see the error then we see that clustering help minimize the rate of error.

Quote:

I run a LiveCD environment. So, for clustering several PCs to accomplish workload under a single-headed "Master" is an accomplishment for certain types of workloads. I applaud your accomplishment as this is significant.


thank you

Quote:
* Is your work for 32bit or 64bit PCs?
* Does your current work require all of the cluster machines to be same?
* To dispatch work, how do the cluster machines know when to pitch-in (dynamic work allocation or static)?
* Assuming that the master has 2 LAN cards (1 to cluster switch-other to normal LAN)Can the master run a full Puppy desktop configuration as it handles workload management to those in the cluster?


- i believe it was 32 bit.
- seen from my experiment then it is
- mpich will dispatch the workload among cluster, because mpich also installed in the cluster machine and has all program needed to run.
- i believe that if we have enough processor power to handle and we set the setting correctly, ie: we have dualcore so use 1 for puppy and use 1 for cluster work... i think it will work.

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
gcmartin

Joined: 14 Oct 2005
Posts: 3634
Location: Earth

PostPosted: Thu 13 Jan 2011, 13:48    Post subject: Unexplained behavior  

Master...
I hope this makes sense. When I run an OS aware of multi-core processors, the hardware, under OS direction sometimes, is managing what one can consider to be "clustering".

When we move from local hardware clustering into "soft" clustering, we introduce a "new" bus that is used in communication of process controls, where it is significantly slower than the local multi-core bus. This should impact the behavior of some applications. So, application design is a factor depending upon what the objectives are to be.

In this case, we are seeing how "this particular program, only," behaves in your cluster. Another program with different design objectives in its design would be able to exploit you current cluster in a different fashion such that you would meet a different result. Further, there MAY be some cluster OS controls which, too, MAY affect /enhance program's performance in soft cluster.

Hope this helps.
P.S, My question on Puppy was to try to determine if you felt that a X-desktop console would contribute to the overall issues that would benefit cluster management, use, and performance.

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send private message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Fri 14 Jan 2011, 14:25    Post subject:  

I try to compute how much flop does my laptop has....

it s just 200...
so imagine this pararel configuration while 2 other has 700 and my laptop has 200 so what happen is those faster processor just wait idle for the laptop to finish his work and dropping the flop count in the process.

it also makes senses too that
200x3 = 600 equal to the result of -np 3 , because workload is split by 3 processor
200x2 = 400 equal to the result of -np 4, because workload is split by 2 computer

btw my laptop seem just single processor, puppy see it as single processor and mpi treated it as single processor.

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
whatupuppy

Joined: 17 Jan 2011
Posts: 17

PostPosted: Mon 17 Jan 2011, 22:10    Post subject:  

Can you let me know your build process for puppy-cluster?

I'm waiting for a couple of computer and will test your puppy-cluster build.

Can not wait.
Back to top
View user's profile Send private message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Fri 21 Jan 2011, 21:58    Post subject:  

Quote:
Can you let me know your build process for puppy-cluster?


I already post all picture that needed to run the cluster.
all you need is a working network that connect those computer.

if you ask how i build.....
http://www.murga-linux.com/puppy/viewtopic.php?t=63177&start=15


Quote:

I'm waiting for a couple of computer and will test your puppy-cluster build.


wow you are building cluster ??? Laughing
were all the computer identical ? it would be best if they were identical.


btw found few great software too
http://freshmeat.net/search?page=12&q=cluster&submit=Search&with=&without=


some of them....
Quote:
NetCrack is cluster software developed to distribute a hashing algorithm's cracking process work using a brute force attack.


offcourse i havent try any of them them

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
acrocosm


Joined: 12 Sep 2010
Posts: 53

PostPosted: Sun 23 Jan 2011, 20:25    Post subject:  

Hmm so considering I have this properly set up and running on 3-4 machines, can I run blender to render a single image on all of them without having to configure anything else?

Or i need to run blender with the same scene on each of them and from the master machine I'll say: slave 1 render this layer, slave 2 render that layer and so on?
Back to top
View user's profile Send private message Visit poster's website 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Mon 24 Jan 2011, 13:09    Post subject:  

you need a specially patched program.

but maybe i was wrong....


try.... http://svat.sourceforge.net/

they use blender...
Quote:

The first example renders 2,000 frames with Blender on a 512 CPU computer. Notice how mpir uses a command-line template. You specify a frame range length with the '-g' option. Progress is echoed as a table.

# Start the multi-cpu rendering.
mpirun -np $NSLOTS mpir -S 15 -s 1 -e 2000 -g 15 'blender -b scene.blend -o /tmp/ -F PNG -x 1 -t 1 -s %start -e %end -a'



Quote:

* mpir - This is a generic server/client MPI program to delegate independent tasks to many processors. This makes it easy to use graphics tools on MPI-centric hosts. See examples of its use are in Section 4, such as batch rendering or batch image post-processing.

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
acrocosm


Joined: 12 Sep 2010
Posts: 53

PostPosted: Mon 24 Jan 2011, 13:39    Post subject:  

Thanks for the link and of course thank you for your work!! I'll study that site and see what I can figure out.

Also I wanted to ask... is this vanilla luci 245 with a special sfs for clustering? Can I use the sfs on 5.2 or any other pupplet as long as the dependencies are satisfied?


Ultimately what I'd like to do is use the cannibalized spare machines I found on the streets or from friends to build a mini renderfarm for me and a friend. But I'm not so clear about distributed, grid, clustering and so on. I'd be obliged if you could clarify...

I'd like to run those machines diskless and if possible monitorless. Because I have 3-4 machines on the same room I don't see any reason in setting up a network where I run blender with the same scene and from one of those I tell machine2 render frames 1-100, machine3 render 101-200 etc. I can do that on each of them manually, they are next to another. Or I could use VNC to remotely control them and start the rendering.

What I'd like is all of them to act as one, shared memory, shared cpus. I don't expect shared graphics accelerators but it would be a nice touch. Is that possible? Is that a beowulf? Can I do that with this pupplet?

Sorry if I'm such a noob ^_^ and thank you again for your work. Using super fast and light puppy for clustering is such a brilliant idea!
Back to top
View user's profile Send private message Visit poster's website 
acrocosm


Joined: 12 Sep 2010
Posts: 53

PostPosted: Mon 24 Jan 2011, 14:32    Post subject:  

Ok I'm setting this up on 2 machines to check if blender can see all CPUs. I got something odd...

Code:
inet addr:127.0.0.1 Mask:255.0.0.0 inet addr:192.168.1.77 Bcast:192.168.1.255 Mask:255.255.255.0
confirm SERVER commencing...
1
computer: 192.168.1.77 has:4  processor
2
computer: 192.168.1.66 has:2  processor
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet addr:192.168.1.77 
`/root/my-applications/clusterdata/etc/hosts' -> `/etc/hosts'
`/root/my-applications/clusterdata/etc/mpd.conf' -> `/etc/mpd.conf'
`/root/my-applications/clusterdata/home/mpd.hosts' -> `/root/mpd.hosts'
`/root/my-applications/clusterdata/home/.mpd.conf' -> `/root/.mpd.conf'
.mpd.conf                                     100%   20     0.0KB/s   00:00   
mpd.hosts                                     100%   30     0.0KB/s   00:00   
mpd.conf                                      100%   20     0.0KB/s   00:00   
hosts                                         100%   62     0.1KB/s   00:00   
.mpd.conf                                     100%   20     0.0KB/s   00:00   
mpd.hosts                                     100%   30     0.0KB/s   00:00   
mpd.conf                                      100%   20     0.0KB/s   00:00   
hosts                                         100%   62     0.1KB/s   00:00


Check computer 2, it is reported as having 2 proccessors. That's not possible. This machine is a Thinkpad x40 with a 1.2 pentium M. Dual cores didn't even exist back then! I'm puzzled... o.O
Back to top
View user's profile Send private message Visit poster's website 
Display posts from previous:   Sort by:   
Page 1 of 5 [72 Posts]   Goto page: 1, 2, 3, 4, 5 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Puppy Projects
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0970s ][ Queries: 13 (0.0058s) ][ GZIP on ]