Puppy 245mpi cluster made easy

A home for all kinds of Puppy related projects
Message
Author
jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#31 Post by jpeps »

acrocosm wrote:Master_Wrong I'm sorry, I have to clarify...

The original scripts included on the iso would show 2 cpu for the x40 on the server. Both of your later auto-config scripts worked fine. The server would see just one cpu on the x40 as expected

However,

Code: Select all

grep -c processor /proc/cpuinfo
still gives 2 as a result.
grep -c ^processor

gcmartin

Cluster applications

#32 Post by gcmartin »

IFF you are wondering how others have applied clusters, other than the one's I've mentioned earlier, look here for ways to use yours!
Hope this helps

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#33 Post by Master_wrong »

@gcmartin
P.S. It occurs that if we could replicate something like this in a Puppy Cluster, we could run every PUP, every PET, every SFS, and everything in the cluster. Now, hmmmm...this is some real food for those Workers to feed on from the Master. Big Dogs versus Small Dogs rules will still apply, though. But, the work will be spread in a very interesting manner.
I dont think that using cluster will benefit all program, such as office suite etc they dont need that much processing power...
i believe that those computer that waiting idle in screensaver mode would be more useful if used as cluster. but not all program need cluster processing power.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

jpeps
Posts: 3179
Joined: Sat 31 May 2008, 19:00

#34 Post by jpeps »

Master_wrong wrote:
I dont think that using cluster will benefit all program, such as office suite etc they dont need that much processing power
Seems like office suite, etc., would need unique software to make use of clustering (i.e, that can run parallel tasks); doesn't make much sense for basic apps.

gcmartin

Mistaken understanding

#35 Post by gcmartin »

Master_wrong wrote:
P.S. It occurs that if we could replicate something like this in a Puppy Cluster ... .
I dont think that using cluster will benefit all program, such as office suite etc they dont need that much processing power...
i believe that those computer that waiting idle in screensaver mode would be more useful if used as cluster. but not all program need cluster processing power.
Thanks Master_wrong (and @jpeps too). I agree with you that today's Office Suite is NOT an application for clustering (and I don't believe I suggested it either).

The example I provided was one where someone has found a way to dispatch VMs as work in a cluster. That's novel, if it works as they suggest and is a greater way to throw workers at workload. I only threw that our way in Puppyland to inspire ideas for what can be accomplished in a Puppy cluster.

Hope this helps

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#36 Post by Master_wrong »

Thanks Master_wrong (and @jpeps too). I agree with you that today's Office Suite is NOT an application for clustering (and I don't believe I suggested it either).
well...
you did said
we could run every PUP, every PET, every SFS,
,
which i reply
I dont think that using cluster will benefit all program, such as office suite etc
sorry if i make it sound you were suggesting it but office suite does packaged as pet and sfs.

btw if we see from this picture
http://pve.proxmox.com/wiki/File:Pve-vi ... chines.png

its not virtual, it seems real pc, however they they run virtual macine on each pc node.
so its not that we run 1 virtual machine on server that use cluster processors.

I already find software that monitor each node, but its not as polished as theirs.
Description:
------------
Gossimon is a gossip based distributed monitoring system for a cluster of
Linux nodes. Each node in the cluster periodically send information about
itself and others to a randomly selected node. This way each node constantly
receive information about cluster nodes. This information is locally
maintained (constantly updated) by each node and can be used by various
clients.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#37 Post by Master_wrong »

This is my code to enable nfs,
192.168.2.130 is server
192.168.2.118 is client

http://murga-linux.com/puppy/viewtopic.php?t=45090

we only need to install nfs_common.tar.gz from above site, in the end both server and client will be able to access /mnt/my.


(already included at the 1st post)

####NFS ---need nfs_common.tar.gz only
### change drive="sda1" to your mounted device
## please create ~/my-applications/clusterdata/ip-list which contain list of ip
#need passwordless ssh

drive="sda1"
mm=`head -n 1 ~/my-applications/clusterdata/ip-list`
m=`echo ${mm/ /}`
echo "M="$m

rm /etc/exports
echo "/mnt/$drive *(rw,no_root_squash,no_subtree_check,sync)" >>/etc/exports
modprobe nfsd
/etc/rc.d/rc.nfsd start
x="mount-FULL -t nfs $m:/mnt/$drive /mnt/my "
echo "x="$x
for i in `cut --delimiter=: -f1 ~/my-applications/clusterdata/ip-list`;do ssh $i "mkdir /mnt/my"; ssh $i "modprobe nfs";ssh $i "/etc/rc.d/rc.portmap start";ssh $i $x;done
####NFS
Last edited by Master_wrong on Tue 01 Feb 2011, 11:14, edited 1 time in total.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#38 Post by Master_wrong »

Although i did not agree gcmartin vision about possibility running all program using cluster, i make some research and put them here, i hope someone could use them...
from http://www.open-mpi.org/faq/?category=running
12. Can I run non-MPI programs with mpirun / mpiexec?
Yes.

Indeed, Open MPI's mpirun and mpiexec are actually synonyms for our underlying launcher named orterun (i.e., the Open Run-Time Environment layer in Open MPI, or ORTE). So you can use mpirun and mpiexec to launch any application. For example:

shell$ mpirun -np 2 --host a,b uptime

This will launch a copy of the unix command uptime on the hosts a and b.
Other questions in the FAQ section deal with the specifics of the mpirun command line interface; suffice it to say that it works equally well for MPI and non-MPI applications.
i know its open-mpi, but it s the same implementation, we can try...
we also have mpir..
or we can just run ssh to each computer and run the program
like these

# for i in `cut --delimiter=: -f1 ~/mpd.hosts`;do ssh $i "ls /mnt/my/$i";done

if we need to do blender, we can use nfs then create folder on master node with the name of the ip.
then set to run blender using the folder.

so the folder on master node will will look like these
/mnt/my/192.168.2.130/
/mnt/my/192.168.2.118/
/mnt/my/192.168.2.122/
the picture that needed to blended and the result will be saved into that folder, so each folder will contain different picture.

from http://wiki.blender.org/index.php/Doc:M ... ne_Options
In some situations we want to increase the render speed, access blender remotely to render something or build scripts that use blender command line.

One advantage of using command line is that we don't need the X server (in case of Linux) and as a consequence we can render remotely by SSH or telnet.

next...
run python program on mpi
http://sourceforge.net/projects/mpy/fil ... z/download
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

gcmartin

Mistaken understanding

#39 Post by gcmartin »

Master_wrong wrote:
Thanks Master_wrong (and @jpeps too). I agree with you that today's Office Suite is NOT an application for clustering (and I don't believe I suggested it either).
well...
you did said
we could run every PUP, every PET, every SFS,
, which i reply
I dont think that using cluster will benefit all program, such as office suite etc
sorry if i make it sound you were suggesting it but office suite does packaged as pet and sfs.

btw if we see from this picture
http://pve.proxmox.com/wiki/File:Pve-vi ... chines.png

its not virtual, it seems real pc, however they they run virtual macine on each pc node. so its not that we run 1 virtual machine on server that use cluster processors.

I already find software that monitor each node, but its not as polished as theirs.
Yeah, I can see how this was misleading. Newspapers are constantly taking a quote out of context. When I made that quote (look at the whole passage), I was making it meaning to have whole OS's run and their services being delivered from the internal OS's. So, for example, if I dispatched a OS as a cluster application, whatever internal services that OS provides could be available via that OS....not the cluster directly...internal from the OS. In the case of Puppy (an OS), I would have services (PET/SFS) within it available.

On the item of PROXMOX, I agree without lifting the covers on how they achieved their implementation, it might not be a cluster; but it they are, in fact, clustering, then they have an implementation where we can draw upon for understanding and for future Puppy implementations.

Here's the kind of idea that one can draw: Physical Cluster with Virtual machines running with some virtual Machines actually being a cluster with other virtual machines with ... I think everyone can see where this leads to... you know physical to virtual to virtual to... alongside other "real" individual cluster applications being globbled up by physical and virtual cluster machines.

Anyway, you have made a very promising contribution to the Puppy community thus far. And I applaud your achievement!

Hope this helps

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#40 Post by Master_wrong »

Yeah, I can see how this was misleading. Newspapers are constantly taking a quote out of context. When I made that quote (look at the whole passage), I was making it meaning to have whole OS's run and their services being delivered from the internal OS's. So, for example, if I dispatched a OS as a cluster application, whatever internal services that OS provides could be available via that OS....not the cluster directly...internal from the OS. In the case of Puppy (an OS), I would have services (PET/SFS) within it available.
:oops:
i guess.. forgive me for being me.... ?

but i guess that would be too difficult to implement. then again i could be me ? :lol:

btw i just read this...
What is a beowulf cluster or Beowulf cluster vs Supercomputer

Beowulf cluster is essentially a number of commodity or of the
shelf PCs tied together using some sort of protocol. Developed
by NASA, Beowulf clusters gained much attention in the last few
years. Often at a fraction of a cost of a "real" supercomputer
a beowulf cluster is comparable (or even exceeds) the
computational power of a super computer. For instance the a
cluster made out of 320 ubiquitous P3-800 CPUs takes 34th place
on the TOP500 super computer list with peak performance of over
a teraflop!
ok has anyone have 320 p3 computer to test ? :roll:
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

whatupuppy
Posts: 17
Joined: Tue 18 Jan 2011, 02:07

#41 Post by whatupuppy »

Update:
Well things did not go well with my friend this weekend.

So this have to be put on hold for now.

Also, failing hard drive on old laptops causing puppy to hang at Uncompressing linux..OK.booting Linux kernel.

Any suggestion beside buying a new harddrive is welcomed.

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#42 Post by Master_wrong »

I was about to suggest buying flashdisk....

so there is no cdrom, i guess ?
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#43 Post by Master_wrong »

i spend last night testing like couple dozens of program....
it end up only couple of program that compiled successfully

here is 2 interesting program that might be useful,
1. tentakel :
Tentakel is a program for executing the same command on many hosts in parallel
using various remote methods. It can make use of several sets of hosts that are
defined in a configuration file as groups.

It also supports an interactive mode that can be used for repeated commands.

The author uses tentakel to simultaneously install software on many
workstations or ask and set parameters on a linux compute cluster. With the
power of format strings tentakel can also be used for monitoring purposes.

Supported remote methods are ssh(1) and rsh(1). Both must be configured to
allow for password-less logins. Password-protected keyfiles for ssh can be
used with ssh-agent(1).

A plugin mechanism allows users to implement their own remote methods in
addition to the builtin ones.

For more information on available options please refer to the manpage
tentakel(1).



The project homepage is: http://tentakel.biskalar.de/
too bad i cannot get anything to work, i simply not smart enough to create config file, if anyone capable please let me know





2. unfs3-0.9.22.tar.gz :Great NFS for puppy, it capable of export and mount /root

INTRODUCTION
============

UNFS3 is a user-space implementation of the NFSv3 server
specification.

UNFS3 supports all NFSv3 procedures with the exception of the
READDIRPLUS procedure. It tries to provide as much information
to NFS clients as possible, within the limits possible from
user-space.

See the unfsd(8) manpage for restrictions imposed on NFS
operations (section RESTRICTIONS) and for possible races
with local file system activity (section BUGS).

It is not possible to export to netgroups or wildcard hostnames
via /etc/exports, all other addressing methods should work. The
following options are recognized in the exports file: ro, rw,
root_squash, all_squash, no_root_squash, no_all_squash. If
other options are present, they are ignored.

Cluster extensions compatible to the older ClusterNFS project
are supported when the source is configured with --enable-cluster.
all you need to do

1. create /etc/exports file: contain=====> /root (ro) --->for read only
open terminal at unfsd directrory
2. ./configure
3. make
4. make install
5. ./unfsd -s -d -p -n 4711 -m 4711 -e /etc/exports


6. at client : mkdir /mnt/try
7. copy unfsd,./configure,make,make install
8. mount -o port=4711,mountport=4711,mountvers=3,nfsvers=3,nolock,tcp 192.168.2.130:/root /mnt/try
Attachments
unfs3-test.png
(25.94 KiB) Downloaded 1110 times
tentakel.zip
(42.15 KiB) Downloaded 449 times
unfs3-0.9.22.tar.gz
(164.5 KiB) Downloaded 473 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#44 Post by Master_wrong »

made pet of unfs3 at http://www.murga-linux.com/puppy/viewtopic.php?t=64491

btw pxe booting work using
http://www.murga-linux.com/puppy/viewtopic.php?t=64107

also other program tested
pconsole does not work through daemons. The pconsole.sh script opens up a
number of connections to each node of your cluster of workstations. Then,
the pconsole binary attaches to the tty devices that these windows are
using, and it copies the input you type to all open connections......
.......
By default, pconsole tries to use SSH to make connections. If you do not have
'ssh', it will try to use 'telnet'.
Attachments
pconsole-test.png
(32.67 KiB) Downloaded 488 times
Last edited by Master_wrong on Sun 06 Feb 2011, 06:12, edited 2 times in total.

gcmartin

What is a cluster

#45 Post by gcmartin »

@Master _Wrong has done all of us a very valuable service. I am showing this picture here because over the past couple months, many who come here have an inaccurate perception of a cluster.

Image

This picture is worth a thousand words, as, it shows where to place the monitor, keyboard, and mouse for the cluster. In a true cluster, the nodes/workers/slave PCs DO NOT HAVE A MONITOR, KEYBOARD, OR MOUSE. But they all work together on behalf of the workload within the cluster.

Hope this helps
Attachments
Beowulf.png
(86.79 KiB) Downloaded 521 times
Last edited by gcmartin on Sat 05 Feb 2011, 18:28, edited 1 time in total.

gcmartin

#46 Post by gcmartin »

.

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#47 Post by Master_wrong »

Thank you gcmartin,

your effort in pxe documentation also help this project somehow.

btw here is some article about cluster:
http://www.clustermonkey.net//content/view/89/32/

the site also has ton of info about mpi
http://www.clustermonkey.net//content/view/14/28/
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
Aitch
Posts: 6518
Joined: Wed 04 Apr 2007, 15:57
Location: Chatham, Kent, UK

#48 Post by Aitch »

Wow, Master_wrong....you HAVE been busy, and got away from this thread...

http://www.murga-linux.com/puppy/viewtopic.php?t=34647

Since my initial interest, I've had a change in hardware direction, and now don't use much of my old scsi server boxes, as they use too much power, and make too much noise :wink:

However, I still think clustering is under-rated, especially in the puppyworld, with so much old hardware available

A couple or 3/more of useful links

http://www.linuxclusters.com/books/clus ... index.html

http://freshmeat.net/articles/linux-clustering-software

http://www.jumpingbean.co.za/linux-clus ... ailability

http://linuxreviews.org/howtos/virtual_ ... ter_howto/

http://www.ultramonkey.org/

http://www.gdargaud.net/Hack/ClusterNotes.html#Software

http://www.cs.wisc.edu/condor/

Good luck guys....some spaghetti examples

http://www.calvin.edu/~adams/research/m ... f/related/

Aitch :)

Master_wrong
Posts: 452
Joined: Thu 20 Mar 2008, 01:48

#49 Post by Master_wrong »

:D
Hi Aitch, welcome aboard...
Wow, Master_wrong....you HAVE been busy, and got away from this thread...
yes it quite complicated...
but it was fun, still one thing is missing... howto incorporate pxe into it, yes i can do it already but how to make it more simple ?

I'm thinking about IBS atm, eh no no not irritable bowel syndrome :lol:

here:
http://www.stsx.org/projects/ibs/ibs-manual.html
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]

User avatar
Aitch
Posts: 6518
Joined: Wed 04 Apr 2007, 15:57
Location: Chatham, Kent, UK

#50 Post by Aitch »

Try iPXE

http://ipxe.org/

I just saw it demo'd at a linux fair in London....it looks cool, but don't ask me how to combine it with what you've done...that's quite an achievement, already :D

btw, here's a tempting project to build to run it on

http://www.mini-itx.com/projects/cluster/?p

Aitch :)

Post Reply