grep -c ^processoracrocosm wrote:Master_Wrong I'm sorry, I have to clarify...
The original scripts included on the iso would show 2 cpu for the x40 on the server. Both of your later auto-config scripts worked fine. The server would see just one cpu on the x40 as expected
However,still gives 2 as a result.Code: Select all
grep -c processor /proc/cpuinfo
Puppy 245mpi cluster made easy
Cluster applications
IFF you are wondering how others have applied clusters, other than the one's I've mentioned earlier, look here for ways to use yours!
Hope this helps
Hope this helps
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
@gcmartin
i believe that those computer that waiting idle in screensaver mode would be more useful if used as cluster. but not all program need cluster processing power.
I dont think that using cluster will benefit all program, such as office suite etc they dont need that much processing power...P.S. It occurs that if we could replicate something like this in a Puppy Cluster, we could run every PUP, every PET, every SFS, and everything in the cluster. Now, hmmmm...this is some real food for those Workers to feed on from the Master. Big Dogs versus Small Dogs rules will still apply, though. But, the work will be spread in a very interesting manner.
i believe that those computer that waiting idle in screensaver mode would be more useful if used as cluster. but not all program need cluster processing power.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
Seems like office suite, etc., would need unique software to make use of clustering (i.e, that can run parallel tasks); doesn't make much sense for basic apps.Master_wrong wrote:
I dont think that using cluster will benefit all program, such as office suite etc they dont need that much processing power
Mistaken understanding
Thanks Master_wrong (and @jpeps too). I agree with you that today's Office Suite is NOT an application for clustering (and I don't believe I suggested it either).Master_wrong wrote:I dont think that using cluster will benefit all program, such as office suite etc they dont need that much processing power...P.S. It occurs that if we could replicate something like this in a Puppy Cluster ... .
i believe that those computer that waiting idle in screensaver mode would be more useful if used as cluster. but not all program need cluster processing power.
The example I provided was one where someone has found a way to dispatch VMs as work in a cluster. That's novel, if it works as they suggest and is a greater way to throw workers at workload. I only threw that our way in Puppyland to inspire ideas for what can be accomplished in a Puppy cluster.
Hope this helps
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
well...Thanks Master_wrong (and @jpeps too). I agree with you that today's Office Suite is NOT an application for clustering (and I don't believe I suggested it either).
you did said
,we could run every PUP, every PET, every SFS,
which i reply
sorry if i make it sound you were suggesting it but office suite does packaged as pet and sfs.I dont think that using cluster will benefit all program, such as office suite etc
btw if we see from this picture
http://pve.proxmox.com/wiki/File:Pve-vi ... chines.png
its not virtual, it seems real pc, however they they run virtual macine on each pc node.
so its not that we run 1 virtual machine on server that use cluster processors.
I already find software that monitor each node, but its not as polished as theirs.
Description:
------------
Gossimon is a gossip based distributed monitoring system for a cluster of
Linux nodes. Each node in the cluster periodically send information about
itself and others to a randomly selected node. This way each node constantly
receive information about cluster nodes. This information is locally
maintained (constantly updated) by each node and can be used by various
clients.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
This is my code to enable nfs,
192.168.2.130 is server
192.168.2.118 is client
http://murga-linux.com/puppy/viewtopic.php?t=45090
we only need to install nfs_common.tar.gz from above site, in the end both server and client will be able to access /mnt/my.
(already included at the 1st post)
####NFS ---need nfs_common.tar.gz only
### change drive="sda1" to your mounted device
## please create ~/my-applications/clusterdata/ip-list which contain list of ip
#need passwordless ssh
drive="sda1"
mm=`head -n 1 ~/my-applications/clusterdata/ip-list`
m=`echo ${mm/ /}`
echo "M="$m
rm /etc/exports
echo "/mnt/$drive *(rw,no_root_squash,no_subtree_check,sync)" >>/etc/exports
modprobe nfsd
/etc/rc.d/rc.nfsd start
x="mount-FULL -t nfs $m:/mnt/$drive /mnt/my "
echo "x="$x
for i in `cut --delimiter=: -f1 ~/my-applications/clusterdata/ip-list`;do ssh $i "mkdir /mnt/my"; ssh $i "modprobe nfs";ssh $i "/etc/rc.d/rc.portmap start";ssh $i $x;done
####NFS
192.168.2.130 is server
192.168.2.118 is client
http://murga-linux.com/puppy/viewtopic.php?t=45090
we only need to install nfs_common.tar.gz from above site, in the end both server and client will be able to access /mnt/my.
(already included at the 1st post)
####NFS ---need nfs_common.tar.gz only
### change drive="sda1" to your mounted device
## please create ~/my-applications/clusterdata/ip-list which contain list of ip
#need passwordless ssh
drive="sda1"
mm=`head -n 1 ~/my-applications/clusterdata/ip-list`
m=`echo ${mm/ /}`
echo "M="$m
rm /etc/exports
echo "/mnt/$drive *(rw,no_root_squash,no_subtree_check,sync)" >>/etc/exports
modprobe nfsd
/etc/rc.d/rc.nfsd start
x="mount-FULL -t nfs $m:/mnt/$drive /mnt/my "
echo "x="$x
for i in `cut --delimiter=: -f1 ~/my-applications/clusterdata/ip-list`;do ssh $i "mkdir /mnt/my"; ssh $i "modprobe nfs";ssh $i "/etc/rc.d/rc.portmap start";ssh $i $x;done
####NFS
Last edited by Master_wrong on Tue 01 Feb 2011, 11:14, edited 1 time in total.
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
Although i did not agree gcmartin vision about possibility running all program using cluster, i make some research and put them here, i hope someone could use them...
from http://www.open-mpi.org/faq/?category=running
we also have mpir..
or we can just run ssh to each computer and run the program
like these
# for i in `cut --delimiter=: -f1 ~/mpd.hosts`;do ssh $i "ls /mnt/my/$i";done
if we need to do blender, we can use nfs then create folder on master node with the name of the ip.
then set to run blender using the folder.
so the folder on master node will will look like these
/mnt/my/192.168.2.130/
/mnt/my/192.168.2.118/
/mnt/my/192.168.2.122/
the picture that needed to blended and the result will be saved into that folder, so each folder will contain different picture.
from http://wiki.blender.org/index.php/Doc:M ... ne_Options
next...
run python program on mpi
http://sourceforge.net/projects/mpy/fil ... z/download
from http://www.open-mpi.org/faq/?category=running
i know its open-mpi, but it s the same implementation, we can try...12. Can I run non-MPI programs with mpirun / mpiexec?
Yes.
Indeed, Open MPI's mpirun and mpiexec are actually synonyms for our underlying launcher named orterun (i.e., the Open Run-Time Environment layer in Open MPI, or ORTE). So you can use mpirun and mpiexec to launch any application. For example:
shell$ mpirun -np 2 --host a,b uptime
This will launch a copy of the unix command uptime on the hosts a and b.
Other questions in the FAQ section deal with the specifics of the mpirun command line interface; suffice it to say that it works equally well for MPI and non-MPI applications.
we also have mpir..
or we can just run ssh to each computer and run the program
like these
# for i in `cut --delimiter=: -f1 ~/mpd.hosts`;do ssh $i "ls /mnt/my/$i";done
if we need to do blender, we can use nfs then create folder on master node with the name of the ip.
then set to run blender using the folder.
so the folder on master node will will look like these
/mnt/my/192.168.2.130/
/mnt/my/192.168.2.118/
/mnt/my/192.168.2.122/
the picture that needed to blended and the result will be saved into that folder, so each folder will contain different picture.
from http://wiki.blender.org/index.php/Doc:M ... ne_Options
In some situations we want to increase the render speed, access blender remotely to render something or build scripts that use blender command line.
One advantage of using command line is that we don't need the X server (in case of Linux) and as a consequence we can render remotely by SSH or telnet.
next...
run python program on mpi
http://sourceforge.net/projects/mpy/fil ... z/download
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
Mistaken understanding
Yeah, I can see how this was misleading. Newspapers are constantly taking a quote out of context. When I made that quote (look at the whole passage), I was making it meaning to have whole OS's run and their services being delivered from the internal OS's. So, for example, if I dispatched a OS as a cluster application, whatever internal services that OS provides could be available via that OS....not the cluster directly...internal from the OS. In the case of Puppy (an OS), I would have services (PET/SFS) within it available.Master_wrong wrote:well...Thanks Master_wrong (and @jpeps too). I agree with you that today's Office Suite is NOT an application for clustering (and I don't believe I suggested it either).
you did said, which i replywe could run every PUP, every PET, every SFS,sorry if i make it sound you were suggesting it but office suite does packaged as pet and sfs.I dont think that using cluster will benefit all program, such as office suite etc
btw if we see from this picture
http://pve.proxmox.com/wiki/File:Pve-vi ... chines.png
its not virtual, it seems real pc, however they they run virtual macine on each pc node. so its not that we run 1 virtual machine on server that use cluster processors.
I already find software that monitor each node, but its not as polished as theirs.
On the item of PROXMOX, I agree without lifting the covers on how they achieved their implementation, it might not be a cluster; but it they are, in fact, clustering, then they have an implementation where we can draw upon for understanding and for future Puppy implementations.
Here's the kind of idea that one can draw: Physical Cluster with Virtual machines running with some virtual Machines actually being a cluster with other virtual machines with ... I think everyone can see where this leads to... you know physical to virtual to virtual to... alongside other "real" individual cluster applications being globbled up by physical and virtual cluster machines.
Anyway, you have made a very promising contribution to the Puppy community thus far. And I applaud your achievement!
Hope this helps
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
Yeah, I can see how this was misleading. Newspapers are constantly taking a quote out of context. When I made that quote (look at the whole passage), I was making it meaning to have whole OS's run and their services being delivered from the internal OS's. So, for example, if I dispatched a OS as a cluster application, whatever internal services that OS provides could be available via that OS....not the cluster directly...internal from the OS. In the case of Puppy (an OS), I would have services (PET/SFS) within it available.
i guess.. forgive me for being me.... ?
but i guess that would be too difficult to implement. then again i could be me ?
btw i just read this...
ok has anyone have 320 p3 computer to test ?What is a beowulf cluster or Beowulf cluster vs Supercomputer
Beowulf cluster is essentially a number of commodity or of the
shelf PCs tied together using some sort of protocol. Developed
by NASA, Beowulf clusters gained much attention in the last few
years. Often at a fraction of a cost of a "real" supercomputer
a beowulf cluster is comparable (or even exceeds) the
computational power of a super computer. For instance the a
cluster made out of 320 ubiquitous P3-800 CPUs takes 34th place
on the TOP500 super computer list with peak performance of over
a teraflop!
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
-
- Posts: 17
- Joined: Tue 18 Jan 2011, 02:07
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
i spend last night testing like couple dozens of program....
it end up only couple of program that compiled successfully
here is 2 interesting program that might be useful,
1. create /etc/exports file: contain=====> /root (ro) --->for read only
open terminal at unfsd directrory
2. ./configure
3. make
4. make install
5. ./unfsd -s -d -p -n 4711 -m 4711 -e /etc/exports
6. at client : mkdir /mnt/try
7. copy unfsd,./configure,make,make install
8. mount -o port=4711,mountport=4711,mountvers=3,nfsvers=3,nolock,tcp 192.168.2.130:/root /mnt/try
it end up only couple of program that compiled successfully
here is 2 interesting program that might be useful,
too bad i cannot get anything to work, i simply not smart enough to create config file, if anyone capable please let me know1. tentakel :
Tentakel is a program for executing the same command on many hosts in parallel
using various remote methods. It can make use of several sets of hosts that are
defined in a configuration file as groups.
It also supports an interactive mode that can be used for repeated commands.
The author uses tentakel to simultaneously install software on many
workstations or ask and set parameters on a linux compute cluster. With the
power of format strings tentakel can also be used for monitoring purposes.
Supported remote methods are ssh(1) and rsh(1). Both must be configured to
allow for password-less logins. Password-protected keyfiles for ssh can be
used with ssh-agent(1).
A plugin mechanism allows users to implement their own remote methods in
addition to the builtin ones.
For more information on available options please refer to the manpage
tentakel(1).
The project homepage is: http://tentakel.biskalar.de/
all you need to do
2. unfs3-0.9.22.tar.gz :Great NFS for puppy, it capable of export and mount /root
INTRODUCTION
============
UNFS3 is a user-space implementation of the NFSv3 server
specification.
UNFS3 supports all NFSv3 procedures with the exception of the
READDIRPLUS procedure. It tries to provide as much information
to NFS clients as possible, within the limits possible from
user-space.
See the unfsd(8) manpage for restrictions imposed on NFS
operations (section RESTRICTIONS) and for possible races
with local file system activity (section BUGS).
It is not possible to export to netgroups or wildcard hostnames
via /etc/exports, all other addressing methods should work. The
following options are recognized in the exports file: ro, rw,
root_squash, all_squash, no_root_squash, no_all_squash. If
other options are present, they are ignored.
Cluster extensions compatible to the older ClusterNFS project
are supported when the source is configured with --enable-cluster.
1. create /etc/exports file: contain=====> /root (ro) --->for read only
open terminal at unfsd directrory
2. ./configure
3. make
4. make install
5. ./unfsd -s -d -p -n 4711 -m 4711 -e /etc/exports
6. at client : mkdir /mnt/try
7. copy unfsd,./configure,make,make install
8. mount -o port=4711,mountport=4711,mountvers=3,nfsvers=3,nolock,tcp 192.168.2.130:/root /mnt/try
- Attachments
-
- unfs3-test.png
- (25.94 KiB) Downloaded 1110 times
-
- tentakel.zip
- (42.15 KiB) Downloaded 449 times
-
- unfs3-0.9.22.tar.gz
- (164.5 KiB) Downloaded 473 times
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
made pet of unfs3 at http://www.murga-linux.com/puppy/viewtopic.php?t=64491
btw pxe booting work using
http://www.murga-linux.com/puppy/viewtopic.php?t=64107
also other program tested
btw pxe booting work using
http://www.murga-linux.com/puppy/viewtopic.php?t=64107
also other program tested
pconsole does not work through daemons. The pconsole.sh script opens up a
number of connections to each node of your cluster of workstations. Then,
the pconsole binary attaches to the tty devices that these windows are
using, and it copies the input you type to all open connections......
.......
By default, pconsole tries to use SSH to make connections. If you do not have
'ssh', it will try to use 'telnet'.
- Attachments
-
- pconsole-test.png
- (32.67 KiB) Downloaded 488 times
Last edited by Master_wrong on Sun 06 Feb 2011, 06:12, edited 2 times in total.
What is a cluster
@Master _Wrong has done all of us a very valuable service. I am showing this picture here because over the past couple months, many who come here have an inaccurate perception of a cluster.
This picture is worth a thousand words, as, it shows where to place the monitor, keyboard, and mouse for the cluster. In a true cluster, the nodes/workers/slave PCs DO NOT HAVE A MONITOR, KEYBOARD, OR MOUSE. But they all work together on behalf of the workload within the cluster.
Hope this helps
This picture is worth a thousand words, as, it shows where to place the monitor, keyboard, and mouse for the cluster. In a true cluster, the nodes/workers/slave PCs DO NOT HAVE A MONITOR, KEYBOARD, OR MOUSE. But they all work together on behalf of the workload within the cluster.
Hope this helps
- Attachments
-
- Beowulf.png
- (86.79 KiB) Downloaded 521 times
Last edited by gcmartin on Sat 05 Feb 2011, 18:28, edited 1 time in total.
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
Thank you gcmartin,
your effort in pxe documentation also help this project somehow.
btw here is some article about cluster:
http://www.clustermonkey.net//content/view/89/32/
the site also has ton of info about mpi
http://www.clustermonkey.net//content/view/14/28/
your effort in pxe documentation also help this project somehow.
btw here is some article about cluster:
http://www.clustermonkey.net//content/view/89/32/
the site also has ton of info about mpi
http://www.clustermonkey.net//content/view/14/28/
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
Wow, Master_wrong....you HAVE been busy, and got away from this thread...
http://www.murga-linux.com/puppy/viewtopic.php?t=34647
Since my initial interest, I've had a change in hardware direction, and now don't use much of my old scsi server boxes, as they use too much power, and make too much noise
However, I still think clustering is under-rated, especially in the puppyworld, with so much old hardware available
A couple or 3/more of useful links
http://www.linuxclusters.com/books/clus ... index.html
http://freshmeat.net/articles/linux-clustering-software
http://www.jumpingbean.co.za/linux-clus ... ailability
http://linuxreviews.org/howtos/virtual_ ... ter_howto/
http://www.ultramonkey.org/
http://www.gdargaud.net/Hack/ClusterNotes.html#Software
http://www.cs.wisc.edu/condor/
Good luck guys....some spaghetti examples
http://www.calvin.edu/~adams/research/m ... f/related/
Aitch
http://www.murga-linux.com/puppy/viewtopic.php?t=34647
Since my initial interest, I've had a change in hardware direction, and now don't use much of my old scsi server boxes, as they use too much power, and make too much noise
However, I still think clustering is under-rated, especially in the puppyworld, with so much old hardware available
A couple or 3/more of useful links
http://www.linuxclusters.com/books/clus ... index.html
http://freshmeat.net/articles/linux-clustering-software
http://www.jumpingbean.co.za/linux-clus ... ailability
http://linuxreviews.org/howtos/virtual_ ... ter_howto/
http://www.ultramonkey.org/
http://www.gdargaud.net/Hack/ClusterNotes.html#Software
http://www.cs.wisc.edu/condor/
Good luck guys....some spaghetti examples
http://www.calvin.edu/~adams/research/m ... f/related/
Aitch
-
- Posts: 452
- Joined: Thu 20 Mar 2008, 01:48
Hi Aitch, welcome aboard...
yes it quite complicated...Wow, Master_wrong....you HAVE been busy, and got away from this thread...
but it was fun, still one thing is missing... howto incorporate pxe into it, yes i can do it already but how to make it more simple ?
I'm thinking about IBS atm, eh no no not irritable bowel syndrome
here:
http://www.stsx.org/projects/ibs/ibs-manual.html
Cluster-Pup v.2-Puppy Beowulf Cluster
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
[url]http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199[/url]
Try iPXE
http://ipxe.org/
I just saw it demo'd at a linux fair in London....it looks cool, but don't ask me how to combine it with what you've done...that's quite an achievement, already
btw, here's a tempting project to build to run it on
http://www.mini-itx.com/projects/cluster/?p
Aitch
http://ipxe.org/
I just saw it demo'd at a linux fair in London....it looks cool, but don't ask me how to combine it with what you've done...that's quite an achievement, already
btw, here's a tempting project to build to run it on
http://www.mini-itx.com/projects/cluster/?p
Aitch