Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Sat 25 Oct 2014, 17:09
All times are UTC - 4
 Forum index » Advanced Topics » Puppy Projects
Puppy 245mpi cluster made easy
Moderators: Flash, JohnMurga
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
Page 3 of 5 Posts_count   Goto page: Previous 1, 2, 3, 4, 5 Next
Author Message
jpeps

Joined: 31 May 2008
Posts: 3220

PostPosted: Fri 28 Jan 2011, 23:22    Post_subject:  

acrocosm wrote:
Master_Wrong I'm sorry, I have to clarify...

The original scripts included on the iso would show 2 cpu for the x40 on the server. Both of your later auto-config scripts worked fine. The server would see just one cpu on the x40 as expected

However,
Code:
grep -c processor /proc/cpuinfo
still gives 2 as a result.

grep -c ^processor
Back to top
View user's profile Send_private_message 
gcmartin

Joined: 14 Oct 2005
Posts: 4368
Location: Earth

PostPosted: Sat 29 Jan 2011, 13:20    Post_subject: Cluster applications  

IFF you are wondering how others have applied clusters, other than the one's I've mentioned earlier, look here for ways to use yours!
Hope this helps

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send_private_message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Sat 29 Jan 2011, 15:01    Post_subject:  

@gcmartin

Quote:
P.S. It occurs that if we could replicate something like this in a Puppy Cluster, we could run every PUP, every PET, every SFS, and everything in the cluster. Now, hmmmm...this is some real food for those Workers to feed on from the Master. Big Dogs versus Small Dogs rules will still apply, though. But, the work will be spread in a very interesting manner.


I dont think that using cluster will benefit all program, such as office suite etc they dont need that much processing power...
i believe that those computer that waiting idle in screensaver mode would be more useful if used as cluster. but not all program need cluster processing power.

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send_private_message 
jpeps

Joined: 31 May 2008
Posts: 3220

PostPosted: Sat 29 Jan 2011, 16:12    Post_subject:  

Master_wrong wrote:


I dont think that using cluster will benefit all program, such as office suite etc they dont need that much processing power

Seems like office suite, etc., would need unique software to make use of clustering (i.e, that can run parallel tasks); doesn't make much sense for basic apps.
Back to top
View user's profile Send_private_message 
gcmartin

Joined: 14 Oct 2005
Posts: 4368
Location: Earth

PostPosted: Sat 29 Jan 2011, 16:59    Post_subject: Mistaken understanding  

Master_wrong wrote:
Quote:
P.S. It occurs that if we could replicate something like this in a Puppy Cluster ... .
I dont think that using cluster will benefit all program, such as office suite etc they dont need that much processing power...
i believe that those computer that waiting idle in screensaver mode would be more useful if used as cluster. but not all program need cluster processing power.
Thanks Master_wrong (and @jpeps too). I agree with you that today's Office Suite is NOT an application for clustering (and I don't believe I suggested it either).

The example I provided was one where someone has found a way to dispatch VMs as work in a cluster. That's novel, if it works as they suggest and is a greater way to throw workers at workload. I only threw that our way in Puppyland to inspire ideas for what can be accomplished in a Puppy cluster.

Hope this helps

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send_private_message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Sat 29 Jan 2011, 23:14    Post_subject:  

Quote:
Thanks Master_wrong (and @jpeps too). I agree with you that today's Office Suite is NOT an application for clustering (and I don't believe I suggested it either).


well...
you did said
Quote:
we could run every PUP, every PET, every SFS,
,
which i reply
Quote:
I dont think that using cluster will benefit all program, such as office suite etc


sorry if i make it sound you were suggesting it but office suite does packaged as pet and sfs.

btw if we see from this picture
http://pve.proxmox.com/wiki/File:Pve-virtual-machines.png

its not virtual, it seems real pc, however they they run virtual macine on each pc node.
so its not that we run 1 virtual machine on server that use cluster processors.

I already find software that monitor each node, but its not as polished as theirs.

Quote:

Description:
------------
Gossimon is a gossip based distributed monitoring system for a cluster of
Linux nodes. Each node in the cluster periodically send information about
itself and others to a randomly selected node. This way each node constantly
receive information about cluster nodes. This information is locally
maintained (constantly updated) by each node and can be used by various
clients.
gossimon-1.8.8-Source.tar.gz
Description 
gz

 Download 
Filename  gossimon-1.8.8-Source.tar.gz 
Filesize  392.03 KB 
Downloaded  282 Time(s) 

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send_private_message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Sun 30 Jan 2011, 07:31    Post_subject:  

This is my code to enable nfs,
192.168.2.130 is server
192.168.2.118 is client

http://murga-linux.com/puppy/viewtopic.php?t=45090

we only need to install nfs_common.tar.gz from above site, in the end both server and client will be able to access /mnt/my.


(already included at the 1st post)

####NFS ---need nfs_common.tar.gz only
### change drive="sda1" to your mounted device
## please create ~/my-applications/clusterdata/ip-list which contain list of ip
#need passwordless ssh

drive="sda1"
mm=`head -n 1 ~/my-applications/clusterdata/ip-list`
m=`echo ${mm/ /}`
echo "M="$m

rm /etc/exports
echo "/mnt/$drive *(rw,no_root_squash,no_subtree_check,sync)" >>/etc/exports
modprobe nfsd
/etc/rc.d/rc.nfsd start
x="mount-FULL -t nfs $m:/mnt/$drive /mnt/my "
echo "x="$x
for i in `cut --delimiter=: -f1 ~/my-applications/clusterdata/ip-list`;do ssh $i "mkdir /mnt/my"; ssh $i "modprobe nfs";ssh $i "/etc/rc.d/rc.portmap start";ssh $i $x;done
####NFS

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199

Edited_time_total
Back to top
View user's profile Send_private_message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Mon 31 Jan 2011, 13:04    Post_subject:  

Although i did not agree gcmartin vision about possibility running all program using cluster, i make some research and put them here, i hope someone could use them...
from http://www.open-mpi.org/faq/?category=running
Quote:

12. Can I run non-MPI programs with mpirun / mpiexec?
Yes.

Indeed, Open MPI's mpirun and mpiexec are actually synonyms for our underlying launcher named orterun (i.e., the Open Run-Time Environment layer in Open MPI, or ORTE). So you can use mpirun and mpiexec to launch any application. For example:

shell$ mpirun -np 2 --host a,b uptime

This will launch a copy of the unix command uptime on the hosts a and b.
Other questions in the FAQ section deal with the specifics of the mpirun command line interface; suffice it to say that it works equally well for MPI and non-MPI applications.


i know its open-mpi, but it s the same implementation, we can try...
we also have mpir..
or we can just run ssh to each computer and run the program
like these

# for i in `cut --delimiter=: -f1 ~/mpd.hosts`;do ssh $i "ls /mnt/my/$i";done

if we need to do blender, we can use nfs then create folder on master node with the name of the ip.
then set to run blender using the folder.

so the folder on master node will will look like these
/mnt/my/192.168.2.130/
/mnt/my/192.168.2.118/
/mnt/my/192.168.2.122/
the picture that needed to blended and the result will be saved into that folder, so each folder will contain different picture.

from http://wiki.blender.org/index.php/Doc:Manual/Render/Command_Line_Options

Quote:
In some situations we want to increase the render speed, access blender remotely to render something or build scripts that use blender command line.

One advantage of using command line is that we don't need the X server (in case of Linux) and as a consequence we can render remotely by SSH or telnet.



next...
run python program on mpi
http://sourceforge.net/projects/mpy/files/mpych/0.1/mpi_wrap_mpych-0.1.linux-i686.tar.gz/download

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send_private_message 
gcmartin

Joined: 14 Oct 2005
Posts: 4368
Location: Earth

PostPosted: Mon 31 Jan 2011, 13:14    Post_subject: Mistaken understanding  

Master_wrong wrote:
Quote:
Thanks Master_wrong (and @jpeps too). I agree with you that today's Office Suite is NOT an application for clustering (and I don't believe I suggested it either).
well...
you did said
Quote:
we could run every PUP, every PET, every SFS,
, which i reply
Quote:
I dont think that using cluster will benefit all program, such as office suite etc
sorry if i make it sound you were suggesting it but office suite does packaged as pet and sfs.

btw if we see from this picture
http://pve.proxmox.com/wiki/File:Pve-virtual-machines.png

its not virtual, it seems real pc, however they they run virtual macine on each pc node. so its not that we run 1 virtual machine on server that use cluster processors.

I already find software that monitor each node, but its not as polished as theirs.
Yeah, I can see how this was misleading. Newspapers are constantly taking a quote out of context. When I made that quote (look at the whole passage), I was making it meaning to have whole OS's run and their services being delivered from the internal OS's. So, for example, if I dispatched a OS as a cluster application, whatever internal services that OS provides could be available via that OS....not the cluster directly...internal from the OS. In the case of Puppy (an OS), I would have services (PET/SFS) within it available.

On the item of PROXMOX, I agree without lifting the covers on how they achieved their implementation, it might not be a cluster; but it they are, in fact, clustering, then they have an implementation where we can draw upon for understanding and for future Puppy implementations.

Here's the kind of idea that one can draw: Physical Cluster with Virtual machines running with some virtual Machines actually being a cluster with other virtual machines with ... I think everyone can see where this leads to... you know physical to virtual to virtual to... alongside other "real" individual cluster applications being globbled up by physical and virtual cluster machines.

Anyway, you have made a very promising contribution to the Puppy community thus far. And I applaud your achievement!

Hope this helps

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send_private_message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Mon 31 Jan 2011, 13:43    Post_subject:  

Quote:
Yeah, I can see how this was misleading. Newspapers are constantly taking a quote out of context. When I made that quote (look at the whole passage), I was making it meaning to have whole OS's run and their services being delivered from the internal OS's. So, for example, if I dispatched a OS as a cluster application, whatever internal services that OS provides could be available via that OS....not the cluster directly...internal from the OS. In the case of Puppy (an OS), I would have services (PET/SFS) within it available.


Embarassed
i guess.. forgive me for being me.... ?

but i guess that would be too difficult to implement. then again i could be me ? Laughing

btw i just read this...
Quote:

What is a beowulf cluster or Beowulf cluster vs Supercomputer

Beowulf cluster is essentially a number of commodity or of the
shelf PCs tied together using some sort of protocol. Developed
by NASA, Beowulf clusters gained much attention in the last few
years. Often at a fraction of a cost of a "real" supercomputer
a beowulf cluster is comparable (or even exceeds) the
computational power of a super computer. For instance the a
cluster made out of 320 ubiquitous P3-800 CPUs takes 34th place
on the TOP500 super computer list with peak performance of over
a teraflop!


ok has anyone have 320 p3 computer to test ? Rolling Eyes

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send_private_message 
whatupuppy

Joined: 17 Jan 2011
Posts: 17

PostPosted: Wed 02 Feb 2011, 12:02    Post_subject:  

Update:
Well things did not go well with my friend this weekend.

So this have to be put on hold for now.

Also, failing hard drive on old laptops causing puppy to hang at Uncompressing linux..OK.booting Linux kernel.

Any suggestion beside buying a new harddrive is welcomed.
Back to top
View user's profile Send_private_message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Wed 02 Feb 2011, 20:59    Post_subject:  

I was about to suggest buying flashdisk....

so there is no cdrom, i guess ?

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send_private_message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Wed 02 Feb 2011, 21:19    Post_subject:  

i spend last night testing like couple dozens of program....
it end up only couple of program that compiled successfully

here is 2 interesting program that might be useful,

Quote:
1. tentakel :
Tentakel is a program for executing the same command on many hosts in parallel
using various remote methods. It can make use of several sets of hosts that are
defined in a configuration file as groups.

It also supports an interactive mode that can be used for repeated commands.

The author uses tentakel to simultaneously install software on many
workstations or ask and set parameters on a linux compute cluster. With the
power of format strings tentakel can also be used for monitoring purposes.

Supported remote methods are ssh(1) and rsh(1). Both must be configured to
allow for password-less logins. Password-protected keyfiles for ssh can be
used with ssh-agent(1).

A plugin mechanism allows users to implement their own remote methods in
addition to the builtin ones.

For more information on available options please refer to the manpage
tentakel(1).



The project homepage is: http://tentakel.biskalar.de/

too bad i cannot get anything to work, i simply not smart enough to create config file, if anyone capable please let me know




Quote:



2. unfs3-0.9.22.tar.gz :Great NFS for puppy, it capable of export and mount /root

INTRODUCTION
============

UNFS3 is a user-space implementation of the NFSv3 server
specification.

UNFS3 supports all NFSv3 procedures with the exception of the
READDIRPLUS procedure. It tries to provide as much information
to NFS clients as possible, within the limits possible from
user-space.

See the unfsd(Cool manpage for restrictions imposed on NFS
operations (section RESTRICTIONS) and for possible races
with local file system activity (section BUGS).

It is not possible to export to netgroups or wildcard hostnames
via /etc/exports, all other addressing methods should work. The
following options are recognized in the exports file: ro, rw,
root_squash, all_squash, no_root_squash, no_all_squash. If
other options are present, they are ignored.

Cluster extensions compatible to the older ClusterNFS project
are supported when the source is configured with --enable-cluster.


all you need to do

1. create /etc/exports file: contain=====> /root (ro) --->for read only
open terminal at unfsd directrory
2. ./configure
3. make
4. make install
5. ./unfsd -s -d -p -n 4711 -m 4711 -e /etc/exports


6. at client : mkdir /mnt/try
7. copy unfsd,./configure,make,make install
8. mount -o port=4711,mountport=4711,mountvers=3,nfsvers=3,nolock,tcp 192.168.2.130:/root /mnt/try
unfs3-test.png
 Description   
 Filesize   25.94 KB
 Viewed   991 Time(s)

unfs3-test.png

tentakel.zip
Description 
zip

 Download 
Filename  tentakel.zip 
Filesize  42.15 KB 
Downloaded  264 Time(s) 
unfs3-0.9.22.tar.gz
Description 
gz

 Download 
Filename  unfs3-0.9.22.tar.gz 
Filesize  164.5 KB 
Downloaded  285 Time(s) 

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send_private_message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Fri 04 Feb 2011, 21:43    Post_subject:  

made pet of unfs3 at http://www.murga-linux.com/puppy/viewtopic.php?t=64491

btw pxe booting work using
http://www.murga-linux.com/puppy/viewtopic.php?t=64107

also other program tested
Quote:

pconsole does not work through daemons. The pconsole.sh script opens up a
number of connections to each node of your cluster of workstations. Then,
the pconsole binary attaches to the tty devices that these windows are
using, and it copies the input you type to all open connections......
.......
By default, pconsole tries to use SSH to make connections. If you do not have
'ssh', it will try to use 'telnet'.
pconsole-1.0.tar.gz
Description 
gz

 Download 
Filename  pconsole-1.0.tar.gz 
Filesize  210.38 KB 
Downloaded  251 Time(s) 
pconsole-test.png
Description 
png

 Download 
Filename  pconsole-test.png 
Filesize  32.67 KB 
Downloaded  238 Time(s) 

Edited_times_total
Back to top
View user's profile Send_private_message 
gcmartin

Joined: 14 Oct 2005
Posts: 4368
Location: Earth

PostPosted: Sat 05 Feb 2011, 13:58    Post_subject: What is a cluster  

@Master _Wrong has done all of us a very valuable service. I am showing this picture here because over the past couple months, many who come here have an inaccurate perception of a cluster.

[img]http://docs.google.com/drawings/edit?id=16jMd2gmt5YmmOlLWmb1q7fhufv4m0qKoKCZQJ348J54&hl=en[/img]

This picture is worth a thousand words, as, it shows where to place the monitor, keyboard, and mouse for the cluster. In a true cluster, the nodes/workers/slave PCs DO NOT HAVE A MONITOR, KEYBOARD, OR MOUSE. But they all work together on behalf of the workload within the cluster.

Hope this helps
Beowulf.png
Description 
png

 Download 
Filename  Beowulf.png 
Filesize  86.79 KB 
Downloaded  255 Time(s) 

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile

Edited_time_total
Back to top
View user's profile Send_private_message 
Display_posts:   Sort by:   
Page 3 of 5 Posts_count   Goto page: Previous 1, 2, 3, 4, 5 Next
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
 Forum index » Advanced Topics » Puppy Projects
Jump to:  

Rules_post_cannot
Rules_reply_cannot
Rules_edit_cannot
Rules_delete_cannot
Rules_vote_cannot
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1194s ][ Queries: 12 (0.0083s) ][ GZIP on ]