Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Sat 20 Sep 2014, 16:22
All times are UTC - 4
 Forum index » Advanced Topics » Puppy Projects
Puppy 245mpi cluster made easy
Moderators: Flash, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 4 of 5 [72 Posts]   Goto page: Previous 1, 2, 3, 4, 5 Next
Author Message
gcmartin

Joined: 14 Oct 2005
Posts: 4276
Location: Earth

PostPosted: Sat 05 Feb 2011, 14:26    Post subject:  

.
Back to top
View user's profile Send private message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Mon 07 Feb 2011, 10:38    Post subject:  

Thank you gcmartin,

your effort in pxe documentation also help this project somehow.

btw here is some article about cluster:
http://www.clustermonkey.net//content/view/89/32/

the site also has ton of info about mpi
http://www.clustermonkey.net//content/view/14/28/

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
Aitch


Joined: 04 Apr 2007
Posts: 6825
Location: Chatham, Kent, UK

PostPosted: Tue 08 Feb 2011, 01:07    Post subject:  

Wow, Master_wrong....you HAVE been busy, and got away from this thread...

http://www.murga-linux.com/puppy/viewtopic.php?t=34647

Since my initial interest, I've had a change in hardware direction, and now don't use much of my old scsi server boxes, as they use too much power, and make too much noise Wink

However, I still think clustering is under-rated, especially in the puppyworld, with so much old hardware available

A couple or 3/more of useful links

http://www.linuxclusters.com/books/clusters/chapter1/index.html

http://freshmeat.net/articles/linux-clustering-software

http://www.jumpingbean.co.za/linux-cluster-load-balancing-high-availability

http://linuxreviews.org/howtos/virtual_linux_cluster_howto/

http://www.ultramonkey.org/

http://www.gdargaud.net/Hack/ClusterNotes.html#Software

http://www.cs.wisc.edu/condor/

Good luck guys....some spaghetti examples

http://www.calvin.edu/~adams/research/microwulf/related/

Aitch Smile
Back to top
View user's profile Send private message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Tue 08 Feb 2011, 09:25    Post subject:  

Very Happy
Hi Aitch, welcome aboard...

Quote:

Wow, Master_wrong....you HAVE been busy, and got away from this thread...


yes it quite complicated...
but it was fun, still one thing is missing... howto incorporate pxe into it, yes i can do it already but how to make it more simple ?

I'm thinking about IBS atm, eh no no not irritable bowel syndrome Laughing

here:
http://www.stsx.org/projects/ibs/ibs-manual.html

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
Aitch


Joined: 04 Apr 2007
Posts: 6825
Location: Chatham, Kent, UK

PostPosted: Tue 08 Feb 2011, 19:20    Post subject:  

Try iPXE

http://ipxe.org/

I just saw it demo'd at a linux fair in London....it looks cool, but don't ask me how to combine it with what you've done...that's quite an achievement, already Very Happy

btw, here's a tempting project to build to run it on

http://www.mini-itx.com/projects/cluster/?p

Aitch Smile
Back to top
View user's profile Send private message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Wed 09 Feb 2011, 11:21    Post subject:  

@aitch

Quote:
You can use iPXE to replace the existing PXE ROM on your network card, or you can chainload into iPXE to obtain the features of iPXE without the hassle of reflashing.


from above statement we know that
1. its used to replace existing pxe rom on network card ----> client side
2. chainload into ipxe ----> which mean pxe chainload into ipxe ---->client side

what we need is server side... such as IBS ----> Instant Boot Server

once we have the server surely we can use ipxe... maybe to create cloud puppy supercomputer ?
that would be cool Cool

Btw speaking of pxe....

I just realize that when remastering, when program ask about separate zl245332.... you have to answer "no"...

Quote:

that's quite an achievement, already Very Happy

thanks aitch, the best part is you can actually install other newer version of mpich and use my script because the setting is the same Idea
all you need is correct version of python...

for example you want version 1.1.1 of mpich2, you will need python 2.6 at least...

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
Aitch


Joined: 04 Apr 2007
Posts: 6825
Location: Chatham, Kent, UK

PostPosted: Wed 09 Feb 2011, 12:16    Post subject:  

Master_wrong

Ah yes, I see

I realize this also plugs into Sunburnt's previous posts, here

http://www.murga-linux.com/puppy/viewtopic.php?t=5797

and

http://www.murga-linux.com/puppy/viewtopic.php?t=9198

which is back in 2006....

Haven't seen Sunburnt around for a while though....but this
Sunburnt wrote:
I think a Puppy server - client setup on a CD is what John Murga has in mind.
Keep an eye out for possable [sic] developments.

seems to have finally become a distinct possible

Maybe time to think about some old thin client hardware on ebay....?

'....cloud puppy supercomputer' ?

mmm, please Laughing

Aitch Smile
Back to top
View user's profile Send private message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Thu 10 Feb 2011, 12:29    Post subject:  

Quote:


seems to have finally become a distinct possible



here is the IBS, i try but cannot understand how to create chroot dir...
supposedly this can create pxe server with nfs and dhcp all in one.
ibs.zip
Description 
zip

 Download 
Filename  ibs.zip 
Filesize  173.5 KB 
Downloaded  419 Time(s) 

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Sat 12 Feb 2011, 12:51    Post subject:  

i have research about the trend of supercomputer using GPU, and here is what i found...

some information...
http://rcc.uga.edu/equipment/subpages/gpu_computing/

driver...
http://developer.nvidia.com/object/cuda_3_2_downloads.html#Linux

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
Aitch


Joined: 04 Apr 2007
Posts: 6825
Location: Chatham, Kent, UK

PostPosted: Mon 14 Feb 2011, 19:39    Post subject:  

How about a Puppy users p2p cluster?

http://gpu.sourceforge.net/index.php

http://sourceforge.net/projects/gpu/

Harness

http://www.csm.ornl.gov/harness/

Condor [previously listed?]

http://www.cs.wisc.edu/condor/

Using Network mapper

http://sourceforge.net/project/screenshots.php?group_id=58134

High speed compile environment, maybe? Looks fun!

Linux HA [high availability cluster] Users Guide

http://www.linux-ha.org/doc/users-guide/users-guide.html

Aitch Smile
Back to top
View user's profile Send private message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Tue 15 Feb 2011, 08:03    Post subject:  

from
Linux HA [high availability cluster] Users Guide

Quote:
This utility will copy the configuration files over to any node listed in /etc/ha.d/ha.cf using scp.


scp, that's what i use to set-up my puppy-nodes and server Cool

btw i have successfully test automatic setting of all node and server when using pxe...
it means puppy-cluster will only need 1 cd (and nodes with 1gb ram) that is it.
just boot server, boot all nodes, then after all computer boot up properly, all setting are set automagically, no need to input ip setting at server anymore but i open geany anyway just in case...

so I have accomplish my goal for the project

My goal for the project is
1. make puppy cluster computer ---> Done
2. setting all cluster from server ----> Done
3. automatic setting ----> Done

_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
Aitch


Joined: 04 Apr 2007
Posts: 6825
Location: Chatham, Kent, UK

PostPosted: Tue 15 Feb 2011, 19:33    Post subject:  

You forgot the 'crucial ending line'....

Testers needed! Very Happy

As soon as I can get some time/space I intend to test it - thanks!

Have you got a working software list, or do I need to learn C++? Confused Wink

Aitch Smile
Back to top
View user's profile Send private message 
Master_wrong

Joined: 19 Mar 2008
Posts: 456

PostPosted: Wed 16 Feb 2011, 03:32    Post subject:  

Very Happy Very Happy Very Happy

Quote:

Have you got a working software list, or do I need to learn C++? Confused Wink

indeed, john the ripper is working, netcrack seem working...
btw you do know that mpir.c from svat can be used to run normal program into mpi-cluster environment ?

Quote:
You forgot the 'crucial ending line'....

Testers needed! Very Happy

sure aitch...

just make sure you create the humongous initrd first to boot pxe...

that means copy this script, make it autoload (also the unfs server if needed), remove the old autoconfig, remaster..., create humongous-initrd from remastered-iso...

then reboot with the new iso, set pxe into normal or debug mode,
then boot all other pc with pxe enabled when server ask
"Press ok if THIS PC IS SERVER..."

after all pc boot properly, click ok, see list of ip in geany, after you close geany, the cluster is ready to run mpich2.

attached is my test pic

Have a nice day

edit... I forgot to add dhcpcd -d
i use seperate file at startup which run dhcpcd -d


Quote:

#!/bin/sh
tail -n +4 $0 > /tmp/test1.txt
exec rxvt -e sh /tmp/test1.txt

#------------------begin ssh setting
echo "ALL: ALL" > /etc/hosts.allow
echo " " > /etc/hosts.deny
if [ `egrep sshd /etc/passwd|wc -l` -eq 0 ]
then
echo "sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:sbin/nologin" >> /etc/passwd
fi
mkdir -p /var/empty/sshd
/usr/sbin/sshd
#-----------------end ssh setting

#begin check connection to network and show ip
no_network=""
ipp1=`/sbin/ifconfig eth0 | sed '/inet\ /!d;s/.*r://g;s/\ .*//g'`
until [[ $ipp1 != $no_network ]];do ipp1=`/sbin/ifconfig eth0 | sed '/inet\ /!d;s/.*r://g;s/\ .*//g'` ;sleep 3;echo;echo "Not connected to network ?";echo "please connect this computer to network"
done
echo $ipp1
#end

#check hostname master if available and run as server or client
ipp=$ipp1
ip=$ipp1"."
ipp2=`echo ${ip%.*.}`
ip_master=$ipp2".115"

echo "server ip= "$ip_master

m=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $ip_master "ls"`

if [ "$m" = "" ] ; then
echo "setting pc as server"
ifconfig eth0 $ip_master netmask 255.255.255.0 up
ifconfig eth0
#no master node--->be master
hostname puppy-1
rm ~/my-applications/clusterdata/ip-list
rm ~/my-applications/clusterdata/etc/hosts
rm /etc/hosts
echo "$ip_master puppy-1" >>~/my-applications/clusterdata/etc/hosts
echo "$ip_master puppy-1" >>/etc/hosts
echo "$ip_master" >~/my-applications/clusterdata/ip-list
/usr/sbin/netboot-server-jb.sh
echo "this computer is now server"

Xdialog --yesno "Press ok if THIS PC IS SERVER and make sure all pc-node are ready" 0 0 && server="confirm" ;echo $server "SERVER commencing..."
#--------------begin server
if [ "$server" == "confirm" ]; then
geany ~/my-applications/clusterdata/ip-list
xmessage "Please create list of all ip that you have,
put server on top of the list,
press ok after you save the list"
#---begin probing
n=0
rm ~/my-applications/clusterdata/home/mpd.hosts
rm ~/my-applications/clusterdata/etc/hosts

#ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no localhost -v
echo 127.0.0.1 localhost>>~/my-applications/clusterdata/etc/hosts
for i in `cut --delimiter=: -f1 ~/my-applications/clusterdata/ip-list`;do process=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $i "cat /proc/cpuinfo|grep ^processor|wc -l"`;let n="$n+1";echo $n;echo $i:$process>>~/my-applications/clusterdata/home/mpd.hosts;echo $i puppy-$n>>~/my-applications/clusterdata/etc/hosts;ssh $i hostname puppy-$n ;echo "computer:" $i "has:"$process " processor" ;done
#end probing

# begin setting server
cp -prv ~/my-applications/clusterdata/etc/* /etc/
cp -prv ~/my-applications/clusterdata/home/* ~
cp -prv ~/my-applications/clusterdata/home/.mpd.conf ~
# end setting server

#begin setting cluster
# for i in `cut --delimiter=: -f1 ~/mpd.hosts`;do ssh $i mkdir -p .ssh; cat ~/.ssh/id_rsa.pub |ssh $i "cat>>~/.ssh/authorized_keys";done
for i in `cut --delimiter=: -f1 ~/mpd.hosts`;do scp ~/.mpd.conf $i:~;scp ~/mpd.hosts $i:~;scp /etc/mpd.conf $i:/etc;scp /etc/hosts $i:/etc;done
fi
#end setting cluster
###set client nfs
for i in `cut --delimiter=: -f1 ~/my-applications/clusterdata/ip-list`;do process=`ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $i "mkdir /mnt/try"; ssh $i "mount -o port=4711,mountport=4711,mountvers=3,nfsvers=3,nolock,tcp $ip_master:/root /mnt/try
"`;done
####NFS



else
#master exist---->send ip data to master
echo "server exist"
ssh -q -o BatchMode=yes -o StrictHostKeyChecking=no $ip_master "echo $ipp>>~/my-applications/clusterdata/ip-list"

echo "this pc is ready as node"
echo $ipp
hostname
fi

cd ~/my-applications
rox
xmessage " this pc is ready to run cluster "
cd ~
urxvt
test-server1.png
 Description   
 Filesize   16.74 KB
 Viewed   1019 Time(s)

test-server1.png

test-server2.png
 Description   
 Filesize   14.95 KB
 Viewed   341 Time(s)

test-server2.png

test-node.png
 Description   
 Filesize   5.06 KB
 Viewed   966 Time(s)

test-node.png


_________________
Cluster-Pup v.2-Puppy Beowulf Cluster
http://www.murga-linux.com/puppy/viewtopic.php?p=499199#499199
Back to top
View user's profile Send private message 
gcmartin

Joined: 14 Oct 2005
Posts: 4276
Location: Earth

PostPosted: Thu 17 Feb 2011, 14:00    Post subject: Excellent  

Master_wrong wrote:
from Linux HA [high availability cluster] Users Guide .... so I have accomplish my goal for the project ... My goal for the project is ... ----> Done
Thanks for clarifying everything you've done thru-out this Thread. I have followed you work religiously. Very clear objectives, approaches, conclusion.

I have one complex request.

  • Would you setup your LiveCd to automatically start the PXE environment when it boots without user intervention.
  • Would you post the LiveCD/LiveDVD that would boot the Cluster Master starting the PXE environment with those files necessary for the cluster nodes to PXE from, please.
This would make an automatic (another word for dummies like me) cluster where all I do is boot "Live" and start nodes, wthout ever typing or clicking on anything, to begin and run my cluster.

You may want to start a new Thread announcing your Multi PC "ClusterPUP".
See my PM to you.

Again, your work here is phenomenal.
Hope this helps

_________________
Get ACTIVE Create Circles; Do those good things which benefit people's needs!
We are all related ... Its time to show that we know this!
3 Different Puppy Search Engine or use DogPile
Back to top
View user's profile Send private message 
Aitch


Joined: 04 Apr 2007
Posts: 6825
Location: Chatham, Kent, UK

PostPosted: Fri 18 Feb 2011, 09:26    Post subject:  

gcmartin wrote:
You may want to start a new Thread announcing your Multi PC "ClusterPUP".
[See my PM to you.]

Again, your work here is phenomenal


2nd that, with big thumbs up Very Happy [even though I haven't implemented it yet]

Anyone doing video/high intensity graphics/blender will benefit from this, however, can compiling be speeded up on a cluster, it is common to hear it taking 10/12 hrs?

Aitch Smile
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 4 of 5 [72 Posts]   Goto page: Previous 1, 2, 3, 4, 5 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Puppy Projects
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1082s ][ Queries: 12 (0.0083s) ][ GZIP on ]