Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 31 Oct 2014, 09:02
All times are UTC - 4
 Forum index » Advanced Topics » Puppy Projects
Puppy for parallel computation using openMPI
Moderators: Flash, JohnMurga
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
Page 2 of 3 Posts_count   Goto page: Previous 1, 2, 3 Next
Author Message
Everitt

Joined: 19 Dec 2006
Posts: 331
Location: Leeds,UK or Birmingham, UK

PostPosted: Tue 04 Sep 2007, 08:47    Post_subject:  

I'm going to stick to 2.16 while I get a few things set up properly. I had tried 2.17, but nobody seems to be able to make the humungous initrd.gz boot propely.
As for existing code, I don't really know. OpenMPI has support for C[++] and fortran. Java should be possible, there are open-source projects avaliable, but the trouble is that to use java you need the java runtime enviroment. According to the Java Parallel Processing Framework FAQ you'd need a good 512MB of RAM. Not very puppy huh?
For comparision, a standard netboot of 2.16 needs somewhere between 128MB and 192MB (192 works, 128 doesn't). I plan on stripping a lot out for the nodes, so that should fall a bit too.
Back to top
View user's profile Send_private_message 
Everitt

Joined: 19 Dec 2006
Posts: 331
Location: Leeds,UK or Birmingham, UK

PostPosted: Wed 05 Sep 2007, 16:33    Post_subject:  

I spent most of today putting together the node version. I've got it working, but by the look of it there is a reasonable amount in the root filesystem that I don't need/want, so cleaning that out is going to take a while. Sad
Anyway, free reports a total of 63568KB used (62MB)
Back to top
View user's profile Send_private_message 
cb88


Joined: 28 Jan 2007
Posts: 1169
Location: USA

PostPosted: Wed 05 Sep 2007, 17:25    Post_subject:  

that is great about the 62mb ram(is that ram usage guess it is) usage hope you can get it even lower might be worth looking at how ipup or icepup and barebones are set up to try taking some stuff out
_________________
Taking Puppy Linux to the limit of perfection. meanwhile try "puppy pfix=duct_tape" kernel parem eater.
X86: Sager NP6110 3630QM 16GB ram, Tyan Thunder 2 2x 300Mhz
Sun: SS2 , LX , SS5 , SS10 , SS20 ,Ultra 1, Ultra 10 , T2000
Mac: Platinum Plus, SE/30
Back to top
View user's profile Send_private_message Visit_website 
bobwrit


Joined: 12 Mar 2007
Posts: 286

PostPosted: Wed 05 Sep 2007, 21:08    Post_subject:  

This project is perticualary interesting to me right now. I find supercomputer networks intreaging. c's also my specialty language(I have exp. in pascal,basic,python, and java.). At your local thirft store in America you can get a computer for $3. Wink
_________________
I need help with my forum. LINK:http://www.programers.co.nr/
http://www.freewebs.com/programm/iframe.html is my gateway page...
Back to top
View user's profile Send_private_message Visit_website YIM 
Everitt

Joined: 19 Dec 2006
Posts: 331
Location: Leeds,UK or Birmingham, UK

PostPosted: Thu 06 Sep 2007, 10:07    Post_subject:  

cb88 wrote:
that is great about the 62mb ram(is that ram usage guess it is) usage hope you can get it even lower might be worth looking at how ipup or icepup and barebones are set up to try taking some stuff out

Yep, at the moment it uses just over 62MB of RAM. Hopefully I should be able to lower that some by cleaning out things I'm not using in the root filesystem package, and by removing some modules from zdrv. (No point having graphics of sound drivers for example.) I'll certainly take a look at those varients, thanks for the tips. Smile

bobwrit wrote:
This project is perticualary interesting to me right now. I find supercomputer networks intreaging. c's also my specialty language(I have exp. in pascal,basic,python, and java.). At your local thirft store in America you can get a computer for $3. Wink

What, pay for a computer? Never! Razz I'm in the UK anyway.
I plan to post something usable at some point, but I warn you, it probably won't be soon and it will me rather restriced in what it can do. If you want to try something like this I'd recoment parallelknoppix. That at least has something you can download!

Success! Just as I was typing this I got a positive result, a node booting and running a job without any setup on my part. That leaves me a single test away from declaring it usable and starting work (again) on the server. (aka. the hard bit.)
Back to top
View user's profile Send_private_message 
cb88


Joined: 28 Jan 2007
Posts: 1169
Location: USA

PostPosted: Thu 06 Sep 2007, 11:06    Post_subject:  

could you compile the octave pankage for puppy that is in parallelknopix with matrixpup it seems to be a powerful matematical application and puppy doesn't have one yet i think it needs python which i don't have it will ./configure and make but fails because there is not enough dependacies on my pc a recompile of gcc would be nice too that would even be advantagious to barry sice he could compile faster in theory

probaby best to do theses as sfs modules

i realize that it might take a while to get all this stuff probably what you planed anyway

BTW my laptop has 64mb ram but no net card LOL might need to get a router and a card

_________________
Taking Puppy Linux to the limit of perfection. meanwhile try "puppy pfix=duct_tape" kernel parem eater.
X86: Sager NP6110 3630QM 16GB ram, Tyan Thunder 2 2x 300Mhz
Sun: SS2 , LX , SS5 , SS10 , SS20 ,Ultra 1, Ultra 10 , T2000
Mac: Platinum Plus, SE/30

Edited_time_total
Back to top
View user's profile Send_private_message Visit_website 
Everitt

Joined: 19 Dec 2006
Posts: 331
Location: Leeds,UK or Birmingham, UK

PostPosted: Thu 06 Sep 2007, 11:57    Post_subject:  

As far as I can see octave needs to be present on ever node, and so needs python on every node too. I'd rather not include that.
sfs files are a little impractical, they'd need to be on over nodes HDD. I could share them over NFS or SMB, but I'm trying to avoid that to reduce network traffic (so it runs better)

I'm not sure if gcc can be run in parallel, but there is a version of make that can (dmake) I'll certainlly look into that, but I won't include it at the moment.

What I'm planning at the moment could well be the solution to all of this. I plan to include the unleashed system, with a few basic scripts to make things easier, so you can create a initrd.gz on the fly, containing what you think you'll need. That way you aren't tied to what I have time to do, or my vision for the project. Of course, it'll mean you need a reasonable amount of ext3 space, but that's not so hard to come by.
Back to top
View user's profile Send_private_message 
cb88


Joined: 28 Jan 2007
Posts: 1169
Location: USA

PostPosted: Thu 06 Sep 2007, 15:27    Post_subject:  

could an sfs be uploaded to every node to the correct place on the hard drive then have the PC reboot? are local sfs files detected by net booting puppies? boot>>load sfs files>>reboot with the sfs files better than remastering and would only be a burden on the net during boot

that would be a good feature since it is likely that most pc that are used with this will be dedicated nodes and not used for other stuff(in my case that is what they will be)

i am gonna try geting python and octave working on my pc then i may wat to set up a small home cluster with some old'puters probably in the P3-4 256-512mb range.

guess gigabit ethernet would be good too wouldn't it? good idea making unleashed that way there is less chance of the project fizzleing out.

_________________
Taking Puppy Linux to the limit of perfection. meanwhile try "puppy pfix=duct_tape" kernel parem eater.
X86: Sager NP6110 3630QM 16GB ram, Tyan Thunder 2 2x 300Mhz
Sun: SS2 , LX , SS5 , SS10 , SS20 ,Ultra 1, Ultra 10 , T2000
Mac: Platinum Plus, SE/30
Back to top
View user's profile Send_private_message Visit_website 
Everitt

Joined: 19 Dec 2006
Posts: 331
Location: Leeds,UK or Birmingham, UK

PostPosted: Thu 06 Sep 2007, 19:29    Post_subject:  

It would certainly possible to move files onto nodes. I'm already making use of SSH, so there is no reason why you couldn't use the scp command. sfs files do load on a netbooted puppy, as long as you have a save file on that machine, just like from the CD. This does of course require nodes to have disks, and involves copying the file to each indervidual node one at a time (or with a script, still in a linear, therefore slow, manner.) It's good idea though, if you plan to use the same nodes time and time again. I'll add a script to do that to my ever growing todo list.

I'm planning to add the ability to poweroff or reboot node using MPI. If you try to run the commands directly then they never return 0 (how could they?), so you have to wait for it to time out on the server. A couple of simple binaries should fix that though. Yet another thing on my todo list. Smile

Gigabit makes transfers faster, but actually has a slightly higher latency, so the benifits actually depend on the code you run. I believe that overall it is a big improvement, but then again it's still nastily expensive. Sad
Back to top
View user's profile Send_private_message 
cb88


Joined: 28 Jan 2007
Posts: 1169
Location: USA

PostPosted: Thu 06 Sep 2007, 23:04    Post_subject:  

it may be costly as an pci board but many computers even old ones (p4 cerca 2002) have gigabit. is it not possible to have multiple links between computers perhaps have both 100mb and gigabit(that really would be expensive unless you found some for cheap)? still haven't gotten octave i suspect that it may be the old gcc puppy 2.16 uses 2.20 will have 4.x so maybe i will try again with it.

also 2.20 may come in a new kernel and old kernel versions so you could probably use most of the same stuff in 2.20 with 2.18x kernel

i really have no experience with net booting at all.

i did notice that python was very big when i compiled it the .pet ended up being 5.3mb

_________________
Taking Puppy Linux to the limit of perfection. meanwhile try "puppy pfix=duct_tape" kernel parem eater.
X86: Sager NP6110 3630QM 16GB ram, Tyan Thunder 2 2x 300Mhz
Sun: SS2 , LX , SS5 , SS10 , SS20 ,Ultra 1, Ultra 10 , T2000
Mac: Platinum Plus, SE/30
Back to top
View user's profile Send_private_message Visit_website 
Everitt

Joined: 19 Dec 2006
Posts: 331
Location: Leeds,UK or Birmingham, UK

PostPosted: Fri 07 Sep 2007, 06:18    Post_subject:  

The switchs are also expensive, which is what put me off. I guess I'm just cheap. Razz
Multiple NIC's would be somewhat complicated to set up, and even harder to get an app to use them seperately for different data loads. I'd suggest just using what you have, chances are you wouldn't really notice a diference anyway. I don't know, I don't have any real experiance.

You shouldn't really need much. Just boot the CD in on computer, then on the nodes all you need to do is ensure they're set to boot from the network. If there isn't an option in the bios (pre 2001 machines) you can get boot floppies.
Back to top
View user's profile Send_private_message 
cb88


Joined: 28 Jan 2007
Posts: 1169
Location: USA

PostPosted: Fri 07 Sep 2007, 17:13    Post_subject:  

sounds easy enough

if you are in the us this might be a good buy even though everitt must already have a switch

http://www.newegg.com/Product/Product.aspx?Item=N82E16833180028

_________________
Taking Puppy Linux to the limit of perfection. meanwhile try "puppy pfix=duct_tape" kernel parem eater.
X86: Sager NP6110 3630QM 16GB ram, Tyan Thunder 2 2x 300Mhz
Sun: SS2 , LX , SS5 , SS10 , SS20 ,Ultra 1, Ultra 10 , T2000
Mac: Platinum Plus, SE/30
Back to top
View user's profile Send_private_message Visit_website 
Everitt

Joined: 19 Dec 2006
Posts: 331
Location: Leeds,UK or Birmingham, UK

PostPosted: Fri 07 Sep 2007, 17:49    Post_subject:  

Well, I can do it, so yeah, it's got to be pretty damned easy. Smile

Progress report: Work on the server has reached a point where it is now possible to create a working CD. I hope. It works booting the server itself over PXE. with a little fiddeling for reasons of filesize. No reason why a CD shouldn't work. I'm calling this a milestone anyway. Now it's a matter of tidying up a few things, rewriting the unleashed scripts, and intagrating that. First up though I'm going to try and write a few scripts/binaries I've been thinking of to make using it a bit easier.
Back to top
View user's profile Send_private_message 
cb88


Joined: 28 Jan 2007
Posts: 1169
Location: USA

PostPosted: Fri 07 Sep 2007, 22:43    Post_subject:  

like i said i tried compiling octave again and it said that a sanity check failed i tried to compile a program i had already packaged and it did the same so i guess my devx is messed up i'll fix it tomorrow i guess

also there is a version of octave for mpi or is it a patch it seems that octave is a near exact clone of command line matlab but i am not sure

my calculus professor said he would give the windows version a look see "happily" he still uses win 98 and 2000 and there is hope he will never use vista!


here is octavempi if you haven't seen it already: https://sourceforge.net/projects/octavempi/

_________________
Taking Puppy Linux to the limit of perfection. meanwhile try "puppy pfix=duct_tape" kernel parem eater.
X86: Sager NP6110 3630QM 16GB ram, Tyan Thunder 2 2x 300Mhz
Sun: SS2 , LX , SS5 , SS10 , SS20 ,Ultra 1, Ultra 10 , T2000
Mac: Platinum Plus, SE/30
Back to top
View user's profile Send_private_message Visit_website 
Everitt

Joined: 19 Dec 2006
Posts: 331
Location: Leeds,UK or Birmingham, UK

PostPosted: Fri 07 Sep 2007, 23:09    Post_subject:  

cb88 wrote:
here is octavempi if you haven't seen it already: https://sourceforge.net/projects/octavempi/

There is also MPITB. Seems more mature to me, or at least to have better documentatiopn; and itseems to have more suitable requirements. (No NFS.)
Who knows? It's 4AM, I've spent most of the night scripting. Razz
Back to top
View user's profile Send_private_message 
Display_posts:   Sort by:   
Page 2 of 3 Posts_count   Goto page: Previous 1, 2, 3 Next
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
 Forum index » Advanced Topics » Puppy Projects
Jump to:  

Rules_post_cannot
Rules_reply_cannot
Rules_edit_cannot
Rules_delete_cannot
Rules_vote_cannot
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0825s ][ Queries: 11 (0.0041s) ][ GZIP on ]