Puppy for parallel computation using openMPI

A home for all kinds of Puppy related projects
Message
Author
Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

Puppy for parallel computation using openMPI

#1 Post by Everitt »

A week or so I started to look into parellel code execution on clusters of standard computers. Being a bit of a computer geek, and having a few old computers laying about the place I thought I'd have a go at it. Like a good little nerd I read through some tutorials and began to digest the basics of writting C to be executed using openMPI. Seeing as I'm still a total beginer at C, and have no experiance you'd have thought that that would be enough. Nope. Not me. :D I decided that it would be far more fun to train by beloved puppy to run my code for me.
After maybe a week of work I'm maybe a few days away from burning my first CD. Everything is still very much in the proof of concept stage, but I have had this working, and I'm not too far away from producing something that'll work 'out of the box'
Inital steps to use this:
  • Set up a SIMPLE network, just some computers and a switch. No servers, and, importantly no outside connections.
  • Boot one of the machines from the CD
  • This will set up a PXE boot server, so boot the other PC's from there
  • At this stage you should have a cluster of PC's ready to run code.
ok, what you would now have in front of you is, in effect, a pile of old hardware. :) But, and here's the important thing, if you're a geek like me you can (just about) justify calling that pile of old hardware a 'scalable supercomputer.'
Yep, that's right, supercomputer. :D
Admit it, that got you're attention.
Ok, so to actually qualify as a supercomputer proper you'd need a LOT of expensive computers, not just the old junk you found in a skip somewhere, but I did say scalable!
I admit it, I'm joking around. I don't intened for this thing to ever actually be a proper clustering solution, I just want an enviroment to teach myself something about MPI, and I though, if I'm going to do this, why not have a little fun? :P When I think of a supercomputer I think of big shiney things with flashy lights and steam errupting everywhere. I think of the cutting edge of modern computing.
When I think of puppy I think of a small, fast, light OS perfectly suited to running on older, slower computers. I just love the idea of running it on a supercomputer! What else other than an OS callled puppy could make that steaming beamoth of modern computing cutter! What machine could be better suited to the low requirements of Puppy.

I guess by now you've all guessed I'm a teenager with a tendency to exagerate and a love of irony. I would like you all to know that I do yes know that anything I could set up in my bedroom is a massive distance from a supercomputer, and that what I'm trying to do with puppy has been done many times with OS's far better suited to the job.
I just thought it could be amusing. Puppy and supercomputers? Why not!?
If anyone can come up woth a good name for this then perhaps we could use it to attract a bit of attention puppies way?
Anyway, if anyone actually cares, I should be able to get an iso uploaded in the next few days, if I get time to work on it.
Last edited by Everitt on Wed 29 Aug 2007, 19:43, edited 1 time in total.

User avatar
Lobster
Official Crustacean
Posts: 15522
Joined: Wed 04 May 2005, 06:06
Location: Paradox Realm
Contact:

#2 Post by Lobster »

Matrix Pup

:)
Puppy Raspup 8.2Final 8)
Puppy Links Page http://www.smokey01.com/bruceb/puppy.html :D

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#3 Post by Everitt »

Ok, be honest with me, do you think I'm nuts? :)
Sorry, I'm just having a little too much fun with the idea of a HPC (High performance cluster) running something called puppy. :)

Matrix pup... I like the sound of that. :)

User avatar
Lobster
Official Crustacean
Posts: 15522
Joined: Wed 04 May 2005, 06:06
Location: Paradox Realm
Contact:

#4 Post by Lobster »

Everitt wrote:Ok, be honest with me, do you think I'm nuts? :)
The imaginary psychiatrist I keep in the cupboard, says I am not qualified to give an answer.
Sorry, I'm just having a little too much fun with the idea of a HPC (High performance cluster) running something called puppy. :)
:D Keep up the good work
Matrix pup... I like the sound of that. :)
Image

You are the One, Neo
Puppy Raspup 8.2Final 8)
Puppy Links Page http://www.smokey01.com/bruceb/puppy.html :D

User avatar
Dougal
Posts: 2502
Joined: Wed 19 Oct 2005, 13:06
Location: Hell more grotesque than any medieval woodcut

#5 Post by Dougal »

I think it's a fine idea and you seem to be spending your time on better tings than 90% of the people on this forum.

Keep us updated.
What's the ugliest part of your body?
Some say your nose
Some say your toes
But I think it's your mind

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#6 Post by Everitt »

I really need to learn that trying to write things at 2AM isn't the best of ideas. Sorry about that :D

Thanks for the pic, I love it, and it could just be the solution to a problem I haven't thought about yet. :)

Not much progress I'm afraid, I've been roped into washing the garage roof today, and probably painting it tomorrow. It's really irritating, I'm only a few steps away from something I properly test, and I've got no time to do anything!

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#7 Post by Everitt »

Happily enough we can't get the ladders out, so I got a roof free day. Unfortunately I didn't make as much progress as I'd have liked. However, I do think I'm getting somewhere, finaly. :)

This would be so much easier if I could work out how puppy works. It seems like process' I start from /etc/profile aren't starting, or something...
I'll probably start a new thread if I can't work it out soon

User avatar
Lobster
Official Crustacean
Posts: 15522
Joined: Wed 04 May 2005, 06:06
Location: Paradox Realm
Contact:

#8 Post by Lobster »

any use?
http://www.puppylinux.com/development/h ... works.html

I am replacing parts of a floor (woodworm) - joists and planks.
:shock:

Neo: I know you're out there. I can feel you now. I know that you're afraid... afraid of us. You're afraid of change. I don't know the future. I didn't come here to tell you how this is going to end. I came here to tell how it's going to begin. I'm going to hang up this phone, and then show these people what you don't want them to see. I'm going to show them a world without you. A world without rules or controls, borders or boundaries. A world where anything is possible. Where we go from there is a choice I leave to you.
Puppy Raspup 8.2Final 8)
Puppy Links Page http://www.smokey01.com/bruceb/puppy.html :D

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#9 Post by Everitt »

Unfortunately not, it stops before reaching the part of the startup that I'm interested in. Facinating though. *Goes back to read more*

My day today was filled with emulsion, so I didn't get anywhere. For some reason sshd stopped working. I guess it's probably a permissions error (I made the mistake of copying everything onto a Samba share, and in doing so lost all the permissions.)
Other than that I shouldn't be too far away from something that works. *Crosses fingers*

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#10 Post by Everitt »

Well, I wasn't that far away, then my hdd lost it all. Well, some of it. I've been too busy to do anything about it though, so it's probably going to be a while before I get anything working again.
To be honest I was going to have to redo a fair amount at some point, so why try to hard to recover lost data?

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#11 Post by Everitt »

Ok, so I've been doing some thinking. I've been pretty busy of late. Well, not really, I've been don't next to nothing, but it feels like I'm busy. I've been sleeping a lot to be honest, but that's not the point. :D The point is that whenever I've been looking to do some work on this I've been given some other job to do.
Part of me want to get something, anything out, preferably in the next two weeks. That might not be so easy now I've lost everything. I'm going to university, so things are going to get a whole lot busier, getting ready, and then of course, once I'm there I'm not going to have a lot of free time for a while.
Now I origionally posted this for a joke, but I started it because I was interested. If I'm going to do this then I want to end up producing something that's the way I want it to be. I take a pride in my work, and I don't want to make comprimises where I don't have to. The origional plan was to quickly bash out a proof of concept, based on a single image. Now this is, I believe, how the other distrobutions I've seen that do this work. It was never what I invisioned though. I just wanted to prove to myself that it could be done, and that it would work under Puppy. This plan was somewhat squashed when I lost a fair amount of progress. In working getting that version to work I've pretty much convince myself that it could work, so why mess about with an interim test?
So then, I haven't forgotten about this, I haven't given up, but I doubt there will be anything much happening for a while.

Is anyone actually interested? Shall I post notes as to how I want it to work, and progress reports, or shall we just let the joke die here?

Trobin
Posts: 968
Joined: Fri 19 Aug 2005, 03:16
Location: BC Canada

#12 Post by Trobin »

Don't let it die.

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#13 Post by Everitt »

Hokay. No saying when I'll find time though. :(

eriefisher
Posts: 22
Joined: Sun 26 Aug 2007, 17:27

#14 Post by eriefisher »

I say keep going.

I think the name should be "Wolf Pack"

eriefisher

User avatar
Lobster
Official Crustacean
Posts: 15522
Joined: Wed 04 May 2005, 06:06
Location: Paradox Realm
Contact:

#15 Post by Lobster »

:) We will have available Slack binary code compatibility as 2.20 evolves

Are there java multi processor programs or suitable Slack code?
Puppy Raspup 8.2Final 8)
Puppy Links Page http://www.smokey01.com/bruceb/puppy.html :D

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#16 Post by Everitt »

I'm going to stick to 2.16 while I get a few things set up properly. I had tried 2.17, but nobody seems to be able to make the humungous initrd.gz boot propely.
As for existing code, I don't really know. OpenMPI has support for C[++] and fortran. Java should be possible, there are open-source projects avaliable, but the trouble is that to use java you need the java runtime enviroment. According to the Java Parallel Processing Framework FAQ you'd need a good 512MB of RAM. Not very puppy huh?
For comparision, a standard netboot of 2.16 needs somewhere between 128MB and 192MB (192 works, 128 doesn't). I plan on stripping a lot out for the nodes, so that should fall a bit too.

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#17 Post by Everitt »

I spent most of today putting together the node version. I've got it working, but by the look of it there is a reasonable amount in the root filesystem that I don't need/want, so cleaning that out is going to take a while. :(
Anyway, free reports a total of 63568KB used (62MB)

User avatar
cb88
Posts: 1165
Joined: Mon 29 Jan 2007, 03:12
Location: USA
Contact:

#18 Post by cb88 »

that is great about the 62mb ram(is that ram usage guess it is) usage hope you can get it even lower might be worth looking at how ipup or icepup and barebones are set up to try taking some stuff out
Taking Puppy Linux to the limit of perfection. meanwhile try "puppy pfix=duct_tape" kernel parem eater.
X86: Sager NP6110 3630QM 16GB ram, Tyan Thunder 2 2x 300Mhz
Sun: SS2 , LX , SS5 , SS10 , SS20 ,Ultra 1, Ultra 10 , T2000
Mac: Platinum Plus, SE/30

User avatar
bobwrit
Posts: 283
Joined: Mon 12 Mar 2007, 23:33
Contact:

#19 Post by bobwrit »

This project is perticualary interesting to me right now. I find supercomputer networks intreaging. c's also my specialty language(I have exp. in pascal,basic,python, and java.). At your local thirft store in America you can get a computer for $3. :wink:
I need help with my forum. [b][u]LINK:[/u][/b][url]http://www.programers.co.nr/[/url]
[url]http://www.freewebs.com/programm/iframe.html[/url] is my gateway page...

Everitt
Posts: 331
Joined: Tue 19 Dec 2006, 21:59
Location: Leeds,UK or Birmingham, UK

#20 Post by Everitt »

cb88 wrote:that is great about the 62mb ram(is that ram usage guess it is) usage hope you can get it even lower might be worth looking at how ipup or icepup and barebones are set up to try taking some stuff out
Yep, at the moment it uses just over 62MB of RAM. Hopefully I should be able to lower that some by cleaning out things I'm not using in the root filesystem package, and by removing some modules from zdrv. (No point having graphics of sound drivers for example.) I'll certainly take a look at those varients, thanks for the tips. :)
bobwrit wrote:This project is perticualary interesting to me right now. I find supercomputer networks intreaging. c's also my specialty language(I have exp. in pascal,basic,python, and java.). At your local thirft store in America you can get a computer for $3. :wink:
What, pay for a computer? Never! :P I'm in the UK anyway.
I plan to post something usable at some point, but I warn you, it probably won't be soon and it will me rather restriced in what it can do. If you want to try something like this I'd recoment parallelknoppix. That at least has something you can download!

Success! Just as I was typing this I got a positive result, a node booting and running a job without any setup on my part. That leaves me a single test away from declaring it usable and starting work (again) on the server. (aka. the hard bit.)

Post Reply