Puppy 245mpi cluster made easy

[Master_wrong]

@gcmartin

Thanks for clarifying everything you've done thru-out this Thread. I have followed you work religiously. Very clear objectives, approaches, conclusion.

I have one complex request.


* Would you setup your LiveCd to automatically start the PXE environment when it boots without user intervention.
* Would you post the LiveCD/LiveDVD that would boot the Cluster Master starting the PXE environment with those files necessary for the cluster nodes to PXE from, please.

This would make an automatic (another word for dummies like me) cluster where all I do is boot "Live" and start nodes, wthout ever typing or clicking on anything, to begin and run my cluster.

You may want to start a new Thread announcing your Multi PC "ClusterPUP".
See my PM to you.

well i can put netboot folder into root which will be saved during remaster, but the iso size will be enormously 600+M

what's important is that now it can boot headless pc. but you still need to click

@Aitch Smile

Anyone doing video/high intensity graphics/blender will benefit from this, however, can compiling be speeded up on a cluster, it is common to hear it taking 10/12 hrs?

here i test mpir...
it seems good

# mpirun -np 2 ~/mpir -s 101 -e 1000 'blender -b in.blend -s %start -e %end -a'
sh: blender: command not found
sh: blender: command not found
^C

# mpirun -np 2 ~/mpir -s 101 -e 1000 'dvdauthor -b in.blend -s %start -e %end -a'
DVDAuthor::dvdauthor, version 0.6.12-alpha.
Build options: gnugetopt magick iconv freetype
Send bugs to <dvdauthor-users@lists.sourceforge.net>

ERR: Must first specify -t, -m, or -x.
DVDAuthor::dvdauthor, version 0.6.12-alpha.
Build options: gnugetopt magick iconv freetype
Send bugs to <dvdauthor-users@lists.sourceforge.net>
^C

off course i didnt install blender and i dont know howto run dvdauthor...
this is called renderfarm i guess
Anyone willing to test ? [/quote]

[Master_wrong]

I have test Mpich2-132 on 5.2.
.
i have install required fortran compile and install it also run few test.

it seem new mpich didnt use mpdboot anymore, but hydra and what it need is just list of ip...
so i might have to create new script again.
edit... false alarm i didn configure it properly

[gcmartin]

Don't know if you are aware, but, SAMBA has been used om Cllusters for awhile. You may find this use a lot in cluster community. See here for starters. then you should join a SAMBA forum where a large community of user are available that will help.

See SAMBA cluster reference here and the Cluster support covers here.

Hope this helps

[Master_wrong]

Thank gcmartin, it's more like database solution for cluster and i bet the DB stands for Database...

Requirements
CTDB relies on a clustered filesystem being available and shared on all nodes that participate in the CTDB cluster. This filesystem must be mounted and available on all nodes in the CTDB cluster.

On top of this cluster filesystem, CTDB then provides clustered HA features so that data from the clustered filesystem can be exported through multiple nodes in the CTDB cluster using various services. Currently included with CTDB are the necessary hooks for Samba, NFS and ftp exports. Support for new service types can easily be added.

but sure very useful for some cluster application.

[gcmartin]

If you are familiar with sharing files from one Windows PC with the other PCs on the LAN, then this allows ClusterPUP to become a PC that can also share on the LAN, same as all Windows PCs.

Thus, ClusterPUP can not only "mount" Windows share resources, but with CTDB, it can make available folders and files for LAN PCs to read and write to for the cluster.. Now, as a simple example, I can load data, for cluster management/applications, directly from somewhere else on my LAN, AND, I can run applications within my cluster where the resulting data can be viewed, used, manipulated, or managed by things on my LAN. All this done with only lifting a finger!

Think of it as a "secure" data sharing gateway for your cluster.See this and PDF reference at bottom
and this wiki

Hope this helps

[Master_wrong]

great additional info,

btw unfs3 can be configured with cluster, which means client can read and run program which stored in server.

otherwise we can also use glusterfs and cnfs
http://www.gluster.com/community/docume ... figuration

http://clusternfs.sourceforge.net/Presentation.html

[gcmartin]

[quote="Master_wrong"]great additional info,

btw unfs3 can be configured with cluster, which means client can read and run program which stored in server.

Here's another note on cluster SAMBA use: NO Microsoft PC delivers with nfs or dfs. A simple user KNOWS how to use Network Neighborhood to find, read and write to LAN locations.

That's the primary reason any of us needs SAMBA. Remember, 99.44% of ALL of the worlds LANs have Microsoft PC users operating on their LANs.

This does NOT mean that Microsoft users can't install something or learn something new, it only means that I don't have to teach any Microsoft user anything about reading and writing information using the "out-of-box" Microsoft OS on their PCs. Thus, if you have ANY Linux PC with SAMBA, all systems "see" each other on the LAN without any changes in Microsoft.

Thats the only reason I bring this to our attention.

Hope this helps.

[Master_wrong]

@ gcmartin...

That's the primary reason any of us needs SAMBA. Remember, 99.44% of ALL of the worlds LANs have Microsoft PC users operating on their LANs.

sure for casual work which involve windows i agree but my design for cluster was designed to work entirely closed environment, so there is no windows attached.
but samba capable would be great addition anyway.

[gcmartin]

To EVERYONE
If you haven't seen it, "ClusterPUP - A Very Simple Cluster Implementation" is a great document put together by our very own Master_Wrong.

Great explanation, with diagrams and picturs, which makes it easy for all of us to implement our own High performance cluster! Any of us can, now, take advantage of this and easily get a cluster operational in less than a half-hour.

Thanks

[Aitch]

Hi guys

It's been awhile since this thread got commented.....

I've just come across a new reason for clustering/parallel computing....the green agenda/energy saving

"Parallel Computing Infrastructures (PCI) - computer clusters with many CPUs interconnected via a dedicated network - constitute the preferred option for many Institutions requiring massive computational power. PCI typically run 24h per day, even when not all CPUs are being requested. This leads to unnecessary energy expenditure and associated CO2 emissions. Here we offer a solution saving up to 90% of PCI’s energy consumption during idle time in existing (and future) PCI. It is provided in the form of a free, open source software tool (we call it SPIRIT) which activates-on-demand requested CPUs, leaving all others in a shutdown state. Given the widespread adoption of PCI across the globe, we anticipate significant reductions in CO2 emissions upon implementation of this protocol."

http://www.ciul.ul.pt/~ATP/SPIRIT/


gcmartin/Master_wrong

Any comment as to a Puppy use for this?

Can the Sun Grid Engine implementation be easily adapted to ClusterPup, or other linux clusters?

Aitch

[drspastic]

i think one day we will have a desktop icon in puppy called CLUSTER and when it is clicked it connects automaticly with every other puppypc similarly activated via network and internet globally sharing ram, storage, and cpu cycles. the microsoft buster finally.
please tell, can a cluster run without a single master? can slaves load balance each other? can the greater cluster run the master on a vm?
and most importantly can a cluster work all its power to run a massive virtual machine that users can log into as if on a normal desktop pc and run normal apps without the need for special adaptations?
maybe a massive puppy cluster vm could run xp at a reasonable speed.

[gcmartin]

Hi guys Can the Sun Grid Engine implementation be easily adapted to ClusterPup ... ?
Aitch

Thanks for this post. I can see application; not just with ClusterPUP, but in additional uses....say for PXE. The model that they present runs on a cluster much like you see in the ClusterPup document.

It appears that the applications they offer, employs 20 year old technology to "wake up" a powered off PC to go to work. And, thusly, when no work exist, to go away. Thus, when workloads are low/non-existent, no electricity is used within the complex.

... most importantly can a cluster work all its power to run a massive virtual machine that users can log into as if on a normal desktop pc and run normal apps without the need for special adaptations? ...

What you seem to be asking for is called a "Terminal server". Citrix, MS, IBM have already done this for Business communities. Google has just implemented there ChromeBook technologies which is an extended form of what you ask. Its happening (for awhile now in some cases) all around us, while the term "cluster" is not in your face.

Since many people don't know what a Terminal Server is, go to this thread and click on its enclosed link. You will be allowed to use a Linux version of a Terminal server.....Free for your evaluation as you desire with 500MB of personal storage.

For Everyone
ClusterPUP is a mechanism that all of us can implement within our local spaces using Puppy Linux. See the official "ClusterPUP - A Very Simple Cluster Implementation" Manual authored by Master_Wrong. Its easy to follow, too.

Take 2-3 PCs and the manual, and do this yourself. You'll be impressed with its power.

hope this helps.
P.S. I think the current and OFFICIAL CLUSTERPUP SITE is here.