Designing Puppy to be a RAM operating system.

[sunburnt]

As said in the other forum post... UnionFS & AUFS both take a fair amount of cpu power & lots of ram to work well.
That said, it should be obvious that if there's another way to do it... GOOD!

[HairyWill]

I just got puppy to boot using multi-session style directories. The only change needed to initrd.gz was to add in a sync in init at about line 945 before

Code: Select all

. $OLDFILESMNTPT/etc/rc.d/BOOTCONFIG

Before I added the sync I was getting a kernel panic.
I haven't tried modifying rc.shutdown yet, but I have written a trivial script that creates a new session directory and saves /initrd/pup_rw into it.

Code: Select all

#!/bin/sh
mkdir -p /mnt/hda2
mount /dev/hda2 /mnt/data
NEWDIR=/mnt/data/$(date '+%Y-%m-%d-%H-%M') 
echo $NEWDIR
mkdir -p $NEWDIR
cp -r /initrd/pup_rw/root $NEWDIR
cp -r /initrd/pup_rw/etc $NEWDIR
cp -r /initrd/pup_rw/usr $NEWDIR
cp -r /initrd/pup_rw/bin $NEWDIR
cp -r /initrd/pup_rw/sbin $NEWDIR
cp -r /initrd/pup_rw/opt $NEWDIR
cp -r /initrd/pup_rw/lib $NEWDIR
cp -r /initrd/pup_rw/mnt $NEWDIR
cp -r /initrd/pup_rw/var $NEWDIR
cp -r /initrd/pup_rw/archive $NEWDIR
rm  $NEWDIR/etc/.XLOADED
umount /mnt/data

It will need some logic in rc.shutdown to work out whether the session should be burnt to CD or saved to disc.
Here is what my PUPSTATE looks like

Code: Select all

PUPMODE=77
PDEV1=''
DEV1FS=''
PUPSFS='hda2,ext2,/pup_400.sfs'
PUPSAVE='hda2,ext2,/2008-06-20-01-01'
PMEDIA=''
SATADRIVES=''
#these directories are unionfs layers in /initrd...
SAVE_LAYER='/pup_ro1'
PUP_LAYER='/pup_ro2'
#The partition that has the pup_save file is mounted here...
PUP_HOME=''
#(in /initrd) ...note, /mnt/home is a link to it.
#this file has extra kernel drivers and firmware...
ZDRV='hda2,ext2,/puppy-4.00/zdrv_400.sfs'
PSWAPFILE=''
PSAVEMARK=''

If I had realised it was this easy I would have done this ages ago.
Minor downside. Once you create 2008-06-23-09-01 style directories in a partition you cannot boot any frugal install on that partition because the boot process automatically uses the multisession stuff

[sunburnt]

Hi HarryWill; How's it goin'?
An image file could be used on the partition to separate the partition from it of course.

Have you looked at the other forum discussion I pointed out?
It uses initramfs for the / file system & links to SFS & image files, & also partitions.
So the real files can be anywhere on an any type of device with any file system.
It splits up the big Puppy SFS file into smaller parts so apps. can be quickly added to them.
If there's space the SFS & image files can be loaded into ram individually with a Boot Manager type GUI.
And as suggested here, the settings are saved by themselves (~ 8 MB),
probably in a image file but could be directly on a partition (no backup & restore).
Have a look & tell me what you think...

http://www.murga-linux.com/puppy/viewto ... 709#208709

[HairyWill]

hi sunburnt, I'm still digesting your post.

I found the cause of the kernel panic. In init there is

Code: Select all

#older pup_save.2fs <v2.16 will not have this file...
[ ! -f $OLDFILESMNTPT/etc/rc.d/BOOTCONFIG ] && mkdir -p $OLDFILESMNTPT/etc/rc.d && touch $OLDFILESMNTPT/etc/rc.d/BOOTCONFIG
#note, OLDFILESMNTPT can also be "" so BOOTCONFIG needs to exist in initrd also.
. $OLDFILESMNTPT/etc/rc.d/BOOTCONFIG #can have EXTRASFSLIST variable.

the touch fails if /etc/rc.d doesn't exist, so I changed it to

Code: Select all

[ ! -f $OLDFILESMNTPT/etc/rc.d/BOOTCONFIG ] && mkdir -p $OLDFILESMNTPT/etc/rc.d && touch $OLDFILESMNTPT/etc/rc.d/BOOTCONFIG

alternatively if you don't want to modify init you can boot multisession style by creating a directory structure similar to this /2000-00-00-00-00/etc/rc.d/ inside the partition you want to boot.

This stuff only works on a linux filesystem, ie not fat or ntfs (no symlinks)

Some rather tentative use cases:
forensic analysis of which files change in which sessions
running entirely in ram
multisession on usb storage allowing the usb storage to be disconnected or used for something else after boot (as with multisession CD)
choice of whether to save session or not

I still need to work on rc.shutdown.

[erikson]

I'm hacking

I notice that PUPMODE is set in the init script (from initrd.gz) and is used in the rc.shutdown script.

Are these the only scripts in Puppy wherein PUPMODE is used, or are there still other ones?

Re. unionfs and aufs, is there "recommended reading" explaining basic concepts and implementation? I mean in general, not restricted to its usage in Puppy?

[Bruce B]

Timothyli,

{cut}

How can we reduce the size and contents of pup_save to its absolute minimum so that it loads very fast in RAM during startup.?

{snip}

I really appreciate fast startups. But, hopefully we spend much more time computing than we do starting.

Reducing the size of the pup_save and size contents are two considerations. Mind if I address them that way.

contents ) the user determines the majority of the contents in the pup_save file. value considerations of the speed tradeoffs are up to the user

the user can also at his discretion have contents which never get loaded to RAM. this would be things like mounted partitions and etcetera.

pup_save size ) if after building the pup_save file, you discover it has more free space than wanted, adjust its size


--------------------------

From another timothyli post and with more cutting and snipping on my part.

To achieve our goals, It is difficult to avoid a save-back to r/w media, or to a server somewhere on session changes.

True.

Just an note from my experiences, once you have Puppy setup as you want, you'd be surprised how seldom you want to save sessions.

In fact I think one of the strong points of this is the NOT saving sessions, although save back method(s) needs to be developed to make it worthwhile.

-------------------------

All of us are running Puppy sessions. Things are changing during these sessions. Maybe the user at poweroff time can ask himself: Is there anything from this session I want to keep?

The answer will vary, sometimes yes and sometimes no.

In any event, the answers will give the user fair statistic regarding how often he even wants changes made to his Puppy.


Bruce

[Pizzasgood]

I knew rsync worked with single files (though i hadn't thought of using it directly on the pup_save.2fs files themselves, good idea!) What I was wondering about was using rsync to update the contents of a .tar file to match the contents of a normal directory, without extracting the .tar. I don't think you can do that, but I haven't had time to dig up any info on it yet.

[sunburnt]

Hi Pizzasgood; Here's tar help:

-A, --catenate, --concatenate append tar files to an archive
-c, --create create a new archive
-d, --diff, --compare find differences between archive and file system
--delete delete from the archive (not on mag tapes!)
-r, --append append files to the end of an archive
-t, --list list the contents of an archive
--test-label test the archive volume label and exit
-u, --update only append files newer than copy in archive
-x, --extract, --get extract files from an archive

Of paticular interest is -d & -u
Hope it helps... Terry

[Bruce B]

FYI - I've not lost interest, quite to the contrary. I've been working very hard or R&D and this is most time consuming. I probably won't make a write up until I have some wrinkles ironed out.

Also, I never learned the intricacies of Frugal because I've been a pretty avid Full Install user.

I really do appreciate others expertise and interest. Not to mention names, because I don't want to leave anyone out.

[HairyWill]

fixed the save for multisession folders
I replaced multisavefunc in /etc/rc.d/functions4puppy
with

Code: Select all

multisavefunc() {
dialog --yes-label "SAVE" --no-label "NO SAVE" --yesno "Press ENTER to save session...
Or, TAB then ENTER to not save ..." 0 0 >$DEVCONSOLE
if [ $? -eq 0 ];then
	mount /dev/$SAVEPART /mnt/data
	NEWDIR=/mnt/data/$(date '+%Y-%m-%d-%H-%M')
	echo -n "Saving session in $NEWDIR" >$DEVCONSOLE
	mkdir -p $NEWDIR
	cp -vrx /initrd/pup_rw/archive $NEWDIR
	cp -vrx /initrd/pup_rw/bin $NEWDIR 
	cp -vrx /initrd/pup_rw/dev $NEWDIR 
	cp -vrx /initrd/pup_rw/etc $NEWDIR 
	cp -vrx /initrd/pup_rw/lib $NEWDIR 
	cp -vrx /initrd/pup_rw/mnt $NEWDIR 
	cp -vrx /initrd/pup_rw/opt $NEWDIR
	cp -vrx /initrd/pup_rw/root $NEWDIR
	cp -vrx /initrd/pup_rw/sbin $NEWDIR 
	cp -vrx /initrd/pup_rw/usr $NEWDIR 
	cp -vrx /initrd/pup_rw/var $NEWDIR
	echo " Finished saving session" >$DEVCONSOLE
	rm  $NEWDIR/etc/.XLOADED
	umount /mnt/data
else
	echo "Session not saved." >$DEVCONSOLE
fi
}

If I was clever I would have detected whether it was multisession CD or hdd and not had to delete the original multisession save. This will ask on shutdown if you want to save the session. If you hit the save button on the desktop you will get a session saved but it will not clear out /initrd/pup_rw so you will end up with duplicates on shutdown, probably not a big problem but wasteful on space.

[sunburnt]

HarryWill; I'm sure there's an easier way than using the MultiSession stuff, like Pizzasgood's doing.
But perhaps the MultiSession code isn't very complicated & lends itself to this.

The proposed changes to Puppy's file system setup splits up the "static" dirs. & the "dynamic" dirs.
The static ones would be SFS files & the dynamic ones in initramfs, a HD partition, or an image file.
The static ones aren't written to in a normal session, only when installed to.
Dynamic ones are written to every session, this doesn't change the use of rsync but does simplify it..

[timothyli]

I finally read the "How Puppy Works" page from BarryK's site and I was very impressed by its smart, sensible, and practical design.

I came to the conclusion that the current design may be "as-good-as-it-can-get" in terms of overall user experience, given a certain amount of RAM.

We may achieve the technical goal of running everything entirely in RAM (which the current design can "almost" do with pfix=ram), but I don't believe this would translate to any significant overall improvement in user experience, given the same amount of RAM. The (perhaps) faster access to certain apps and data would be traded off by slower load and save times.

If one wants to access a certain app in RAM, one could always remaster the system to ensure it is loaded in RAM.

Nevertheless, this is a very interesting problem to solve.

How about copying the pup_save to a RAM disk and then mounting it using unionfs as a layer?

[sunburnt]

timothyli; That's what Pizzasgood did above...

However... The proposed changes to Puppy would eliminate UnionFS & AUFS, this would improve a number of things.
Faster operation (less cpu intensive), more reliable operation, & way less ram usage.
The "Q & A" here is,,, If you can do the same thing without unions, then you've eliminated an unneeded complexity.

[HairyWill]

HairyWill; I'm sure there's an easier way than using the MultiSession stuff, like Pizzasgood's doing.
But perhaps the MultiSession code isn't very complicated & lends itself to this.

It is extremely simple.
It is possible to get this scenario working without modifying initrd.gz. I used to use multisession for several months when I first got interested in puppy. My current laptop doesn't have an optical drive, so I am partly motivated by being able to use puppy this way without needing a CD drive. It also provides a rough and ready file versioning system because files are never deleted from previous sessions.

Multisession is one of the capabilities that really makes puppy stand out.
My impression is that the multisession userbase is quite small. Maybe this development will increase the userbase by at least one and provide an easier mechanism for testing. Considering how easy it was to implement it wouldn't surprise me if Barry is reading this thread and chuckling because he already does testing this way.

The single layered filesystem seems much more revolutionary, I look forward to seeing further developments.

[Aitch]

Hi Guys

Can I resurrect an idea I threw up yonks ago, when fastboot was being discussed, whilst all this design theory is etheric?

My suggestion was more to do with the actual hardware probe phase, which every OS goes through each & every time I hit the on switch

If it's the same PC I turned off last night, why can't the probe begin with probing me
"Hello Aitch, did you change any hardware?"
If I answer no, it reloads an image without checks > 2seconds? <

[up to] 5 seconds later I'm using it ~~~

Similarly with shutdown
"Did you make any changes?"
no? bang - off

or am I dreaming?

I believe a hard drive is the only thing requiring a delay/write on shutdown

Chips is chips, yes/no?

Sorry I don't have the knowledge that's going on here, but I sure like the sunburnt/PG/BruceB/HairyWill/erikson/timothyli discussion

Aitch

[timothyli]

...I know this is a bit off topic.

I totally resonate with Aitch about boot time.

I use an ASUS EeePC 4G (RAM upgraded to 2G) . It boots in 23 seconds using the default customized Xandors OS!

I boot pupeee (based on Puppy 3.01) off an 8Gig SDHC card with a 512meg pup_save.2fs and devx and kernel_src SFS. It takes one minute flat.

Of course the built-in Solid State HD is faster than SD card and it is not an apples-to-apples comparison. However, it would be nice to shave some seconds off booting puppy, which is my day-to-day OS.

I think the new Puppy version 4 has the "whether-to-probe" option Aitch was talking about in the boot config panel. I stay with version 3.01 because I did not find the new s/w Install tool work as slick as the old one.

[erikson]

However, it would be nice to shave some seconds off booting puppy, which is my day-to-day OS.

Hint: have you already commented out the serialdetect section (for serial mouse & modem) in /etc/rc.d/rc.local0 ?

[erikson]

One more consideration re. loading or not loading in RAM.

The objective of fast execution at runtime (by having everything available in RAM) is at least partially at odds with the objective of fastboot.

At boot, physically copying the whole filesystem into RAM may take significantly more time than just mounting (with or without unioning) various parts of it.

Actually we would need objective benchmark tools to measure timing & performance for the various options on a mix of typical hardware. Are such tools available in Puppy?

[Bruce B]

Progress Report

I've been working on the project for several days. I want to
give a somewhat detailed report on what I've done, the
approach I've used, how its going, and other things.

Puppy pfix=ram (thoughts about)

This puts your entire operating system in RAM and provides
about all the durability one could ask for.

As a permanent way of running Puppy it has some
drawbacks, namely that changes are not saved and it takes
a little time to setup the graphics, network and maybe
even the printer. Also, on the reboot you have to opt not
to make a save file (or make one).

One fairly easy way to make it more versatile is to mount a
device and save downloads and user files to that device.

One of the things I don't like about running pfix=ram is
that I want more software, programs, utilities, aliases and
scripts than come with Puppy.

I could work around this by having these things available
and import them on a session by session basis.

A drawback to this approach is it might take several
minutes to set things up for each session.

Modifying Pup_400.sfs

This is where I've been putting most of my energy, once I
decided this might be the most practical way for me. Here
are the basic steps outlined:

The Setup

Two boot options, one for normal booting and one to run in
ram. The reason for the ram option is basically for deleting
the pup_save.2fs file when the time is right, also filesystem
checking on unmount partitions.

I run in normal frugal mode, that is with a pup_save file. I
enhance Puppy suit my tastes.

After a number of enhancements, I run some scripts which
mount the pup_400.sfs and pup_save.2fs files and copy the
contents to two directories. I clean up the copied pup_save
directory, deleting the whiteout files and other things I
know should not be copied to the pup_400 directory.

Once pup_save cleanup is done, I run another script which
copies the pupsave directory to the pup400 directory.
Then a new pup_400.sfs is made. The script backs up the
existing one and installs the new one.

I boot to RAM and delete the pup_save.2fs file, because
whatever data was in it is either not wanted or now
installed in pup_400.sfs, then make a new pup_save file as I
reboot.

I'm at the point now where the pup_400.sfs boots and my
screen resolution, mouse and Internet connection is
pre-setup. My many enhancements are included in the
pup_400.sfs.

When to do away with the pup_save ?

Maybe I won't ever do away with the option to make one.
The reason why is, having a pup_save makes it very easy to
add applications and modify the Puppy. On the other hand I
can presently run in the absence of one without any of the
setup and saving problems associated with the generic
pfix=ram (or booting with no pup_save)


Is it too much work?

After things are configured as I like it's not any work at all.

The majority of the work was developing a system for
making changes. After the system and scripts were built, it
seemed pretty straight forward and simple. Also when one
has a genuine interest in something it's more fun than
work.

Of course, it's the benefits I'm after really

An operating system that can take the heat. Even if it were
exploited, compromised, or corrupted in a given session,
on the next boot it wouldn't matter because it's as clean as
it was when I started the previous session.

(Not to mention high speed of a full OS in RAM)

Conclusion

All I'm doing here is sharing what I've been doing and the
results I've achieved. I'm certainly not trying to say it's the
way or the best way. I'd have know idea about that. Except
to say - it's a way.

[erikson]

All I'm doing here is sharing what I've been doing and the results I've achieved. I'm certainly not trying to say it's the way or the best way. I'd have know idea about that. Except to say - it's a way.

Very good Bruce, thanks for sharing.

I notice with satisfaction some similarity with what I'be been doing lately (see How to make Puppy bulletproof?) following HairyWill's suggestion --- i.e. merge pup_xxx.sfs with pup_save.2fs into a new pup_xxx.sfs (with all personalization included), and then one can boot with pfix=ram.

Didn't I say we're roughly on the same page