Loading SFS to RAM Without Requiring 2X RAM (Solved)

Using applications, configuring, problems
Message
Author
l0wt3ch

#31 Post by l0wt3ch »

rjbrewer wrote:That was after comments about Luci not being able to eject disk with 256mb ram. Wonder if "Studio" was based on one of those "problem" versions?
No, it's based on the latest 520.
playdayz wrote:Does standard Lucid 5.2 Live-CD copy to ram on boot in that machine with 512MB ram?

I do remember that you said Studio does copy to ram on a machine with 1GB. Is that still the case?
Yes, and yes.
playdayz wrote:So, In a 512MB machine both the standard Lucid-52 LiveCD copied to ram and so did the remastered 181Mb Live-CD.
Did you try it with the "puppy pfix=ram, copy" options? My test disc also loads into RAM when booted without those options. (I should have mentioned that earlier.)
playdayz wrote:Given these results I think it might be possible that there is something unique about the 170MB test disk that l0wt3ch made.
I only added a few archive files to make the sfs bigger, and nothing else. That was the whole point of the "experiment".
emil wrote:can you do a testrun with the 170 MB sfs and the initrd.gz I supplied ?
I will see what I can do.

Thanks again to everyone.

If the ratio of RAM/size of sfs is "working properly", does that mean it was broken in 511? Because Studio 3.1 was 395 MB, and still loads properly with 512 MB of RAM.

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#32 Post by jamesbond »

l0wt3ch wrote:Thanks for the responses.

The strangest thing for me was that it didn't matter how many extra files I added - whether that was five 47 MB files (235 MB), eleven files (517 MB), or nineteen (893 MB), the resulting sfs was always 170 MB. And upon examination on boot, the files were still there!

Either this is the result of the alien technology given to Barry, or something is seriously wrong. :lol:
Ah, we don't need to give alien some credits for it when mere mortals like us can do it ourselves: http://tldp.org/HOWTO/SquashFS-HOWTO/mksqoverview.html. The pertinent portion of that link says (emphasis is mine):
Notes for default mksquashfs behavior:
- Duplicate files will be removed, so there will be only one physical instance (By the SquashFS 2.x, you can disable the detection/removal of the duplicates with the -no-duplicates option).
If you really want to enlarge your resulting squashfs file, instead of using 1.tar.bz2 multiple times, create your 47M file like this instead:

Code: Select all

dd if=/dev/random of=your-big-file bs=1M count=47
. Do this a couple of times if you need to create multiple files.
I've uploaded the dummy file I used, as well as the resulting isos (test1 is 520, test2 is 511).
. Thanks for this, I tried to run test1.iso with 350MB of RAM, and it loaded direct to RAM - no even pfix=copy required. Hmmm....
Attachments
vbox.png
(63.87 KiB) Downloaded 286 times
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#33 Post by jamesbond »

rcrsn51 wrote:
jamesbond wrote:Thus, when you do frugal install, the behaviour is *not* to load into RAM unless you request it explicitly by adding pfix=copy.
Thanks. All my tests were done with manual frugal installs - not by using the Universal Installer. So I was not using the "pfix=copy" argument in my menu.lst entries.

This is a change in the default behaviour of Puppy that I believe appeared around Quirky 1.3 and has since migrated into Lupu.

In the past, we always said that Puppy automatically loaded into RAM if there was enough space. Now it loads into RAM if there is enough space AND you ask it to.
Yes, I'm surprised also, because as far as I can remember, Puppy has always loaded into ram *unless* I ask it not to. Apparently this behaviour is now reversed for frugal install (it's still true for LiveCD) but I can't pinpoint when the change happened.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#34 Post by jamesbond »

playdayz wrote:
In the past, we always said that Puppy automatically loaded into RAM if there was enough space. Now it loads into RAM if there is enough space AND you ask it to.
I *think* the way Barry sees it is that Puppy copies into ram on first boot (if it can).

The welcome1stboot.htm popup, which we removed from Lucid 5.2, says
As this is the first time that you have started Puppy, he is running totally in RAM!
This is still true because when Puppy runs for the first time (or when it runs with pfix=ram), PUPMODE is 5 and therefore it will try to load into RAM.

cheers!
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#35 Post by jamesbond »

Jim1911 wrote:Is it possible that the compression algorithm recognizes that your 47MB file is duplicated X times and ignores or links the duplications?
Gotcha ! :D mksquashfs does this (to be honest, I don't even know this before this thread).
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

l0wt3ch

#36 Post by l0wt3ch »

jamesbond wrote: Thanks for this, I tried to run test1.iso with 350MB of RAM, and it loaded direct to RAM - no even pfix=copy required.
As I mentioned to Playdayz, above, I know that the 170 MB sfs loads into RAM without any boot options. But when "puppy pfix=ram, copy" is given, it does not!

This suggests that something is broken in regard to the "sfs-loading-to-RAM" feature of Puppy.

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#37 Post by jamesbond »

l0wt3ch wrote:
jamesbond wrote: Thanks for this, I tried to run test1.iso with 350MB of RAM, and it loaded direct to RAM - no even pfix=copy required.
As I mentioned to Playdayz, above, I know that the 170 MB sfs loads into RAM without any boot options. But when "puppy pfix=ram, copy" is given, it does not!

This suggests that something is broken in regard to the "sfs-loading-to-RAM" feature of Puppy.
Ah sorry I didn't notice that post. Anyway - your comment above is for booting from LiveCD? It does load into RAM!

Anyway, I notice something - not sure if it's a typo in the forum here, but the correct syntax is "pfix=ram,copy" (there is no space between the comma and "copy" - in fact there is not space anywhere). In the case of booting from LiveCD, regardless of the syntax error, it will still load to RAM. For frugal install, I think it will not load into RAM until you fix the syntax error.

cheers!
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

l0wt3ch

#38 Post by l0wt3ch »

Ya, that's a typo. See my above posts where I typed it correctly if you just think I'm stupid or something.

Thanks. :lol:

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#39 Post by jamesbond »

l0wt3ch wrote:Ya, that's a typo. See my above posts where I typed it correctly if you just think I'm stupid or something.

Thanks. :lol:
Nah everybody makes mistakes :) Hmm then I can't explain because that test1.iso you posted, works fine with 350MB ram with or without the pfix=ram,copy parameter (booting from LiveCD) - in both cases it loads into RAM. Haven't tried frugal - because your original problem is booting from LiveCD, right?
Then really you need to use emil's initrd and see what you see what output you get when you boot that test1.iso.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

ICPUG
Posts: 1308
Joined: Mon 25 Jul 2005, 00:09
Location: UK

#40 Post by ICPUG »

I have a 512MB RAM laptop (although some is used for integrated graphics).

I use Lin'N'Win frugal so create my own menu.lst. I have NEVER used pfix=copy.

I am pretty sure my standard Lupu 5.2 is loading to RAM.

As I understand it the default is to copy the base sfs to RAM if it meets the rules in the init script.

The pfix=copy is used when you want to copy ADDITIONAL sfs files into RAM as well. That's my interpretation of what this says:
http://bkhome.org/blog/?viewDetailed=01813

I will try booting from CD and I will certainly know if it running in RAM

Can I point out that emil's post to Barry's blog deals specifically with Pupmode 6,7 where a whole partition is used for a save file. So the comments about the default there are not relevent to the more common use of Puppy with a pupsave file.

My link above is much more recent and is I think the time when Barry was messing with this.

User avatar
bigpup
Posts: 13886
Joined: Sun 11 Oct 2009, 18:15
Location: S.C. USA

#41 Post by bigpup »

Do you know about this:
http://bkhome.org/blog/?viewDetailed=02034
This is not fixed in the latest Lucid 5.2.

emil
Posts: 633
Joined: Tue 10 Nov 2009, 08:36
Location: Austria
Contact:

#42 Post by emil »

Barry Kauler created an enigma :D.
I suggest to go back to the code. There are the facts. Extract initrd.gz and view the init script.

jamesbond already posted the part of the code where it is decided if the pup.sfs is loaded to Ram or not. Setting the COPY2RAM flag is the crucuial line:

Code: Select all

[ "$COPYCONTENDER" = "yes" ] && [ $RAMSIZE -gt 220000 ] && [ $RAMSIZE -gt $MINRAM2CPY ] && COPY2RAM="yes"
Ramsize and Minram2cpy were already discussed. Last remaining condition is the flag COPYCONTENDER: This is empty by default but get's modified just in the few lines before the abovementioned conditional:

Code: Select all

if [ "$PNOCOPY" != "yes" ];then
  [ "`echo -n "$ATADRIVES" | grep "$pdPATTERN"`" = "" ] && COPYCONTENDER='yes' #test if not fast internal drive.
  [ $PUPMODE -eq 5 ] && COPYCONTENDER='yes'
  [ "$DEV1FS" = "iso9660" ] && COPYCONTENDER='yes'
 fi
 [ "$PCOPY" = "yes" ] && COPYCONTENDER='yes'

I translate this as follows:
default is: don't copy to RAM

Under the assumption that enough RAM is present and
a) pfix=nocopy is NOT explicitly given:
  • 1) if there is a slow drive copy to RAM ( :shock: how does that work?! )
    2) if Pupmode 5 (first boot) copy to RAM
    3) if the device is of type iso 9660 (CD-ROM) copy to RAM


b) pfix=copy is explicitly given
  • copy to RAM


I think Puppy is so ingenious with its different modes, layers, flexibilities to install that it is hard not to be confused. At least I was not aware of this exact behaviour. I always vaguely thought about Puppy is running "mostly" in RAM "somehow".
But please, if trying to find a possible bug, lets stick to the code and possibly get some hard facts for debugging. Everything else is speculation.

Lots of the variables in the init script are written to the file /etc/rc.d/PUPSTATE, so it is easier to follow the code.

I modified the "debug" version of initrd.gz a bit. It writes now the file debug.log to /root.
8) emil

l0wt3ch

#43 Post by l0wt3ch »

@ emil: Well, I remastered the disc with your initrd, and, while I couldn't find the debug file it was supposed to put in root, it seems to have fixed the problem. The sfs now loads into RAM with "puppy pfix=ram,copy".

So I remastered the Studio 3.3 non-realtime disc with your initrd, but no change. According to the posters on here, that's normal. 373 x 2 = more than 512 MB. But why does Studio 3.1, which was 394 MB and based on 511, load fine on the same machine?

I give up.

User avatar
john biles
Posts: 1458
Joined: Sun 17 Sep 2006, 14:05
Location: Australia
Contact:

#44 Post by john biles »

Hello l0wt3ch,
Maybe because your test compressed files are all the same ie 5 x same or 15 x same the compression app takes this in to account and knows it just has to unpack the same compressed file 5 or 15 times?

Have you tried making bigger compressed files and seeing if the final iso grows the same amount?

I've seen 1mb files compressed to 50kb's and thought something must have gone wrong. I uncompress them again and there fine?

Just a thought.
Legacy OS 2017 has been released.

l0wt3ch

#45 Post by l0wt3ch »

Hey John.

I don't think it matters that much. I had expanded the size of the sfs by 47 MB, and it wouldn't load into RAM with "puppy pfix=ram,copy". I thought that settled it. But now I used Emil's initrd (which had a debugging log feature), and somehow it fixed it. Not my original problem of why my 373 MB sfs won't load into RAM on boot with 520 when it would with 511, though.
Last edited by l0wt3ch on Fri 18 Feb 2011, 08:15, edited 1 time in total.

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#46 Post by jamesbond »

l0wtech,

Barry has a lot on his plate, see http://bkhome.org/blog/?viewDetailed=02146 - and he specifically mentioned he won't be able to respond to PMs as fast as he'd like.

Back to your problem, though. I get Pstudio 3.1 (which is based on Lupu 511?) - and guess what? It has a different init! (older version I guess).

It will load into RAM, because before the check for RAMSIZE and MINRAM2CPY, way way earlier, it has this code

Code: Select all

#v405 decide whether to copy sfs's to ram...
COPY2RAM=""
COPYMSG='copying to ram' #purple
#v4.00 lowered rom 230000 to 220000... v403 added PUPSFSDEVMNTPT test... v404 explicit PCOPY needed...
[ $PUPMODE -eq 5 ] && PCOPY="yes" #well, override on first boot.
[ $PUPMODE -eq 77 ] && PCOPY="yes" #v406 multisession dvd.

#100406 this came about because aufs did not work if an sfs mounted via loop device as one layer
#was actually resident on another layer. however latest patched aufs may allow this...
if [ $KERNELSUBVER -lt 33 ];then
 #v404 absolutely must copy to ram, otherwise layerfs conflict...
 [ $PUPMODE -eq 6 -o $PUPMODE -eq 7 ] && COPY2RAM="yes"
 [ "$COPY2RAM" = "yes" ] && COPYMSG='forced copying to ram' #purple
fi

#w482 lower this again, so multisession will work in 256MB system...
#[ $RAMSIZE -gt 260000 -a "$PCOPY" = "yes" ] && COPY2RAM="yes" #256MB system. note, only checking physical ram. w003 incr. from 220000.
[ $RAMSIZE -gt 220000 -a "$PCOPY" = "yes" ] && COPY2RAM="yes" #note, only checking physical ram.
Basically, this one says that in PUPMODE=5 (ie first boot livecd), as long as your RAM is bigger than 220MB, it will copy to RAM. Because this check is done earlier and it sets the COPY2RAM variable directly, it will override any later checks - thus, no matter what, it will always try to load to RAM. This code has been known to cause problems - if you make a 700MB SFS, it will still try to load to RAM on a 512MB system - which will fail of course - you'll end up with kernel panic.

All your questions answered, now back to Puppy Studio, shall we? :D

cheers!
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

l0wt3ch

#47 Post by l0wt3ch »

Do you really think that if I swap out some piece of code out of the initrd's with that one it'll fix it? :D
Last edited by l0wt3ch on Fri 18 Feb 2011, 08:16, edited 1 time in total.

jamesbond
Posts: 3433
Joined: Mon 26 Feb 2007, 05:02
Location: The Blue Marble

#48 Post by jamesbond »

That's all right - I just pointed it out in case you miss it (sometimes Barry can be prolific in writing his blogs that we all miss his announcement one way or another).

Anyway, what objective would you try to achieve - are you trying to get that 3.1 does (ie, 397MB and still load into RAM), or do the right thing in Lupu 5.2?
If you would like to do what 3.1 does, I wouldn't recommend that you swap the code - who knows what other dependencies or side effects in between. Instead, let's use what is already in Lupu 5.2 code and work it out from there. Look at this code this part:

Code: Select all

MINRAM2CPY=$(($SIZESFSK * 2)) #100222 technosaurus: in case of very big puppies. 
Instead of making it 2 x SIZESFSK, just make it equal to SIZESFSK

Code: Select all

MINRAM2CPY=$SIZESFSK
That should do the job.

Good luck.
Fatdog64 forum links: [url=http://murga-linux.com/puppy/viewtopic.php?t=117546]Latest version[/url] | [url=https://cutt.ly/ke8sn5H]Contributed packages[/url] | [url=https://cutt.ly/se8scrb]ISO builder[/url]

emil
Posts: 633
Joined: Tue 10 Nov 2009, 08:36
Location: Austria
Contact:

#49 Post by emil »

Hey it is great :!: that the problem seems fixed,
and I see no objection to change the script specifically for puppy studio, so that it loads to RAM the way jamesbond mentioned. But I don't know what happens if you execute a programm which needs some extra RAM ..
If you want to give a bit more slack say:

Code: Select all

MINRAM2CPY=$(($SIZESFSK +50000))  # this is SFS size + 50 MB
Try it yourself, it is easy to make this change (Howto in first link). If you have problems I can do this to make a customized initrd.gz. I can also remove the debug echos if you like.
Personally I learned lots from your problem, so thank you for going public :D!
emil

l0wt3ch

#50 Post by l0wt3ch »

@jamesbond: Thanks for the suggestion, that works! 3.3 now boots straight into RAM with 512 MB of memory. (I didn't test if "puppy pfix=ram,copy" works, but it boots into RAM now, and that's all I care about.) PuppyStudio 3.3's performance advantages are back!

@emil: Thanks for your efforts in making that debug initrd. Appreciated!

And a big thank you to everyone else for taking the time to stop by and contribute! :D

Post Reply