Loading SFS to RAM Without Requiring 2X RAM (Solved)

[l0wt3ch]

When booting a derivative remastered from 520, "puppy pfix=ram,copy" wasn't working like it was with 511. So I increased the size of the sfs file by 47 MB to 170 by adding a dummy file to see if it had to do with system files being altered, but got the same results.

[emil]

Hi,

I didn't know if this is exactly the same issue, but I once had similar problems with loading or not loading into RAM (Sagelive was a 700 MB pupplet) , despite giving explicit boot parameters..

There is a passage in the initram.gz which decides if enough RAM is present to load everything.

A very well written and enlightning forum post by pizzasgood on a similar topic
http://www.murga-linux.com/puppy/viewtopic.php?t=24899

After I reported some problems there was a change in the memory management section 1 year ago
http://bkhome.org/blog/?viewDetailed=01408

I didn't follow this closly so I dont know if this is really the latest change.
Maybe someone else has more up to date information. Else the above links could give you an idea where to look at.

For example you could change initrd,gz in a way, so that it tries to load to RAM whenever possible and give a warning if it is not possible.
best, emil

[Lobster]

Basically Puppy is compressed and uncompresses in ram
So Puppy Lucid may be 130MB but 256 MB is needed for running from ram - otherwise with 128 MB of ram, programs will be loaded from CD on occasion

With a larger remaster (such as yours) more ram is needed and 1 Gig would seem the minimum - I would not expect 512 to be enough . . .

I hope this goes some way to answering your question
and is not too obvious . . .

[01micko]

hello lowt3ch

The results of your experiment definitely do seem a bit peculiar, sorry I have no explanation at this stage.

In your test system with 512 RAM did it have swap? I don't use swap on my main box as I have 2 gig RAM. If your machine does have swap I'm just wondering if it's being turned on and used in pfix=ram. I believe it is supposed to be. If it doesn't have swap then loading a 395MB main sfs is almost impossible inside of 512 RAM, unless there is a lot of data in said main sfs that doesn't compress well, example: images.

I suppose some more basic tests could be done, such as running "free" as soon as the iso has booted, that will tell you a few things, such as if swap is on etc.

I just did my own experiment with a woofed up puppy I made recently, loosely based on lucid and from the current 20101229 woof. See screeny. My particular 134MB main sfs was reported to be using 307MB RAM straight after booting pfix=ram. Results vary vastly from puppy to puppy I have found.

hth, Cheers

(yes Lobby, this'un is slightly more techy , though I hope someone out there can shed further light))

[l0wt3ch]

@emil: Thanks for the links, I'll read up on it.

@Lobster: Yes, I am well aware that Puppy decompresses the sfs into RAM. Thing is, this is a smaller sfs than 3.1's, which never had any problem booting with 512 MB of RAM.

@01Micko: No, there's no swap partition. Too bad it wasn't something as simple as that.

[aarf]

Two things that you may have already taken into consideration. Different file systems ext2 ext3 etc may report different sizes for the same file. Which may also be attributed to Different sizes being reported by different applications or even different versions of the same application. Eg. Lxtask in Puppeee will sometimes report that just one applications is using 150% of cpu for extended periods.

[jamesbond]

Here is an excerpt from init from recent woof (not latest, but recent enough), which determines whether the SFS will be copied to RAM.

Code: Select all

 #if there's heaps of ram, copy puppy.sfs to a tmpfs...
 #v405 fast media plus more than 256MB ram then definitely worth copying to ram...
 SIZESFSK=`du -k ${PUPSFSDEVMNTPT}${PUPSFSFILE} | cut -f 1`
 SIZESFSK=$(($SIZESFSK + 1000)) #some slack.
 MINRAM2CPY=$(($SIZESFSK * 2)) #100222 technosaurus: in case of very big puppies.
 
 #100920 decide whether to copy .sfs's to ram (see prior decision above)...
 pdPATTERN="`echo "$PUPSFSDEV" | sed -e 's/[0-9]*$//'` " 
 if [ "$PNOCOPY" != "yes" ];then
  [ "`echo -n "$ATADRIVES" | grep "$pdPATTERN"`" = "" ] && COPYCONTENDER='yes' #test if not fast internal drive.
  [ $PUPMODE -eq 5 ] && COPYCONTENDER='yes'
  [ "$DEV1FS" = "iso9660" ] && COPYCONTENDER='yes'
 fi
 [ "$PCOPY" = "yes" ] && COPYCONTENDER='yes'
 [ "$COPYCONTENDER" = "yes" ] && [ $RAMSIZE -gt 220000 ] && [ $RAMSIZE -gt $MINRAM2CPY ] && COPY2RAM="yes"

The code that follows directly from this uses COPY2RAM variable to decide whether to create tmpfs and copy over the SFS, or directly do losetup to the SFS file.

And RAMSIZE is defined as

Code: Select all

RAMSIZE=`free | grep 'Mem:' | tr -s ' ' | cut -f 3 -d ' '` #total physical ram (less shared video).

So - based on the above code, you need a minimum of:
a) 220MB of RAM
b) 2 x (SFS size + 1) MB
before it can be copied into RAM. If your SFS is only 170MB and you have 512MB of RAM, it should definitely be copied into RAM (I doubt that the kernel will gobble up over 160MB of RAM), unless the RAMSIZE variable is somehow calculated wrongly.

I'm not sure whether this portion of code has changed in Lupu 5.2.

[rcrsn51]

So - based on the above code, you need a minimum of:
a) 220MB of RAM
b) 2 x (SFS size + 1) MB
before it can be copied into RAM. If your SFS is only 170MB and you have 512MB of RAM, it should definitely be copied into RAM (I doubt that the kernel will gobble up over 160MB of RAM), unless the RAMSIZE variable is somehow calculated wrongly.I'm not sure whether this portion of code has changed in Lupu 5.2.

Here is my experience. Starting with Quirky 1.3 and now in Lupu 5.2, the default behaviour is to NOT copy to RAM, even on machines with 512 MB. This is easy to check by watching for the "Loading main file ... copying to ram" message, then checking with "free".

However, I can control the behaviour with the "pfix=copy" option.

[Lobster]

Different file systems ext2 ext3 etc may report different sizes for the same file

Thanks for giving more technical responses guys
Puppy did go over to ext 3 and I have noticed 'anomalies' (what Microsoft call features) with file sizing. These are maybe bugs in the filing system but Puppy is very conservative with its filing. I have only recently been using ext3. I wonder if using Fat32 or ext2 make a difference?

Also we changed back (again) from one squash file system to another - could there be a code conflict?

I do remember there was something on Barrys blog about something along these lines . . . maybe someone will remember . . .

[playdayz]

I am glad some of the heavyweights are weighing in here. We did test extensively that the default 5.2 iso would load into ram on a machine with 256MB. To achieve that is when we started shrinking things down to 127MB. So Lucid definitely knows how to load into ram. And as you have said, the new studio loads into ram reliably when the machine has 1024MB (1GB) ram. That suggests to me that there might be something in the code that determines whether or not to load into ram based on size.

I regret not having a machine with 512MB ram.

[rcrsn51]

Maybe it's just me, but I did a standard frugal install of Lupu 520 onto another machine with 512MB and got the same default "nocopy" behaviour.

[playdayz]

As 01micko suggests, if the rule is 2 x (SFS size + 1) MB then the 393 MB sfs should not load into ram on 512MB, but 170MB should. And a standard frugal install definitely should--in fact a standard frugal install loads into ram on my 256MB notebook. So maybe the problem is the 512MB somehow in the code.

[emil]

I think I dug out the blogpost which explains the "intended" behaviour:
http://bkhome.org/blog/?viewDetailed=00225
( Hm, I wrote something, but it was rubbish because you cannot load 170 MB sfsfiles to RAM. If the code in your initrd.gz looks like the code from jamesbond, then something is seriously strange)
I am just curious: You need pfix=copy for the low latency "realtime kernel" behaviour? Have you tested latency if the sfs is not in RAM?
emil

[rjbrewer]

I am glad some of the heavyweights are weighing in here. We did test extensively that the default 5.2 iso would load into ram on a machine with 256MB. To achieve that is when we started shrinking things down to 127MB.

That was after comments about Luci not being able to eject disk
with 256mb ram.

Wonder if "Studio" was based on one of those "problem" versions?

[emil]

I looked at an older initrd.gz, here the line looks like
[ "`echo -n "$FASTPARTS0" | grep "$PUPSFSDEV"`" != "" ] && [ $RAMSIZE -gt 280000 ] && [ $RAMSIZE -gt $MINRAM2CPY ] && COPY2RAM="yes"

in the 5.20 initrd gz (line 1218) this line looks like
[ "$COPYCONTENDER" = "yes" ] && [ $RAMSIZE -gt 220000 ] && [ $RAMSIZE -gt $MINRAM2CPY ] && COPY2RAM="yes"

the whole section looks like it was heavily cleaned up and partly rewritten.
Could it be that the variable COYCONTENDER gives problems? This must be set to yes, and it is done in 3 lines just before nr. 1218:

Is it a possibility to edit the initrd.gz and put some debugging lines just before line 1218 like (or similar)

echo "COPYCONTENDER: " $COPYCONTENDER > /dev/console
echo "RAMSIZE: " $RAMSIZE > /dev/console
echo "MINRAM2CPY: "$MINRAM2CPY >/dev/console

emil

[rcrsn51]

Maybe free memory is calculated differently in the Northern Hemisphere. But I just did a frugal install of Lupu 520 on a machine with 2 GB of RAM and got the same "nocopy" behaviour.

Is no one else seeing this?

[emil]

I just tested 5.2 with my mentioned debug lines in the 5.2 initrd.gz
I have 1 GB RAM

without bootparameter:
COPYCONTENDER=""
COPY2RAM=""
sfs is not copied to RAM
checked with command free, after startup there are < 200 MB used (don't have the exact numbers)

with bootparamter pfix=copy
COPYCONTENDER ="yes"
COPY2RAM="yes"
sfs is copied to RAM
checked with command free, after startup there are ca 480 MB RAM used.
emil

initrd.gz with debug echos

[playdayz]

Maybe free memory is calculated differently in the Northern Hemisphere. Wink But I just did a frugal install of Lupu 520 on a machine with 2 GB of RAM and got the same "nocopy" behaviour.

Is no one else seeing this?

rcrsn5, I know you are not kidding, I will do some systematic testing.

First finding is that pfix=copy is default in the menu.lst entry and the frugal install does copy to ram.

Second finding is that if I remove pfix=copy then the frugal install does *not* copy to ram (even with 4GB).

rcrsn5, is that the case with the frugal install you tested? I think this is the expected behavior for a frugal install. I experimented quickly and didn't see any speed difference whether it copied to ram or did not. That would be different with the Live-CD and 512MB ram.

Code: Select all

 title Lucid Puppy lupu-520 (on /dev/sda2)
  rootnoverify (hd0,1)
  kernel /lupu-520/vmlinuz pmedia=atahd pfix=copy psubdir=lupu-520
  initrd /lupu-520/initrd.gz

[jamesbond]

emil, rcrsn51, apparently, this is the expected behaviour. As this part of the code says:

Code: Select all

 if [ "$PNOCOPY" != "yes" ];then
  [ "`echo -n "$ATADRIVES" | grep "$pdPATTERN"`" = "" ] && COPYCONTENDER='yes' #test if not fast internal drive.
  [ $PUPMODE -eq 5 ] && COPYCONTENDER='yes'
  [ "$DEV1FS" = "iso9660" ] && COPYCONTENDER='yes'
 fi 
 [ "$PCOPY" = "yes" ] && COPYCONTENDER='yes'

if you don't specify pfix=nocopy which actively prohibits from loading to RAM, by default loading to RAM will only happen if:
a) you run off slow media (ie not internal harddisk - e.g. CD)
b) you run with pfix=ram

Thus, when you do frugal install, the behaviour is *not* to load into RAM unless you request it explicitly by adding pfix=copy.

What puzzles me for l0wtech case is that he is running from CD (thus candidate for loading to RAM), he has also explicitly asked for loading to RAM (he uses pfix=copy), he's running in PUPMODE=5 (pfix=ram), his SFS size is 170M and he's got 512MB RAM ... he ticks off everything that would force loading to RAM so he should get the behaviour. But apparently it doesn't.

[playdayz]

l0wt3ch

I forget if you told me, Does standard Lucid 5.2 Live-CD copy to ram on boot in that machine with 512MB ram? It should.

More info for people, I do remember that you said Studio does copy to ram on a machine with 1GB. Is that still the case?