Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 16 Nov 2018, 06:05
All times are UTC - 4
 Forum index » House Training » Users ( For the regulars )
How to bind /tmp folder to Save folder instead of pure RAM?
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [7 Posts]  
Author Message
s243a

Joined: 02 Sep 2014
Posts: 1275

PostPosted: Sun 21 Oct 2018, 02:29    Post subject:  How to bind /tmp folder to Save folder instead of pure RAM?  

I was wondering if anyone had any guidence on how I could bind the tmp folder from the save folder rather than having it pure ram.

My motivation is as follows. I once was linking and binding my save folder into a puppy running in ram rather than using puppies traditional layered file system approach (see this thread). The reason I was doing this was that I was having trouble finding my save file at boot due to a large hard drive (see thread and github issue). Although I had a solution to the finding the save file issue, I didn't do it because my system was slow to boot, which could of caused headaches if I had to do too much troubleshooting.

Anyway, the bind approach appears to have had some behavior that I think might have performance advantages on low ram systems with slow I/O. What I noticed is it only synced the folders as they were used, so nothing was synced into ram unless it was being used.

Now regarding "/tmp", I've had this folder fill up my ram. Some examples where this happened include the getLIbreOffice scirpt on TazPup, the pburn script, and the mono slackbuild script.

In many cases I was able to solve the problem by changing the temporary folder but this can be a large source of frustration, if one is doing a long install or build process and the script fails because of insufficient ram at the end of the process. One might not even know that they have insufficient ram and be left scratching their head. Also one may not know in advance which programs are going to write large amounts of data to the tmp folder.

One solution to the above problems might be swap but the performance of swap is horrible especially if one is running something as slow as USB 2.0 or slower. Also if the system crashes while puppy is saving to storage then any data in swap will be lost. This might be considered a type of corruption to the save file and/or folder. Perhaps puppy should only save the delta to storage and then combine the data at startup and/or shutdown. One way to do this might be to create an sfs file for the current state of puppy and use this sfs file at startup, and start with a new save file/folder every time.

Some people here have done this with the adrv. When this is done we can be sure a new adrv is successfully created before we use it rather then the old version. I think the limitation here though is one needs enough ram to create the sfs file. Therefor perhaps once the adrv gets large enough once could create a second adrv as a delta. Actually I believe one of the puppy drives is built for exactly this purpose (i.e. to apply patches) but maybe when the delta gets too large we need a second delta.

Alternatively one could save stuff directly to the storage device in a folder. This would be faster for starups and shutdowns. On a hard drive. Disk I/O is a limiting factor both in new and old computers. If one uses usbflash mode then the folder resides in ram but we only want to keep a system that is so large in ram. One could could have two folder. A save folder and an underdog. One would need to check that nothing in the top layer blocks the changes in the underdog from coming through and if so then they would need to create an sfs that captured these differences and sat above the base sfs. One could merge this new sfs file with the base sfs.

Now as a final note on this it would be nice if there was a safe way to use this bind and link idea beyond the tmp directory but I haven't thought of it yet. Perhaps, one should do the same thing for browser cache and any other large files in their system. The problem is though if they do this it will limit portability and remastering opportunities for their system.

On a related note, with regards to merging saved progress into the top layer stemsee has a different approach which he mentions in the Fatdog64-720 and 721 thread.
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 5276
Location: Republic of Novo Zelande

PostPosted: Sun 21 Oct 2018, 04:01    Post subject:  

Just getting this on my watch list - this seems complex and I doubt i have anything of assistance to add - but I am always keen to understand ways to get Puppy working in a personalised way.

Just a question - you want the /tmp folder loading to usb instead of in RAM? Did I get that right?
Back to top
View user's profile Send private message 
musher0

Joined: 04 Jan 2009
Posts: 12972
Location: Gatineau (Qc), Canada

PostPosted: Sun 21 Oct 2018, 04:03    Post subject:  

Hi s243a.

I have to read your post above with a rested head. Heavy stuff, since you
are dealing here with one of Puppy's basic structures / caracteristics.

For now I will just comment on this sentence. You say:
Quote:
I think the limitation here though is one needs enough ram to create
the sfs file.
There is no limitation for creation of any sfs file if you create it on a large
physical disk.

It's when you load it -- once it has been created -- that RAM is needed to
store / install it. Even so, if a user does not have enough RAM, the sfs
loader will detect it and leave on disk what does not fit in RAM.

Another quick thought off the top of my head is: move /tmp elsewhere than
in RAM and expect a loss in speed for the Puppy. /tmp in RAM is one of the
reasons why Puppy is fast regardless of the PC it runs on.

A third one: I've tried FatDog a couple of times. A fine distro it is indeed.

However, it is my understanding that FatDog is now derived from a Linux-
from-Scratch build. It can accommodate Puppy, but it is not a Puppy in
structure anymore, AFAIK. So expect glitches and adaptations if you import
into a classic Puppy code or scripts written expressly for FatDog. May I
suggest that you fully deploy your mental sensors if you attempt this! Smile

Finally, please state which breed and version of PuppyLinux you wish to do
this on? A brief description of your PC's capacities will certainly help too,
since RAM size may be a consideration. TIA.

TWYL.

_________________
musher0
~~~~~~~~~~
Je suis né pour aimer et non pas pour haïr. (Sophocle) /
I was born to love and not to hate. (Sophocles)
Back to top
View user's profile Send private message 
s243a

Joined: 02 Sep 2014
Posts: 1275

PostPosted: Sun 21 Oct 2018, 04:39    Post subject:  

greengeek wrote:
Just getting this on my watch list - this seems complex and I doubt i have anything of assistance to add - but I am always keen to understand ways to get Puppy working in a personalised way.

Just a question - you want the /tmp folder loading to usb instead of in RAM? Did I get that right?


That's more or less right. With bind it resides elsewhere than in ram but it is synced into ram as needed. This makes it faster than directly accessing the disc but not as fast as if it resides in ram as it currently does.

I think that things that are accessed a lot can reside in the var folder and if one is using puppy in usbflash mode or ram mode then the var folder will be in ram. usbflash mode is essentially like ram mode except that you have the option to save either periodically or at shutdown.
Back to top
View user's profile Send private message 
s243a

Joined: 02 Sep 2014
Posts: 1275

PostPosted: Sun 21 Oct 2018, 04:50    Post subject:  

musher0 wrote:
Hi s243a.

I have to read your post above with a rested head. Heavy stuff, since you
are dealing here with one of Puppy's basic structures / caracteristics.

For now I will just comment on this sentence. You say:
Quote:
I think the limitation here though is one needs enough ram to create
the sfs file.
There is no limitation for creation of any sfs file if you create it on a large
physical disk.

It's when you load it -- once it has been created -- that RAM is needed to
store / install it. Even so, if a user does not have enough RAM, the sfs
loader will detect it and leave on disk what does not fit in RAM.

That's nice to know Smile. So therefore maybe stemsee approach will work while with computers that vastly differ in capabilities. Note that this is a side issue than my question about the /tmp folder. The discussion about using the adrv (or other sfs file) arose as a way to mitigate the possibility of corrupting ones system due to a crash during a save. I've noticed that when write speeds are slow saving the session can significantly impact performance. On one computer I actually turned off periodic saves because the save was taking too long and my system ran very sluggishly while it saved.

Quote:
Another quick thought off the top of my head is: move /tmp elsewhere than
in RAM and expect a loss in speed for the Puppy. /tmp in RAM is one of the
reasons why Puppy is fast regardless of the PC it runs on.

I know I'll take a speed hit but not as much as if I had the /tmp folder directly accessing the disk. Also the speed hit only occurs up until the point where I need to start swaping. Once one needs to start swapping then I think my suggested approach is faster because it keeps core programs in memory and instead syncs stuff that isn't need from the /tmp folder to disk.



Quote:
A third one: I've tried FatDog a couple of times. A fine distro it is indeed.

However, it is my understanding that FatDog is now derived from a Linux-
from-Scratch build. It can accommodate Puppy, but it is not a Puppy in
structure anymore, AFAIK. So expect glitches and adaptations if you import
into a classic Puppy code or scripts written expressly for FatDog. May I
suggest that you fully deploy your mental sensors if you attempt this! Smile

Last time I checked the /tmp folder on fatdog ran in ram but I didn't check this with the latest version.

Edit: but your right, stemsee script was for fatdog64 so if I decided to use his approach I'd have to be a little carfule. LazyPup/TSH/
ITSMERSH (hopefully I got his name right) has done something simmilary in his LAZYPUP/T.O.P.L.E.S.S/N.E.M.E.S.I.S. system so if I wanted to adapt stemsee's script perhaps I could search Lazypup's posts.

Quote:
Finally, please state which breed and version of PuppyLinux you wish to do
this on? A brief description of your PC's capacities will certainly help too,
since RAM size may be a consideration. TIA.

TWYL.


I'll certainly do this so that we can see what computers might benefit from my proposal.

Edit: the computer has 1GB of ram, I'm running tazpup. With Tazpanel open, two terminal windows and two folders I'm using 14% of that.
Back to top
View user's profile Send private message 
musher0

Joined: 04 Jan 2009
Posts: 12972
Location: Gatineau (Qc), Canada

PostPosted: Sun 21 Oct 2018, 13:39    Post subject:  

Hi, s243a.

I queried ask.com with a general question about /tmp:
https://www.ask.com/web?l=dir&qsrc=0&o=ffx&q=tmp+directory+in+PuppyLinux

This old thread initiated by Moose-on-the-Loose came up:
http://www.murga-linux.com/puppy/viewtopic.php?t=67891
Its eight pages are worth a browse-through, I think.
It contains a couple of good ideas.

A couple of other questions popped up in my mind about your question, as
to why do you wish to do this?
---- because your machine has too little RAM?
---- because you wish to emulate, in PuppyLinux, the WhineDose capacity to
extend RAM on a Flash drive?

A note --
/tmp has a tmpfs structure.
---- If more available RAM is your goal, could a zram approach be
appropriate for /tmp?

I'm just asking. I have never researched this possibility, but maybe another
Puppyist did.

Zram is a compression process for RAM. For the rest, please do a search on
"zram" on the Internet; there's really too much material on that subject to
summarize it here.

Finally, a dumb question, for which I beg your forgiveness in advance:
you're cash-strapped and cannot consider buying more RAM for your PC at
the moment? (That's ok, I'm cash-strapped too!) Wink Sometimes, though,
good old hardware solutions are just the thing!

IHTH.

_________________
musher0
~~~~~~~~~~
Je suis né pour aimer et non pas pour haïr. (Sophocle) /
I was born to love and not to hate. (Sophocles)
Back to top
View user's profile Send private message 
musher0

Joined: 04 Jan 2009
Posts: 12972
Location: Gatineau (Qc), Canada

PostPosted: Sun 21 Oct 2018, 14:36    Post subject: Re: How to bind /tmp folder to Save folder instead of pure RAM?  

Hi, s243a.

To revisit these concerns of yours:
s243a wrote:
(...)
Now regarding "/tmp", I've had this folder fill up my ram. (...)

In many cases I was able to solve the problem by changing the temporary
folder but this can be a large source of frustration, if one is doing a long
install or build process and the script fails because of insufficient ram at the
end of the process. One might not even know that they have insufficient ram
and be left scratching their head.(...)

A couple of ways around this would be, I think:
-- run the < free > utility before you start compiling. Then you know how
much RAM you have available before you start compiling.

-- most ./configure scripts offered to compile apps have a way to define a
cache file, something like this when you run
Code:
./configure --help | more
Quote:
--cache-file=FILE cache test results in FILE [disabled]
-C, --config-cache alias for `--cache-file=config.cache'

If you enable this cache file when compiling and define it to be on disk, you
should be able to avoid the RAM "strangulation" issue you are describing.

~~~~~~~~
Another useful thing to know about RAM in PuppyLinux is:
---- if you indicate < pfix=nocopy > on your grub4dos launch line,
PuppyLinux will try to keep as much as possible of its main sfs on disk.

In this case, "nocopy" means: "Do not copy to RAM". I noticed that Puppy
uses approx. 200 Mg's less RAM when I specify this parameter at boot, and
it pretty much runs as fast.

IHTH.

_________________
musher0
~~~~~~~~~~
Je suis né pour aimer et non pas pour haïr. (Sophocle) /
I was born to love and not to hate. (Sophocles)
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 1 [7 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » House Training » Users ( For the regulars )
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1693s ][ Queries: 12 (0.0499s) ][ GZIP on ]