Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Tue 11 Dec 2018, 00:46
All times are UTC - 4
 Forum index » House Training » Users ( For the regulars )
How to reduce the amount of file caching in memory?
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [7 Posts]  
Author Message
s243a

Joined: 02 Sep 2014
Posts: 1305

PostPosted: Wed 19 Sep 2018, 14:22    Post subject:  How to reduce the amount of file caching in memory?  

I have a program, (or at least the consequence of two programs interacting), where the file cache is growing too fast and eating up my ram. At first I thought it was an issue my swap not working correctly but I have read that the system will not swap if the reason that your ram limts (swappiness) is reached is due to file caching. I found some paramaters that I might be able to tune:

Code:

echo cfq > /sys/block/sda/queue/scheduler
echo 10000 > /sys/block/sda/queue/iosched/fifo_expire_async
echo 250 > /sys/block/sda/queue/iosched/fifo_expire_sync
echo 80 > /sys/block/sda/queue/iosched/slice_async
echo 1 > /sys/block/sda/queue/iosched/low_latency
echo 6 > /sys/block/sda/queue/iosched/quantum
echo 5 > /sys/block/sda/queue/iosched/slice_async_rq
echo 3 > /sys/block/sda/queue/iosched/slice_idle
echo 100 > /sys/block/sda/queue/iosched/slice_sync
hdparm -q -M 254 /dev/sda

https://unix.stackexchange.com/questions/30286/can-i-configure-my-linux-system-for-more-aggressive-file-system-caching

I have no idea if these settings are right for me. Does anyone have any recommended value for these settings that I should apply to my system. I would like to normally have between 10 to 40 percent free ram (maybe fluctuate in this range), and have the system either swap or drop cache to keep it in these bounds. I'm using TazPup on a 64bit machine in this example but this should apply to any system.

I'm able to free up memory as follows:
Code:

 echo 3 | tee /proc/sys/vm/drop_caches

https://unix.stackexchange.com/questions/17936/setting-proc-sys-vm-drop-caches-to-clear-cache
Back to top
View user's profile Send private message 
Burn_IT


Joined: 12 Aug 2006
Posts: 3321
Location: Tamworth UK

PostPosted: Wed 19 Sep 2018, 15:34    Post subject:  

Reduce the blocksize??
_________________
"Just think of it as leaving early to avoid the rush" - T Pratchett
Back to top
View user's profile Send private message 
musher0

Joined: 04 Jan 2009
Posts: 13152
Location: Gatineau (Qc), Canada

PostPosted: Wed 19 Sep 2018, 20:19    Post subject:  

Hi, s243a.

You said "file cache", right?

Instead of fiddling with the sysblocks (which, off the top of my head, I
don't remember being recommended, Burn_IT is right to have doubts):

do(es) your program(s) have a configuration panel where you can tell it
(them) to store the file cache on disk rather than in RAM? If so, problem
nicely solved, I'd say!

Browsers have such a config panel; also Open|LibreOffice (in Options,
under "RAM" [click just under the "General" tag]. For other progs, I
wouldn't know, it certainly is worth a look.


Other remarks:
-- the drop-caches one-liner you quoted last is well-known and kosher. In
collaboration with some devs on the French side (augras and Argolance
among others) , I came up with this bilingual RAM cleaner:
http://www.murga-linux.com/puppy/viewtopic.php?p=981053&sort=lastpost&search_id=411234405#981053
which uses it.

You can use it with PSchedule or cron calling it in the background with the
following parameter:
Code:
cleanup_memory histo
("histo" being short for "history")
This will save a time-dated report of every time PSchedule or cron
launches the script in file /root/my-documents/tmp/suivimev.his and show
up a little Before/After window on screen. (Please see the screen capture
in the post quoted above.)

If you use it with parm "chrono"
Code:
cleanup_memory chrono
it does it silently. Nothing is shown on screen, you'll have to consult the
"his" file to know how much RAM was occupied over a period of time.

You determine the interval at which the cleanup_memory script will run
in PSchedule or with a cron wrapper in /root/Startup.

I would think that using this "cleanup_memory" script every half hour
or so would help you solve your problem.


-- your quoted lines about disk-caching, I'm not sure: let's remember that
the original article answers someone using a disk formated in ntfs. I
believe ext4 does part ot this natively (but don't take my word for it, do
a bit of research on this?!).

IHTH.

_________________
musher0
~~~~~~~~~~
Je suis né pour aimer et non pas pour haïr. (Sophocle) /
I was born to love and not to hate. (Sophocles)
Back to top
View user's profile Send private message 
s243a

Joined: 02 Sep 2014
Posts: 1305

PostPosted: Fri 21 Sep 2018, 00:42    Post subject:  

musher0 wrote:
Hi, s243a.

You said "file cache", right?

Instead of fiddling with the sysblocks (which, off the top of my head, I
don't remember being recommended, Burn_IT is right to have doubts):

do(es) your program(s) have a configuration panel where you can tell it
(them) to store the file cache on disk rather than in RAM? If so, problem
nicely solved, I'd say!

Browsers have such a config panel; also Open|LibreOffice (in Options,
under "RAM" [click just under the "General" tag]. For other progs, I
wouldn't know, it certainly is worth a look.


Other remarks:
-- the drop-caches one-liner you quoted last is well-known and kosher. In
collaboration with some devs on the French side (augras and Argolance
among others) , I came up with this bilingual RAM cleaner:
http://www.murga-linux.com/puppy/viewtopic.php?p=981053&sort=lastpost&search_id=411234405#981053
which uses it.

You can use it with PSchedule or cron calling it in the background with the
following parameter:
Code:
cleanup_memory histo
("histo" being short for "history")
This will save a time-dated report of every time PSchedule or cron
launches the script in file /root/my-documents/tmp/suivimev.his and show
up a little Before/After window on screen. (Please see the screen capture
in the post quoted above.)

If you use it with parm "chrono"
Code:
cleanup_memory chrono
it does it silently. Nothing is shown on screen, you'll have to consult the
"his" file to know how much RAM was occupied over a period of time.

You determine the interval at which the cleanup_memory script will run
in PSchedule or with a cron wrapper in /root/Startup.

I would think that using this "cleanup_memory" script every half hour
or so would help you solve your problem.


-- your quoted lines about disk-caching, I'm not sure: let's remember that
the original article answers someone using a disk formated in ntfs. I
believe ext4 does part ot this natively (but don't take my word for it, do
a bit of research on this?!).

IHTH.



I'm searcing your cleanup_memory4c script for var or log and I don't see anything but maybe I don't want a log since they can grow over time but maybe at one line per hour it wouldn't grow that fast. Amway, I'm going to give it a try.

I created a file called
/var/spool/cron/crontabs/mem-clean
Code:

1 * * * * /usr/sbin/cleanup_memory chrono


then I started it with
Code:

/etc/init.d/crond start


Now to see what happens.
Back to top
View user's profile Send private message 
s243a

Joined: 02 Sep 2014
Posts: 1305

PostPosted: Fri 21 Sep 2018, 03:36    Post subject:  

Okay, it's working now. I found out that you have to name the crontab script after the user, and I'm running it as frequently as possible.

/var/spool/cron/crontabs/mem-clean
Code:
Code:

* * * * * /usr/sbin/cleanup_memory histo


I had to create the directories
Code:

/root/my-applications

and
Code:

/root/my-documents

and I symbolicly linked the /tmp folder in mydocuments to the system /tmp folder
Code:

ln -s /tmp /root/my-documents/tmp


I also found out that there is a comma sytax for cron so I could do something like:
Quote:

*/n in any field means every n min (or hr etc) but if this format is not supported in your system then you will have to list all possible values , separated. For example

0,10,20,30,40,50 * * * * script
if I wanted it to run every 10 minutes

https://www.experts-exchange.com/questions/22808403/Schedule-cronjob-every-minute.html

p.s. to change the logging level on crond, I just directly called cornd rather than using the startup script (i.e. /init.d/crond)
Back to top
View user's profile Send private message 
musher0

Joined: 04 Jan 2009
Posts: 13152
Location: Gatineau (Qc), Canada

PostPosted: Fri 21 Sep 2018, 05:34    Post subject:  

Hi.

I'm glad it's working for you.

Please note that the script log's filename --if used -- is -- as mentioned
above -- /root/my-documents/tmp/suivimev.his. It's a ".his" file, as in
"history". Not that the name of the file matters much; but you should not
have had to recreate one, one is already created for the user, by the script.
~~~

Something puzzles me in your post above: why did you have to recreate
the directories "/root/my-documents" and "/root/my-applications" on a
PuppyLinux? They already exist, no?! Smile Unless you did a silly manoeuvre
earlier and had to recreate them? (Which can happen to the best of us!!)
~~~

Also, linking the directories
/tmp

and

/root/my-documents/tmp

is NOT a good thing:

/root/my-documents/tmp is used for back-ups of draft texts in Geany or
similar text editor,

while

the /tmp directory is the active de facto ramdisk in PuppyLinux. It is used
for "events" of the system and similar.

Those are two very different functions. They should be left separate. This
linking may cause you trouble and confusion in the future, I think, if you
leave it.
~~~

BFN.

_________________
musher0
~~~~~~~~~~
Je suis né pour aimer et non pas pour haïr. (Sophocle) /
I was born to love and not to hate. (Sophocles)
Back to top
View user's profile Send private message 
TyroBGinner

Joined: 30 Mar 2016
Posts: 241

PostPosted: Wed 26 Sep 2018, 18:27    Post subject:  

This is an interesting subject. I looked into it when trying to reduce disk accesses so as to keep my drive from dying early....which brings up the subject of load cycling that is never mentioned on this forum but seems to be set to a ridiculously high value in every puppy I have tried.

Perhaps the best thing is to get more memory for your computer or to get a new computer entirely. Failing that, other steps are possible.

Some time ago, I found the stack exchange message that was cited above and used it to good effect. It is one of the better bits of information I found on the subject. Using the "drop_caches" setting does what it says but could be regarded as an extreme approach. In the stack exchange message there is the section on manipulating reading from the disk as you pointed out, but there is also a section on how to control writing back to the disk from cache. Changing these parameters could allow useful caching while keeping memory relatively free. Examine what the stack exchange message says regarding looking at the kernel documentation on the subject.

There are also wyas to set the kernel to report what processes are causing caches pages to be dirtied....and there is a program, called fatrace, that provides useful information on that issue.

_________________
Win10 and lovin' it!
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 1 [7 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » House Training » Users ( For the regulars )
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1208s ][ Queries: 11 (0.0149s) ][ GZIP on ]