How to reduce the amount of file caching in memory?

[s243a]

I have a program, (or at least the consequence of two programs interacting), where the file cache is growing too fast and eating up my ram. At first I thought it was an issue my swap not working correctly but I have read that the system will not swap if the reason that your ram limts (swappiness) is reached is due to file caching. I found some paramaters that I might be able to tune:

Code: Select all

echo cfq > /sys/block/sda/queue/scheduler
echo 10000 > /sys/block/sda/queue/iosched/fifo_expire_async
echo 250 > /sys/block/sda/queue/iosched/fifo_expire_sync
echo 80 > /sys/block/sda/queue/iosched/slice_async
echo 1 > /sys/block/sda/queue/iosched/low_latency
echo 6 > /sys/block/sda/queue/iosched/quantum
echo 5 > /sys/block/sda/queue/iosched/slice_async_rq
echo 3 > /sys/block/sda/queue/iosched/slice_idle
echo 100 > /sys/block/sda/queue/iosched/slice_sync
hdparm -q -M 254 /dev/sda

https://unix.stackexchange.com/question ... em-caching

I have no idea if these settings are right for me. Does anyone have any recommended value for these settings that I should apply to my system. I would like to normally have between 10 to 40 percent free ram (maybe fluctuate in this range), and have the system either swap or drop cache to keep it in these bounds. I'm using TazPup on a 64bit machine in this example but this should apply to any system.

I'm able to free up memory as follows:

Code: Select all

 echo 3 | tee /proc/sys/vm/drop_caches

https://unix.stackexchange.com/question ... lear-cache

[Burn_IT]

Reduce the blocksize??

[musher0]

Hi, s243a.

You said "file cache", right?

Instead of fiddling with the sysblocks (which, off the top of my head, I
don't remember being recommended, Burn_IT is right to have doubts):

do(es) your program(s) have a configuration panel where you can tell it
(them) to store the file cache on disk rather than in RAM? If so, problem
nicely solved, I'd say!

Browsers have such a config panel; also Open|LibreOffice (in Options,
under "RAM" [click just under the "General" tag]. For other progs, I
wouldn't know, it certainly is worth a look.


Other remarks:
-- the drop-caches one-liner you quoted last is well-known and kosher. In
collaboration with some devs on the French side (augras and Argolance
among others) , I came up with this bilingual RAM cleaner:
http://www.murga-linux.com/puppy/viewto ... 405#981053
which uses it.

You can use it with PSchedule or cron calling it in the background with the
following parameter:

Code: Select all

cleanup_memory histo

("histo" being short for "history")
This will save a time-dated report of every time PSchedule or cron
launches the script in file /root/my-documents/tmp/suivimev.his and show
up a little Before/After window on screen. (Please see the screen capture
in the post quoted above.)

If you use it with parm "chrono"

Code: Select all

cleanup_memory chrono

it does it silently. Nothing is shown on screen, you'll have to consult the
"his" file to know how much RAM was occupied over a period of time.

You determine the interval at which the cleanup_memory script will run
in PSchedule or with a cron wrapper in /root/Startup.

I would think that using this "cleanup_memory" script every half hour
or so would help you solve your problem.


-- your quoted lines about disk-caching, I'm not sure: let's remember that
the original article answers someone using a disk formated in ntfs. I
believe ext4 does part ot this natively (but don't take my word for it, do
a bit of research on this?!).

IHTH.

[s243a]

Hi, s243a.

You said "file cache", right?

Instead of fiddling with the sysblocks (which, off the top of my head, I
don't remember being recommended, Burn_IT is right to have doubts):

do(es) your program(s) have a configuration panel where you can tell it
(them) to store the file cache on disk rather than in RAM? If so, problem
nicely solved, I'd say!

Browsers have such a config panel; also Open|LibreOffice (in Options,
under "RAM" [click just under the "General" tag]. For other progs, I
wouldn't know, it certainly is worth a look.


Other remarks:
-- the drop-caches one-liner you quoted last is well-known and kosher. In
collaboration with some devs on the French side (augras and Argolance
among others) , I came up with this bilingual RAM cleaner:
http://www.murga-linux.com/puppy/viewto ... 405#981053
which uses it.

You can use it with PSchedule or cron calling it in the background with the
following parameter:

Code: Select all

cleanup_memory histo

("histo" being short for "history")
This will save a time-dated report of every time PSchedule or cron
launches the script in file /root/my-documents/tmp/suivimev.his and show
up a little Before/After window on screen. (Please see the screen capture
in the post quoted above.)

If you use it with parm "chrono"

Code: Select all

cleanup_memory chrono

it does it silently. Nothing is shown on screen, you'll have to consult the
"his" file to know how much RAM was occupied over a period of time.

You determine the interval at which the cleanup_memory script will run
in PSchedule or with a cron wrapper in /root/Startup.

I would think that using this "cleanup_memory" script every half hour
or so would help you solve your problem.


-- your quoted lines about disk-caching, I'm not sure: let's remember that
the original article answers someone using a disk formated in ntfs. I
believe ext4 does part ot this natively (but don't take my word for it, do
a bit of research on this?!).

IHTH.

I'm searcing your cleanup_memory4c script for var or log and I don't see anything but maybe I don't want a log since they can grow over time but maybe at one line per hour it wouldn't grow that fast. Amway, I'm going to give it a try.

I created a file called
/var/spool/cron/crontabs/mem-clean

Code: Select all

1 * * * * /usr/sbin/cleanup_memory chrono

then I started it with

Code: Select all

/etc/init.d/crond start

Now to see what happens.

[s243a]

Okay, it's working now. I found out that you have to name the crontab script after the user, and I'm running it as frequently as possible.

/var/spool/cron/crontabs/mem-clean
Code:

Code: Select all

* * * * * /usr/sbin/cleanup_memory histo 

I had to create the directories

Code: Select all

/root/my-applications

and

Code: Select all

/root/my-documents

and I symbolicly linked the /tmp folder in mydocuments to the system /tmp folder

Code: Select all

ln -s /tmp /root/my-documents/tmp

I also found out that there is a comma sytax for cron so I could do something like:

*/n in any field means every n min (or hr etc) but if this format is not supported in your system then you will have to list all possible values , separated. For example

0,10,20,30,40,50 * * * * script
if I wanted it to run every 10 minutes

https://www.experts-exchange.com/questi ... inute.html

p.s. to change the logging level on crond, I just directly called cornd rather than using the startup script (i.e. /init.d/crond)

[musher0]

Hi.

I'm glad it's working for you.

Please note that the script log's filename --if used -- is -- as mentioned
above -- /root/my-documents/tmp/suivimev.his. It's a ".his" file, as in
"history". Not that the name of the file matters much; but you should not
have had to recreate one, one is already created for the user, by the script.
~~~

Something puzzles me in your post above: why did you have to recreate
the directories "/root/my-documents" and "/root/my-applications" on a
PuppyLinux? They already exist, no?! Unless you did a silly manoeuvre
earlier and had to recreate them? (Which can happen to the best of us!!)
~~~

Also, linking the directories
/tmp

and

/root/my-documents/tmp

is NOT a good thing:

/root/my-documents/tmp is used for back-ups of draft texts in Geany or
similar text editor,

while

the /tmp directory is the active de facto ramdisk in PuppyLinux. It is used
for "events" of the system and similar.

Those are two very different functions. They should be left separate. This
linking may cause you trouble and confusion in the future, I think, if you
leave it.
~~~

BFN.

[TyroBGinner]

This is an interesting subject. I looked into it when trying to reduce disk accesses so as to keep my drive from dying early....which brings up the subject of load cycling that is never mentioned on this forum but seems to be set to a ridiculously high value in every puppy I have tried.

Perhaps the best thing is to get more memory for your computer or to get a new computer entirely. Failing that, other steps are possible.

Some time ago, I found the stack exchange message that was cited above and used it to good effect. It is one of the better bits of information I found on the subject. Using the "drop_caches" setting does what it says but could be regarded as an extreme approach. In the stack exchange message there is the section on manipulating reading from the disk as you pointed out, but there is also a section on how to control writing back to the disk from cache. Changing these parameters could allow useful caching while keeping memory relatively free. Examine what the stack exchange message says regarding looking at the kernel documentation on the subject.

There are also wyas to set the kernel to report what processes are causing caches pages to be dirtied....and there is a program, called fatrace, that provides useful information on that issue.