Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 17 Nov 2017, 14:04
All times are UTC - 4
 Forum index » House Training » Users ( For the regulars )
Massive memory leak in 3.01 OS!!!
Moderators: Flash, Ian, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 2 [18 Posts]   Goto page: 1, 2 Next
Author Message
ScottD

Joined: 04 Jul 2007
Posts: 39

PostPosted: Fri 16 Nov 2007, 13:27    Post subject:  Massive memory leak in 3.01 OS!!!  

I'm running 3.01 as a DAQ and web server. The data is added to a database every 5 seconds. I ran 48 hour test and noticed the OS memory allocated was growing all the time but the memory used by individual programs was not. Ran mpatrol to verify my application wasn't leaking.

I wrote a simple program that repeatedly opens a file, outputs about 20 bytes of text and closes the file. In about 20 minutes the file grows by about a megabyte, the OS reports 4 additional megabytes used. Closing the application doesn't release the memory. mpatrol reports no leaks in the test app.

This is installed on a flash IDE drive, frugal install.

I had heard there was a problem with the Union FS, where it was not releasing memory even whent eh file system was synced. But I heard this was fixed in 3.0

Is this a new problem? Is this the same problem but still not fixed?

I included the simple test app below.


#include <stdlib.h>
#include <iostream>

using namespace std;

// to test memoroy leak in OS
// open a file, insert text, close the file
// do this as many times as arg1 to program
//
//
int main(int argc, char** argv)
{
// get number of iterations
if(argc != 2) {
cout << "Error, must pass in number off iterations to run\n";
cout << "\n example: test 20000\n";
return -1;
}
int iterations = atoi(argv[1]);
cout << "\nStarting test with " << iterations << " iterations\n\n";

// first truncate file
FILE *fp = fopen("foo", "w");
fclose(fp);
while(iterations--)
{
fp = fopen("foo", "a+");
fputs("Where is this dang memory leak????", fp);
fclose(fp);
usleep(10000);
cout << "+";
cout.flush();
}

cout << "\nTest complete\n";

return (EXIT_SUCCESS);
}
Back to top
View user's profile Send private message 
Bruce B

Joined: 18 May 2005
Posts: 11488
Location: The Peoples Republic of California

PostPosted: Fri 16 Nov 2007, 18:11    Post subject: Re: Massive memory leak in 3.01 OS!!!  

Questions in color[red]

ScottD wrote:
I'm running 3.01 as a DAQ and web server. The data is added to a database every 5 seconds. I ran 48 hour test and noticed the OS memory allocated was growing [what testing program?]all the time but the memory used by individual programs was not [what testing program?]. Ran mpatrol to verify my application wasn't leaking.

I wrote a simple program that repeatedly opens a file, outputs about 20 bytes of text and closes the file. In about 20 minutes the file grows by about a megabyte, the OS reports 4 additional megabytes used [file grows by 1MB but OS reports the file grows by 4MB?, what testing tools and proceedure used?]. Closing the application doesn't release the memory. [again how are you testing? what are your expectations? that closing a 10MB program will give back 10MB?] mpatrol reports no leaks in the test app.

{rest cut}

Back to top
View user's profile Send private message 
friedsonjm

Joined: 10 Mar 2007
Posts: 208

PostPosted: Fri 16 Nov 2007, 18:31    Post subject:  

Wouldn't running Conky be an aide to seeing if this was really happening? jf
Back to top
View user's profile Send private message 
alienjeff


Joined: 08 Jul 2006
Posts: 2291
Location: Winsted, CT - USA

PostPosted: Fri 16 Nov 2007, 22:11    Post subject:  

A screenshot or three, time sequenced, of what "top" reports might be revealing.
_________________
hangout: ##b0rked on irc.freenode.net
diversion: http://alienjeff.net - visit The Fringe
quote: "The foundation of authority is based upon the consent of the people." - Thomas Hooker

Back to top
View user's profile Send private message 
Dougal


Joined: 19 Oct 2005
Posts: 2504
Location: Hell more grotesque than any medieval woodcut

PostPosted: Sat 17 Nov 2007, 09:19    Post subject: Re: Massive memory leak in 3.01 OS!!!  

ScottD wrote:
I had heard there was a problem with the Union FS, where it was not releasing memory even whent eh file system was synced. But I heard this was fixed in 3.0

I don't know exactly what was fixed or not, but note that you can use either unionfs of aufs -- I think it might be aufs that had the "true flushing".

_________________
What's the ugliest part of your body?
Some say your nose
Some say your toes
But I think it's your mind
Back to top
View user's profile Send private message 
ScottD

Joined: 04 Jul 2007
Posts: 39

PostPosted: Sat 17 Nov 2007, 18:54    Post subject:  

Well I installed Captura so I could get some screen shots but when I started Captura, the memused went up by 3 or 4 meg, and after every screen shot the use would rise by a mb, making it a pretty useless way to track system memory.

So I tried something really simple so you could verify the problem.

1.) Boot puppy, don't start any extra apps.
2.) start up two term windows(rxvt) and run top in one of them. Let it run for a few minutes to verify you memusage is stable. It may move up and down every so often but the low and high point will be the same. Total memused is in the upper left corner.
3.) Write down the memused reported by top.
4.) then copy a large file 5 times(cp foo foo1, cp foo foo2, etc..), something about 1mb in size. Give it 15 seconds to stablize. Note how much memused rises. Then delete the copied files. Note how much memused comes down.

On my machine I loose a couple hundred kb .. every time I do the copy delete routine.

help!!!!

Scott
Back to top
View user's profile Send private message 
ScottD

Joined: 04 Jul 2007
Posts: 39

PostPosted: Sat 17 Nov 2007, 22:00    Post subject:  

I was also suprised that that the memused goes up when I copy a file? Why would ram be used unless the filesystem is in ram?

I'm installed onto a IDE drive, which I thought would allow puppy to not use ram for the file system ... Though I am using a frugal install.

Scott
Back to top
View user's profile Send private message 
Everitt

Joined: 19 Dec 2006
Posts: 331
Location: Leeds,UK or Birmingham, UK

PostPosted: Sat 17 Nov 2007, 22:12    Post subject:  

ScottD wrote:
I'm installed onto a IDE drive, which I thought would allow puppy to not use ram for the file system ... Though I am using a frugal install.

Frugal installs use the RAM just like the live CD's
Back to top
View user's profile Send private message 
Flash
Official Dog Handler


Joined: 04 May 2005
Posts: 12699
Location: Arizona USA

PostPosted: Sat 17 Nov 2007, 22:37    Post subject:  

I think you may be getting excited about something that doesn't mean anything. One of the most important jobs of an OS is memory management, and I can't believe Linux is that bad.Try this: take a program that seems to be hoarding memory even though it should be releasing it, and run it until, by your measurements as given above, it should have used up all the memory and caused Puppy to screw up somehow. I'll bet it never happens.
Back to top
View user's profile Send private message 
PaulBx1

Joined: 16 Jun 2006
Posts: 2307
Location: Wyoming, USA

PostPosted: Sun 18 Nov 2007, 00:28    Post subject:  

Quote:
I was also suprised that that the memused goes up when I copy a file? Why would ram be used unless the filesystem is in ram?


Files have to be copied from disk to a memory buffer, then from the buffer back out to disk. The hardware cannot manage a copy from disk file directly to another disk file. I don't think it can be done even if the source and destination files are on different physical disks.

It used to be in the old days, that there were two buffers. The source file is writing one buffer while the other buffer is being dumped into the destination file. Then the roles switch. I have written such programs myself.
Back to top
View user's profile Send private message 
Bruce B

Joined: 18 May 2005
Posts: 11488
Location: The Peoples Republic of California

PostPosted: Sun 18 Nov 2007, 08:44    Post subject:  

Here's my top usage line:

Mem: 294040K used, 2301872K free, 0K shrd, 19836K buff, 140368K cached

How much memory am I using? In a very real sense I'm using 294040K, but I think it's being used this way: Mem used minus the buffers and cache.

-----------------------

The free utility concurs as shown by the output below:

Code:
[/] free
             total       used       free     shared    buffers     cached
Mem:       2595912     294100    2301812          0      19836     140516
-/+ buffers/cache:     133748    2462164
Swap:       554200          0     554200
[/]


Busybox free does not consider the buffers/cache the way free does, as shown by its output

Code:
[/] busybox free
              total         used         free       shared      buffers
  Mem:      2595912       294100      2301812            0        19836
 Swap:       554200            0       554200
Total:      3150112       294100      2856012
[/]


So when the buffers/cache is factored in (out), I'm using 133748K

Below is a pic of what htop thinks and it says I'm using 131MB RAM, but it's hard to read due to contrast.

Puppy's top and free are Busybox utilities and as such are not as full featured as the actual utilities Busybox intends to substitute.

This is easy enough to note simply by the output of free and busybox free - htop and busybox top.

In conclusion: I think Linux caches and doesn't release its cache all that easily, and when using cut down utilities, this will be reported as increase in used memory, which could be interpreted as a leak, when it is merely normal cache/buffer behavior.
htop-131-mb.png
 Description   
 Filesize   29.23 KB
 Viewed   974 Time(s)

htop-131-mb.png

Back to top
View user's profile Send private message 
ScottD

Joined: 04 Jul 2007
Posts: 39

PostPosted: Fri 23 Nov 2007, 21:58    Post subject:  

According to the Puppy documentation installation on IDE drives do not use RAM for the file system. Frugal or not.

That's why when I shut down I always get the message that the drive is top mounted and therefore doesn't need to be saved.

Scott

Everitt wrote:
ScottD wrote:
I'm installed onto a IDE drive, which I thought would allow puppy to not use ram for the file system ... Though I am using a frugal install.

Frugal installs use the RAM just like the live CD's
Back to top
View user's profile Send private message 
mcewanw

Joined: 16 Aug 2007
Posts: 3194
Location: New Zealand

PostPosted: Sat 24 Nov 2007, 05:13    Post subject:  

Everitt wrote:

Frugal installs use the RAM just like the live CD's


I'm with Everitt on that one...
Back to top
View user's profile Send private message Visit poster's website 
Bruce B

Joined: 18 May 2005
Posts: 11488
Location: The Peoples Republic of California

PostPosted: Mon 26 Nov 2007, 12:21    Post subject:  

ScottD wrote:
According to the Puppy documentation installation on IDE drives do not use RAM for the file system. Frugal or not.

That's why when I shut down I always get the message that the drive is top mounted and therefore doesn't need to be saved.

Scott

Everitt wrote:
ScottD wrote:
I'm installed onto a IDE drive, which I thought would allow puppy to not use ram for the file system ... Though I am using a frugal install.

Frugal installs use the RAM just like the live CD's


ScottD's error was not posting the link to the documentation he read.

Nevertheless, it's patently false to think that a Frugal install loads all of the filesystem into RAM.

It is also false to think that a Frugal install will arbitrarily copy even as much as pup_301.sfs into RAM.

Included with post are two screen shots, one where pup_301.sfs is copied to ram and the other where it is not.

--------------

One screen shot per post?

---------------
301-not-copied-to-ram.png
 Description   
 Filesize   7.98 KB
 Viewed   774 Time(s)

301-not-copied-to-ram.png


Last edited by Bruce B on Mon 26 Nov 2007, 12:28; edited 1 time in total
Back to top
View user's profile Send private message 
Bruce B

Joined: 18 May 2005
Posts: 11488
Location: The Peoples Republic of California

PostPosted: Mon 26 Nov 2007, 12:22    Post subject:  

I guess it wouldn't take two pics, here's the other one. The words to look for are copying to ram ..., although there are other more technical ways to determine if it copied to RAM or is running mounted on the HD.
301-copied-to-ram.png
 Description   
 Filesize   5.38 KB
 Viewed   760 Time(s)

301-copied-to-ram.png

Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 2 [18 Posts]   Goto page: 1, 2 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » House Training » Users ( For the regulars )
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1536s ][ Queries: 14 (0.0158s) ][ GZIP on ]