View topic - Low memory issue, many fds are open

Low memory issue, many fds are open

Discussion about the QNX6 OS.

Low memory issue, many fds are open

Postby ikshudm » Fri Jan 04, 2019 4:00 am

Hi,

I am facing low memory issue where only 5kb is free out of 250MB RAM. The memory used by the processes look same as it was used when system is stable. Only difference I could find is that that around 5000 Fds are open by procnto (80 fds are opens when system is stable). Any hint to find the root cause of this issue shall be greatly appreciated. Thanks in advance for you help.

Thanks,
Ikshu
ikshudm
New Member
 
Posts: 6
Joined: Thu Jan 03, 2019 6:12 am

Re: Low memory issue, many fds are open

Postby maschoen » Fri Jan 04, 2019 8:47 am

ikshudm wrote:Hi,

I am facing low memory issue where only 5kb is free out of 250MB RAM. The memory used by the processes look same as it was used when system is stable. Only difference I could find is that that around 5000 Fds are open by procnto (80 fds are opens when system is stable). Any hint to find the root cause of this issue shall be greatly appreciated. Thanks in advance for you help.

Thanks,
Ikshu


It sounds like an application program bug. A program continually doing an open but not closing it could cause this. What is running in your system.
maschoen
QNX Master
 
Posts: 2712
Joined: Wed Jun 25, 2003 5:18 pm

Re: Low memory issue, many fds are open

Postby ikshudm » Mon Jan 07, 2019 4:23 am

Hi Maschoen,

Thanks so much for your response. It is a confidential code, so I cannot share the code here. It is a very huge code and I have analysed if the fds are being correctly closed in the application. I am suspecting if it is a qnx issue (I mean is the qnx not closing the fd correctly). Is it possible that 5000 fds can consume around 40MB memory?

Regards,
Ikshu
ikshudm
New Member
 
Posts: 6
Joined: Thu Jan 03, 2019 6:12 am

Re: Low memory issue, many fds are open

Postby maschoen » Mon Jan 07, 2019 6:43 am

I don't think you've mentioned which version of QNX you are running. It is unlikely to be a QNX bug. The problem is not that 5000 fds can consume 40MB of memory. The problem is that the number of fds is a limited resource. The error you would see if you exceed it is probably the same as if you run out of memory. If you are running some version of QNX 6, then the command

$ pidin fd

will show you which process has fd's open. If one process has a lot, it will be obvious.

If you are using QNX 4 the command is

$ sin fd

Looking back at your original post, I would guess that if you only have 5K left of 40MB or 250MB it is probably not the fds that are the problem. The technical term for you what you have is probably a memory leak.
maschoen
QNX Master
 
Posts: 2712
Joined: Wed Jun 25, 2003 5:18 pm

Re: Low memory issue, many fds are open

Postby ikshudm » Tue Jan 08, 2019 5:52 am

Hi Maschoen,

Thanks for your time for this issue.
I am using Qnx 6.5.0. I used pdin fd to see which process is having more fds open. It was procnto process. I initially suspected it to be memory leak and compared the memory of the processes in stable and issue state. The memory consumed by all the processes are same when the issue occurs. I compared pidin mem output during stable and issue state. Any malloc will fail after available RAM memory reduces. Please let me know if there are any other items related to memory that I can compare in stable and issue state to find what is causing the memory leak.


Thanks,
Ikshu
ikshudm
New Member
 
Posts: 6
Joined: Thu Jan 03, 2019 6:12 am

Re: Low memory issue, many fds are open

Postby nico04 » Tue Jan 08, 2019 7:26 am

Hi,

Have you tried to use the "QNX System Information" perspective in Momentics ?
With this perspective, you can monitor CPU, RAM, fds...

Regards,
Nicolas
nico04
Senior Member
 
Posts: 141
Joined: Wed Sep 29, 2010 9:59 am
Location: France

Re: Low memory issue, many fds are open

Postby maschoen » Tue Jan 08, 2019 7:48 am

Your system is acting very strange.

In general procnto has only one reason I can think of for creating an fd, to open an executable and load it.

I suspect you are doing something unusual in your application somewhere. Whether the problem is a QNX bug or user error is impossible to know given the information you've provided.
maschoen
QNX Master
 
Posts: 2712
Joined: Wed Jun 25, 2003 5:18 pm

Re: Low memory issue, many fds are open

Postby nico04 » Tue Jan 08, 2019 7:57 am

In general procnto has only one reason I can think of for creating an fd, to open an executable and load it.

In procnto : Is the fd closed as soon as the executable is loaded/running ? Or is the fd closed when the executable exits ?
nico04
Senior Member
 
Posts: 141
Joined: Wed Sep 29, 2010 9:59 am
Location: France

Re: Low memory issue, many fds are open

Postby maschoen » Tue Jan 08, 2019 8:48 am

That's a good question, one that I don't know the answer too. Having the source might answer that. Some kernel and/or proc source was once released before QNX was bought by Blackberry. Then everything was reversed.

I do know that sometimes when I want to update a module, if it is currently running I get an error. That suggests that there is an open fd. However it doesn't always work this way, and I don't know the details of when.

I checked before my last reply on a slightly older version I have running, QNX 6.32, and procnto only listed one open fd, though there were many programs running. All very confusing.
maschoen
QNX Master
 
Posts: 2712
Joined: Wed Jun 25, 2003 5:18 pm

Re: Low memory issue, many fds are open

Postby nico04 » Tue Jan 08, 2019 8:59 am

I'm currently using QNX6.6.
In "QNX System Information" perspective, "System Resources" tab, procnto-smp is shown to be using 95 file descriptors.
The system is running 81 processes (procnto-smp included).
nico04
Senior Member
 
Posts: 141
Joined: Wed Sep 29, 2010 9:59 am
Location: France

Re: Low memory issue, many fds are open

Postby maschoen » Tue Jan 08, 2019 1:08 pm

Upon closer look,I'm getting the same sort of thing. I was running this:

# pidin fds | grep proc

Which only showed 1 fd. This of course was not the way to view this.

So the obvious question is, since pidin fds shows what the fd has open, what is open on the system with 5000 fds?

Clearly the theory that a running process means proc has the executable open is correct.

procnto document shows the following parameter

-F number
The maximum number of file descriptors that can be open at the same time. The minimum allowable value is 100. The default value is 1000, but might be constrained by the RLIMIT_NOFILE system resource.

So if a single process is opening more than 1000 fd's it should fail unless the default is increased. Maybe this is the problem?
maschoen
QNX Master
 
Posts: 2712
Joined: Wed Jun 25, 2003 5:18 pm

Re: Low memory issue, many fds are open

Postby ikshudm » Thu Jan 10, 2019 7:38 am

nico04 wrote:Hi,

Have you tried to use the "QNX System Information" perspective in Momentics ?
With this perspective, you can monitor CPU, RAM, fds...

Regards,
Nicolas


Hi Nicolas,

I do not have qnx IDE support. So I cannot use this.

Thanks,
Ikshu
ikshudm
New Member
 
Posts: 6
Joined: Thu Jan 03, 2019 6:12 am

Re: Low memory issue, many fds are open

Postby ikshudm » Thu Jan 10, 2019 7:42 am

maschoen wrote:Your system is acting very strange.

In general procnto has only one reason I can think of for creating an fd, to open an executable and load it.

I suspect you are doing something unusual in your application somewhere. Whether the problem is a QNX bug or user error is impossible to know given the information you've provided.


Hi Maschoen,

procnto can create fd for process management, memory management and path management as per this link http://www.qnx.com/developers/docs/6.5. ... /proc.html . Please let me know if my understanding is not correct.

Thanks,
Ikshu
ikshudm
New Member
 
Posts: 6
Joined: Thu Jan 03, 2019 6:12 am

Re: Low memory issue, many fds are open

Postby ikshudm » Thu Jan 10, 2019 7:46 am

maschoen wrote:Upon closer look,I'm getting the same sort of thing. I was running this:

# pidin fds | grep proc

Which only showed 1 fd. This of course was not the way to view this.

So the obvious question is, since pidin fds shows what the fd has open, what is open on the system with 5000 fds?

Clearly the theory that a running process means proc has the executable open is correct.

procnto document shows the following parameter

-F number
The maximum number of file descriptors that can be open at the same time. The minimum allowable value is 100. The default value is 1000, but might be constrained by the RLIMIT_NOFILE system resource.

So if a single process is opening more than 1000 fd's it should fail unless the default is increased. Maybe this is the problem?


Hi,

This is a good point, I am looking at setting this limit and trying to debug to find the root cause.

Thanks,
Ikshu
ikshudm
New Member
 
Posts: 6
Joined: Thu Jan 03, 2019 6:12 am


Return to QNX6 - OS

Who is online

Users browsing this forum: No registered users and 1 guest

cron