Toolbox for ITTopicsUNIXAIXGroups Ask a Question IBM AIX
The IBM AIX group is your premier resource for objective technical discussion and peer-to-peer support on IBM AIX. Home | Invite Peers | More UNIX Groups

Memory Utilization is High in Cluster

Asked by praveen kumar korvi | posted 4 days ago | Replies (14)
 Hi Friends,

Wehave 5 node active-passive cluster, one of the node from clusterutilizing more memory. RAM for this node is 16GB and paging space is12GB. It is using total 16GB ram and upto 50% of paging space. Cananybody give a permanent solution to fix this issue? Can i createanother paging space in external disk? or shall i increase the size ofexisting paging space? IBM AIX 5.3 TL3 is running on this servers.14 Replies
0 Pleasesign inorregister to mark
this reply as helpful. Ahmed
Replied 4 days ago
thesystem performance gets effected by the paging activity. may be youmight want to increase the RAM on the system to stop paging. beforedoing anything collect vmstat for a specified period of time and lookfor the highest avm value. the system should have RAM of about 1.2 *highest avm value according to IBM to stop paging.Top
0 Pleasesign inorregister to mark
this reply as helpful. rakshath shetty
Replied 4 days ago
Clearlyyour paging is too low . Create another paging device of 4-8 GB on adifferent PV . As for memory utilization , is it consistently high ?This is a little tricky on AIX unlike other OS's ..how much of pagingdo you see used ?Top
0 Pleasesign inorregister to mark
this reply as helpful. praveen kumar korvi
Replied 4 days ago
Hi rakshat

2months before it was only 20% used but slowly its getting increased.Now it is taking upto 50%. At night times it is reducing to 38% and inthe business hours it is coming to 45 to 50%. Due to clusterenvironment, with this reason heartbeat is missing. previously we facedthe same issue with another node. it used to failover to another nodedue to high memory utilization. so now I want to stop this server forfailover. For example if i create paging on external disk, will iteffect on cluster environment?Top
0 Pleasesign inorregister to mark
this reply as helpful. Ahmed
Replied 4 days ago
It would be best practice if you can extend the existing paging spaceTopRelated White Papers and Webcasts

• Oracle Real Application Clusters 11g Release 2
• Oracle Real Application Clusters One Node: Better Virtualization for Databases
• Using Oracle In-Memory Database Cache to Accelerate the Oracle Database

More White Papers
0 Pleasesign inorregister to mark
this reply as helpful. PRANAYA MOHANTY
Replied 3 days ago
I think you need to study the behavior of you application 1st. Who is consuming Real Memory ( RAM).
Just find how much Computional Memory %Comp and %file caching memory is used . Do you need any tuning in VMM .
a. Ideally, % Minperm =5 and %maxperm=90%.
b. lru_file_repage = 0
Bytuning above parameter , you are giving more room for process to usemore real memory and paging space utilization will be balance.
If all the tuning is correct, then look for Memory leak if any
There is a chance of memory leak on application process. You must know how to monitor the memory leakage.Top
0 Pleasesign inorregister to mark
this reply as helpful. praveen kumar korvi
Replied 3 days ago
Hi Pranay

minperm% = 20 maxperm%=40 it is set this values. and lru_file_repage = 0.
sorry for saying this, i don't know how to monitor memory leakage.Top
1 Pleasesign inorregister to mark
this reply as helpful. PRANAYA MOHANTY
Replied 3 days ago
Hi,
I feel you need to tune your VMM parameters i.e %minperm and %maxperm
It should be like this:
%minperm=5
%maxperm=90
To find the memory leakage is not straight forward. Follow the below procedure:
1. Collect vmstat out put for 2-3 hrs during peak hr while production will be running on full /max load.
- Look the avm column and if the value is keep increase, then there is possible of memory leakage.
Now need to know which process/processes got memory leak. TO know this use the following method

2. capture the svmon output for top 10 or 20 process consumed more memory for some hrs .
- Command : svmon -Pt 20 -> for top 20 processes.
- Look the " work process private segment"
- If the value keep increase, then that particular process have memory lekage.

Generally C and Java programs get some memory leak.



Thanks and Regards,
Pranaya KumarTop
0 Pleasesign inorregister to mark
this reply as helpful. praveen kumar korvi
Replied 2 days ago
Hi Pranaya
15days back I kept svmon command in crontab. I found that one pid inusevalue is keep on increasing. 15days back the value was 17434407 and nowit is 439274352. i checked with application team and they confirmedthat it should run daily. this process is compulsory and they cannotkill or not modify that.. but i thing cant understand, whenever theapplications are restarted at that time the paging space will go upto10 to 20% and memory will also go to 90%. once the application startedand once it becomes full busy and working, pagiing will come up to 25to 30%. as per application team again it should not take the memory.but here day by day slowly paging space is increasing. i dont knowwhere the exact problem is.TopRelated White Papers and Webcasts

• Automatic Service Migration in WebLogic Server
• Achieving Business Agility with Application Grid
• Increase Application Performance With Solid State Disk

More White Papers
0 Pleasesign inorregister to mark
this reply as helpful. PRANAYA MOHANTY
Replied 2 days ago
Based on the SVMON data there the huge jump of the RAM consumed by the process.
Can you find whether the memory utilization of that process varied upand down over period of time OR there is continuous surge .
If it slowly increasing , then there is a chance of memory leakage.

Secondly, what it is the RAM size of your system and swap space size?
Asyou said while you start up the application you paging space increasedto 25% , it means your physical RAM is not enough to handle the process.
If Physical RAM is sufficient, then wait for someday and observe.

Canyou send me vmstat -v command output ( 2 times in 5 mins interval )output and vmstat 2 2 ( for 10- 15 mins ) output. I will let you knowafter I study your data

Thanks and Regards,
Pranaya Kumar

DID: (65)-63946712
Mobile: (65)-90072109Top
0 Pleasesign inorregister to mark
this reply as helpful. praveen kumar korvi
Replied 2 days ago
Hi Pranaya,

Thatparticular process iinuse value is increasing and decreasing but if weobserve on sep14 it was17434407 and now it 439274352. in between ofthese days it increased and decreased(nore more than 17434407). RAM is16gb and paging space is 12gb. present paging space utilization is 50%,if i restart the application it s going down to 25% and real memory isgoing to 90%.
vmstat -v
4079616 memory pages
3870177 lruable pages
19824 free pages
1 memory pools
990414 pinned pages
80.0 maxpin percentage
20.0 minperm percentage
40.0 maxperm percentage
20.5 numperm percentage
796341 file pages
0.0 compressed percentage
0 compressed pages
20.5 numclient percentage
40.0 maxclient percentage
796341 client pages
0 remote pageouts scheduled
13946 pending disk I/Os blocked with no pbuf
9964146 paging space I/Os blocked with no psbuf
2484 filesystem I/Os blocked with no fsbuf
1113 client filesystem I/Os blocked with no fsbuf
103925 external pager filesystem I/Os blocked with no fsbuf
0 Virtualized Partition Memory Page Faults
0.00 Time resolving virtualized partition memory page faults


System configuration: lcpu=4 mem=15936MB

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 0 3894275 25231 0 0 0 0 0 0 60 11089 2759 42 5 53 0
1 1 3892532 26723 0 2 0 0 0 0 67 11965 1940 51 5 43 1Top
0 Pleasesign inorregister to mark
this reply as helpful. PRANAYA MOHANTY
Replied 2 days ago
Hello,
You have to much on tuning part. Basically, we are not allowing the application more room to use the memory.

1st settle VMM tuning . This will bring you down the paging space utilization .
20.0 minperm percentage - > Change it to 5%
40.0 maxperm percentage -> change it to 90%
You should know how to change the parameter . It is by using vmo -p command

I hope after tuning this will solve your problem. We need to wait for some day atleast 1 or 2 days and monitor the server.

Yoursystem is not paging in much . This means no much pg/po. System is notI/O bound now . May be later you may face if heavy page-in and out willoccur.


Thanks and Regards,
Pranaya Kumar

DID: (65)-63946712
Mobile: (65)-90072109Top
0 Pleasesign inorregister to mark
this reply as helpful. Wayne Willcox
Replied 2 days ago
What language is the application written in? Some languages use more
memory then others for the same jobs.

>
>
> Hi Pranaya
> 15 days back I kept svmon command in crontab. I found that one pid inuse
> value is keep on increasing. 15days back the value was 17434407 and now it
> is 439274352. i checked with application team and they confirmed that it
> should run daily. this process is compulsory and they cannot kill or not
> modify that.. but i thing cant understand, whenever the applications are
> restarted at that time the paging space will go upto 10 to 20% and memory
> will also go to 90%. once the application started and once it becomes full
> busy and working, pagiing will come up to 25 to 30%. as per application
> team again it should not take the memory. but here day by day slowly
> paging space is increasing. i dont know where the exact problem is.
>