Debug Paging Space issue IBM AIX

We would have faced issues that paging space is more and need to find which process is using more page space.

Let's start by understanding what is Paging Space.
All process will require memory and that will be taken from RAM .In unix 4kb blocks of memory will be allocated.So in a case if there is no memory in our RAM for the new process then process wont be completed.To nullify this OS has paging space in the disk which will be used by the inactive process in the RAM and the new active process will get space in RAM for processing.This is called Page Out .If the inactive process in hard disk once got active will  move again to RAM.This process is called Page In.So this will be a continuous process in the operating system.

.

lsps -a  command is used to find on which disk paging space is alloted.

Page Space  Physical Volume       Size   %Used  
hd6         hdisk0              768MB      47   

lsps -s will give the  percentage of paging space used.
If percentage is less then no issue.

svmon -G to find free memory.

               size      inuse       free        pin    virtual
memory      1048576     417374     631202      66533     151468
pg space     262144      31993

If free memory is very less then we can increase the RAM size if its less .It seems like application issue please follow the below steps:
Use topas command to find which process is using more paging.
Click here for complete topas explanation .The link has very good explanation.Once spotted the most used process you can use the below two commands .

ps -mo THREAD -p <PID>|sort -nk6.This command be used to know which application is causing more paging .
svmon -P <PID>
In detail svmon commands are explained here.Hope this post will help to spot the bottleneck with paging

Log rotation for last week files

find . -type f -name *.log -mtime +1 -mtime -8 -exec rm -f {} \;

type f =its for file
-mtime +1 = modified time greater than 1 day.
-mtime -8=modified time less than 8 days.
-exec = executes a following commands here its removal of *.log files

J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/xx solution

The reason this error is coming is lack of connection pool to create connections.

Solution:

Increase the maximum connection pool size .If you are increasing the connection pool size at cluster scope remember use the below logic to set.

maxpoolsize * number of cluster members should not be greater than connection setting in database.
If above solution does not resolve then enable database trace to analyse further the PoolManager.
 

JVM argument to solve frequent garbage collection due to System.gc() or Runtime.gc()

System.gc() or Runtime.gc() is called in java codes to do explicit garbage collection.
Since JVM has the capability to do garbage collection on its own we should avoid the
use of sytem.gc() in our codes

If you have the below kind of lines in your gc  logs

<sys id="1" timestamp="Jul 15 12:56:26 2005" intervalms="0.000">
  <time exclusiveaccessms="0.018" />

  <tenured freebytes="821120" totalbytes="4194304" percent="19" >
    <soa freebytes="611712" totalbytes="3984896" percent="15" />
    <loa freebytes="209408" totalbytes="209408" percent="100" />
  </tenured>
  <gc type="global" id="1" totalid="1" intervalms="0.000">
    <classloadersunloaded count="0" timetakenms="0.012" />
    <refs_cleared soft="0" weak="4" phantom="0" />
    <finalization objectsqueued="6" />
    <timesms mark="3.065" sweep="0.138" compact="0.000" total="3.287" />
    <tenured freebytes="3579072" totalbytes="4194304" percent="85" >
      <soa freebytes="3369664" totalbytes="3984896" percent="84" />
      <loa freebytes="209408" totalbytes="209408" percent="100" />
    </tenured>
  </gc>
  <tenured freebytes="3579072" totalbytes="4194304" percent="85" >
    <soa freebytes="3369664" totalbytes="3984896" percent="84" />
    <loa freebytes="209408" totalbytes="209408" percent="100" />
  </tenured>

  <time totalms="3.315" />
</sys>

The tag <sys> meant that garbage collection is done due to System.gc() calls.

The intervalms is the time used to know how frequent the garbage collection 

happened for this kind.

0.00 ms represent that this is the first time gc happened due to system.gc call.

Please add the below JVM argument if you face frequent gc due to system.gc call

IBM JVM:-Xdisableexplicitgc

Sun JVM: -XX:+DisableExplicitGC 

HPEL logging websphere

High Performance Extensible Logging (HPEL) is a new log and trace facility. It provides a convenient

mechanism for storing and accessing log, trace, System.err, and System.out information produced by the application server or applications. It is an alternative to the basic log and trace facility, which provides the JVM logs, diagnostic trace, and service log files commonly named SystemOut.log/SystemErr.log, trace.log, andactivity.log. HPEL provides a log data repository, a trace data repository, and a text log file. 

 

Enable HPEL and configuring using adminconsole:

1.    Go to Troubleshooting and Logs and Trace.

2.Switch to HPEL mode.

3.Configure HPEL logging like path for the logs and trace ,purging time ,maximum log size etc.

 

4.You can configure trace for the logic you wanted.In the above picture I had configured for connection leak .Once done please save the configuration and do a restart to make it effective.

5.Click on View Hpel logs and trace on the server you want to see  the logs.

6.In HPEL you can filter the log and view only the content you want like only warning,fatal, from startdate  to stopdate etc.This can be done in the adminconsole or using LogViewer command you can view .