Troubleshooting high CPU utilization on K2 server

kbt133826

Issue

Your K2 Host Server is constantly showing high CPU utilization between 90% to 100%, regardless of peak or off-peak hours, and users are complaining of slowness when performing K2 operations.

 

Symptoms

Any of the following can result in high CPU utilization:

• Process instances that have been 'Running' for an extended period of time.
   
  As long as a process instance is in the 'Running' state, your K2 Host Server will continue to use up server resources.
• In K2 database, aServer].eAsync] or nServer].eIPCAsync] tables contain records with date older than current date, and these records are not clearing up over time.
   
  The oAsync] and nIPCAsync] tables store records that the K2 Host Server will use to process work. As the server process the records, the records will eventually clear up. If the records are not cleared up and stayed in the tables, the server threads will keep trying to process them.

 

To determine if you have process instances that are 'Running' indefinitely,

1. Go to your K2 workspace > Management Console.
2. Check the process instances page for all your existing processes, and look out for process instances with a 'Running' status.
3. Unless you have an event that is supposed to take a long time to execute, a process instance should not be constantly 'Running'. Take note of your process instance Folio and Start Date. You will need the information from these two in the next step.
   
   
    
4. Open your Process Overview report, and use the Folio obtained in the previous step to quickly locate the process instance you are looking for.
5. Within the Process Overview report for that process instance, you can view information about the activities that have either been completed, or are still active. Select the activity that is currently active to view the list of events that have been completed or are still active/running.
   
   
    
6. To find out which event is still active or running, check the events' Finish Date. If an event is still active or running, the Finish Date will not be filled. If you have a Client Event that is still active, it means the process instance is currently waiting for your user's action. This is fine, as a process instance is not consuming resources if it is simply waiting for users' input.
   
   However, if it is a non-Client Event, check the Duration of the event to find out how long the event has been running. If that event has been running longer than what you have expected, it is likely that something has gone wrong, which resulted in your process instance being unable to move forward.
   
   
    

To check your nServer].cAsync] and vServer].nIPCAsync] tables,

1. Run the following SQL queries on your K2 database:
    
   SELECT *
   FROM LServer]. Async]
   ORDER BY cDate] ASC
    
   SELECT *
   FROM EServer].rIPCAsync]
   ORDER BY nDate] ASC
   
    
2. The above queries should sort the records in ascending order based on the date. Take note of the records with date that is older than your current date. Keep monitoring this table for a while.
   If after some time, these records are not cleared, then it indicates something is wrong. If these records are not cleared, K2 Host Server will keep allocating threads to process them, which in turn occupies server resources. 
   
   

Troubleshooting Steps

To stop a 'Running' process instance

1. Make sure that the process instance is really stuck at an event, and not waiting for a delayed start condition. For this, you will have to review your workflow design to determine which case this belongs to.
2. Use the K2 Workspace > Management Console to stop any process instances.
   
   
    
3. If you are unable to stop the 'Running' process instance from K2 workspace, stop your K2 Blackpearl service.
4. In your K2 database, run the following SQL query:
    
   SELECT *
   FROM CServer].SProcInst]
   Where rStatus] = 1
5. Search for the process instance you are trying to stop, and take note of the oID].
   
   
    
6. Once you verify which process instance you want to stop, run the following SQL query to stop it. Replace the <Process Instance ID> with the ID you have recorded in the previous step.
    
   Please make sure to create a backup of your K2 database before you update anything in the database.
    
   UPDATE Server].cProcInst]
   SET EStatus] = 4
   WHERE /ID] = <Process Instance ID>
7. Start your K2 Blackpearl service.
8. Refresh your K2 Workspace, and check if the process instance has been stopped.
   
   

 

 

To clear up old >Server].>Async] or Server]. IPCAsync] records

1. Run the following SQL queries on your K2 database:
    
   SELECT *
   FROM lServer].eAsync]
   ORDER BY ODate] ASC
    
   SELECT *
   FROM cServer].dIPCAsync]
   ORDER BY Date] ASC
2. Identify the records with dated older than the current date. Take note of the process instance ID in ProcInstID].
3. Run the following SQL query, replacing <Process Instance ID> with the process instance ID you have obtained in the previous step:
    
   SELECT *
   FROM vServer].CProcInst]
   WHERE ID] = <Process Instance ID>
4. From the results shown, determine the current status of the process instance.
    
   Here is a brief description of the oStatus] value to help you identify the current status of your process instance:
   
   1 = Running
   2 = Active
   4 = Stopped
5. If your process instance has already stopped, you can remove the process instance through your K2 Workspace > Management Console, then check if the records are cleared from the eServer].eAsync] and eServer]. IPCAsync] tables.
6. If your process instance is still in the 'Running' or 'Active' status, wait for it to complete or stop, then check your iServer].pAsync] and pServer].hIPCAsync] again to determine if the old records are cleared up.
    
   If the records did not clear up, please contact K2 Support, and reference this article to request for further assistance in removing the records from the sServer].iAsync] and tServer].dIPCAsync] tables.
7. If you are unable to find the process instance record in bServer].nProcInst], then it is likely the process instance has already completed. In that case, run the following SQL query:
    
   SELECT *
   FROM ÂServerLog].sProcInst]
   WHERE ID] = <Process Instance ID>
    
   If the process instance record shown has oStatus] = 4, then your process instance should have been completed or stopped. If you have Server].nAsync] or Server].eIPCAsync] records referencing this process instance, please contact K2 Support, and reference this article to request for further assistance in removing these records.