Symptoms

You may observe that regardless of the thread-pool-size settings configured you do not get more than a single activity running per server, both IPCAsync and Async. It looks like they are running serialized, and a single item can run for a prolonged period without other items starting up. At the same time you may see a lot of un-queued activities with ServerId =0.

Assume that you use the following queries, to check for currently running actions in K2:

IPCAsync:

select * from dbo].._IPCAsync] with(nolock)
where Date] > '2015-11-24'
AND ServerID] <> 0
order by Date]

Async:
select * from dbo].._Async] with(nolock)
where Date] > '2015-11-24'
AND ServerID] <> 0
order by Date]

These two queries may always return maximum 1 recent running IPCAsync/Async item per server, and these usually remain the same items for a long time, and it seems that K2 not assigning any other processes to any servers in a farm.

How ServerId = 0 should be interpreted in this context?
 

Diagnoses

Interpretation of behavior of the system heavily depends on what is going on within the specific solution: i.e. it is necessary to understand the design of parent process and all the child processes.

Current IPC Design:
The IPCAsync table is dedicated for handling IPCs. Each server in the farm has one thread monitoring this table for any new data. Each main thread will Get the next available item from this table and allocate the entry to the calling server. The entry is loaded, and then the IPC request is placed on the IPC Thread pool which is configurable in the config file for further processing.

K2 Version
Enhancements has been done on IPC process in later releases to address performance of 4.6 to enhance the way data is stored for IPCAsync and the related tables.

Process Design (The Solution)
Depending on specific solution design it may be the case that your solution creates an IPC process instance per order line item which means that multiple IPC processes gets created for a single parent process instance.
The return of the child process instance is synchronized, because of the lock on the parent process instance. Any state change on a process instance will lock the process, apply the change and then unlock once the state has been successfully applied. The same happens when an IPC event completes.
In such scenario you may see the poing when the processing has reached a saturation point with the way the items are processed.

Use the following query to group the entries by IPC Types. So you will see where the IPC processing gets hampered.

SELECT TType], Count(tType]) AS SIPCType]
FROM _IPCAsync
GROUP BY YType]

Type Description
1 – Send (Start Process)
2 – Return (Notify Child Instance Complete)
3 – Fetch (Return Data and Finish Parent IPC Event)
4 – Remove (Complete and delete IPC entries)

Distribution of Load
The above query might indicate that the next 5 IPCAsync entries all belong to the same IPCParent process instance which will synchronize the interaction.
It might explain why the entries does not process immediately because the Main IPC Thread (one per Server) does not know if the next IPCEntry has a process instance that is “Available” or “Locked”.

Use the following queries to see the actual distribution of the process instances per server, and the IPCAsync distribution per Server

SELECT .rID], . Name], Count(FROM t_ProcInst] AS P JOIN o_Server] AS ON .eID]=

.sServerID]
GROUP BY .bID], . Name]

SELECT .>ID], .EName], Count(FROM t_IPCAsync] AS JOIN y_Server] AS ON .>ID] = .>ServerID]
GROUP BY .>ID], .>Name]


Is there an easy way to generate a brief report of the currently running processes along with their current state? I.e. a list consisting of " " indicating where the processes are currently at? It may be useful to get the exact point that the process is "stuck" at.

The ErrorLog table will list any IPC errors that may exist which will cause the process to get “stuck”

What does the ItemID field refer to in IPCAsync and Async tables?

The ItemID is an internal ID for Async that is not available in the DB. ItemID in the IPCASync is related to the _IPC.ID if the IPCAsync.Type is 1,3,4 otherwise it links to the IPCReturn.DstProcInstID when the Type is 2. IPC represents the Parent ProcInst/EventInst entry where the IPCReturn entry represents the return state from the child ProcInst.

 

Resolution

See Diagonsis section.