Symptoms

Assume that you observe thread pool locking issues: when number of processes in Running status in ProcInst table exceeds 20, K2 smartforms, K2 Workspace and K2 Designer become unresponsive. Increasing thread pool limit settings does not help to solve this issue.
 

Diagnoses

This issue could be caused by different reasons but one of them is use of custom code which leverages K2 Client API inside of workflow in order to access process instance data (status, data fields) from running workflow.

There is a known issue with thread pool locking which appears when K2 Client API is being called from workflows, and it is not recommended to use it from workflow. Client API calls use the same thread pool as workflows (Worker Thread) which is limited by 20 threads by default. When all worker threads are busy it leads to queue growth, which is not limited by default for Worker Thread pool, so loads queued calls are waiting for current threads processing completion.

Potential workarounds/solutions to this are the following:
1. Increase worker thread pool limit. This is just temporary measure before you address root cause, as often it may help you only temporarily or just postpone next occurrence of your issue. You should be also aware that drastic increase of thread pool limits lead to negative performance impact as it does require more resources. On the other hand default K2 thread pool limits values we have now has been around since 2005, and hardware resources evolved/changed since then - there are multi core CPUs, hyper-threading, so in theory you can safely give little increase to default values.
2. For some scenarios it is possible to use Management API instead of Client API, but it has number of limitations, in particular, it does not allow access process data fields. It is possible to recommend design your workflows in a way which does not require access to data fields, as any data can be stored in external data base (and this is recommended approach), if your processes designed like that there is no necessity for you to use K2 Client API to access workflow data fields. It is also possible to use default process instance SmartObjects to get some basic data related with workflow instances.
3. It is possible to implement Client API calls "throttling" so that you never have 20 or more simultaneous calls of Client API from your workflow. For example if you have such problem with last step of your workflow which, let's say send emails to clients and uses Client API to extract some data at this step, then during mass "go to" from more than 20 instances of such process you may see the situation when more than 20 instances reach this activity at the same time causing thread locking.
As a workaround to this you may add random delay for start rule in this activity set to be between 1 and 180 seconds. This means that in most of the cases email will be sent with 3 minutes delay which is not critical for users. But also it means that when process reaches this activity there is a capacity to process a lot more simultaneous calls/queries. With 180 seconds delay in theory 180 x 20 = 3600 instances can reach this step simultaneously without any issues. If we exceed this number we will see the same thread locking issue again. This approach can be combined with slight increase of thread pool limits, e.g. 180 x 30 = 5400.
This approach may not be acceptable for all cases/scenarios, for example it wont work for you if you need immediately continue processing of activity and adding delay is not an option. Also it is not a good idea to apply this approach to all steps in workflow or even to multiple steps within one workflow.
 

Resolution

In such scenarios you should be able to adjust your workflow to use process instance SmartObjects to get basic information about process instances instead of Client API calls or use other workarounds listed above.