Symptoms

In some cases, the K2 blackpearl service will crash with an Out of Memory exception. Usually, restarting the K2 blackpearl service is sufficient.
There is a workflow running when the Out of Memory exception occurs, always on one of three activities:
- Get User Reference
- Copy Document
- Data Transfer (Copying values from a List Item Reference to local variables)
 

Diagnoses

Several things can lead to increasing memory utilization, and "garbage collection" doesn't always do the greatest job of releasing memory. Based on host server logs, the Out of Memory exceptions were occurring on SmartObject calls. With only error-level logging, it was not possible to narrow down to a specific SmartObject call.

Some questions that may help determine the cause include:
Are there any other symptoms that happen leading up to the out of memory exceptions? For example, do users find that retrieving their worklist, or opening a task, takes increasingly longer?
How many active process instances are there?

This is hardly an exhaustive list. However, these kinds of questions may lead you towards an area to focus on. If not, consider enabling full debug-level logging and waiting for the service to crash again. This would at least allow you to identify what was happening just prior to the crash. I would also encourage you to either have the logs written to a drive other than C:, or at least monitor the drive space carefully. The service will definitely stop responding if you run out of disk space.
 

Resolution

There are two options:
1. Implemented an auto-restart of the K2 blackpearl service every night, to prevent excessive memory buildup
2. Enable verbose host server logging, to isolate which SmartObject method(s) result in the Out of Memory exception, then investigate possible logic loops in workflows, and other areas where looping may occur