Symptoms

Below you may find some generic guidance for the cases when you observe high RAM usage by K2 host server service and suspect some sort of memory leak issue.
 

Diagnoses

First it is pretty clear that post reboot memory consumption is lower than after server works for a while. Once you rebooted server it starts clean all previously allocated memory is clear and hence low RAM usage. But as server warms up it starts checking if it has tasks to process and fires other activities like users and group resolution by ADUM manager, recording data in identity cache and so on. The more task processing threads are active the more memory is required.

Second important point: if your process design or custom code has any design flaws or hardware is poorly sized for intended load processing queue starts growing and it may lead to resource overuse. I.e. it is not a memory leak but bottleneck caused by such things as (listing below based on probability of being cause of such issue):

1) Custom code or process design (if this is the case then normally you are not able to reproduce this "memory leak" on empty platform with no running processes). Please refer to K2 blackpearl Best Practices document for process design best practices as a starting point of your investigation. There are enough reported cases when high memory usage was caused by inefficient process design choices (something like upload to DB or updating 20 word documents properties in a row designed so that file is being downloaded/uploaded 20 times from SharePoint instead of doing batch update with one download/upload of a file).

To verify this when next time you will see this high memory usage state before doing reboot execute the following queries against K2 database:

Query A:

SELECT * FROM >K2].EServer].RProcInst] WHERE .Status] = 1

It will give you number of running processes at specific point in time. Constantly having 20 process or more in status 1 may indicate a problem, but more importantly to execute this query multiple times with 1-2 minutes interval and see if some of the process instances with the same ID stays running constantly or for a very long time. This will be likely your "offending" process and you will want to check at which step it is so slow and so on.

Query B:

SELECT TOP 200 ID, DATALENGTH(State) AS StateSize, Version, StartDate, Originator, Folio, Status

FROM Server.ProcInst WITH(NOLOCK)

WHERE Status IN (1, 2)

ORDER BY DATALENGTH(State) DESC

This query will return processes with largest state size in bytes. If any of the processes has state size more than 1 MB this is problematic process which causes memory over use most likely due to use of looping within the process.

2) Slow response/high latency of external systems and/or network. Depending on design of your workflows they may have dependencies on external systems (SQL, SharePoint) and it could be the case that slow response from their side cause growth of queue on K2 side with memory usage growth. A) At the time of an issue verify K2 host server logs and ADUM logs to see if there are any time out or communication type of errors/exceptions. Check all servers which comprise your environment (K2, SQL, SharePoint, IIS) - watch out for resource usage spikes and error in event viewer. K2 relies heavily on SQL where K2 DB is hosted and if it is undersized or overloaded (scheduled execution of some SSIS packages, scheduled antivirus scan or backup) so if it is slow to respond you will see memory usage growth/slowness on K2 side. If your servers virtualized confirm your K2 vServers placement with virtualization platform admins - K2 and K2 DB SQL instance should not coexist on the same vHost with I/O intensive apps (especially Exchange, SharePoint). NOTE: It never woks in large organizations if you just ask your colleagues (database or virtualization admins) whether all is OK on their side - you will always get a response that all is OK. You have to ask specific questions and get explicit confirmation of things like VM placement, whether your K2 DB SQL instance is shared with any other I/O intensive apps. You want to have a list and go through it eliminating possibilities. It is not uncommon to see clients spending months troubleshooting performance and reviewing their K2 solutions inside out and searching for a memory leak while issue could have been solved much earlier by something like careful check of bottlenecks/load on K2 DB SQL server instance (note that issue on SQL side can be something as intermittent as SSIS package running on schedule once a day).

3) Memory leak. This is highly unlikely as K2 code goes through strict QA and testing, and only rarely we see cases with memory leak type of an issue which had to be fixed in K2 ? usually it happens with old versions and in very specific, not frequent scenarios.
 

Resolution

If you not observe prolonged memory usage spikes which not going away by themselves, but just see that server at times maxing out resources but then it goes away with no intervention from your side (like K2 service/server restart) then it looks more like insufficient hardware type of situation. Memory leak implies that you need to restart K2 service or something like this to resolve it. And in this case and you verified all the points mentioned above it is worth placing K2 support case and prepare all K2 logs along with memory dumps collected in the low and high memory usage states.

Speaking about hardware sizing for K2 server you can refer to the following documents:

K2 blackpearl Performance Testing Results

K2 blackpearl Performance Testing White Paper

Performance and Capacity Planning

K2 blackpearl Performance Testing

Also see this K2 community KB - "K2 Host Service CPU usages close to 100%"