Symptoms
Workspace header loads, but displays "blank" space below it.
Browser Compatibility settings had no effect.
Diagnoses
Noticed In Event Log:
ISAPI 'C:WindowsMicrosoft.NETFrameworkv2.0.50727aspnet_isapi.dll' reported itself as unhealthy for the following reason: 'Deadlock detected'.
Followed By:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 6/5/2015 4:33:41 PM
Event time (UTC): 6/5/2015 9:33:41 PM
Event ID: 4f3317d65be84f30a1e026cfeceab121
Event sequence: 38
Event occurrence: 1
Event detail code: 0
Application information:
Application domain: /
Trust level: Full
Application Virtual Path:
Application Path:
Machine name: DT00RW53
Process information:
Process ID: 5896
Process name: w3wp.exe
Account name: someDomaink2service
Exception information:
Exception type: ClientException
Exception message: Send failed with error: An existing connection was forcibly closed by the remote host.
Request information:
Request URL: http://k2prod.denallix.com/redirect.aspx?process=Some Form 0419andfolio=200613andactivity=Supervisor Approval
Request path: /denallixSF/redirect.aspx
User host address: 10.132.113.27
User: SomeDomainrennan
Is authenticated: True
Authentication Type: NTLM
Thread account name: DENALLIXk2service
Thread information:
Thread ID: 44
Thread account name: DENALLIXk2service
Is impersonating: False
Stack trace: at SourceCode.Workflow.Client.SocketClient.Send(UInt32 sock, MessageType type, SByte* buf, Int32 len)
at SourceCode.Workflow.Client.SocketClient.Send(ClientMessage Msg, Int32 Start, Int32 Length)
at SourceCode.Workflow.Client.InternalConnection.AuthenticateSSPI(String domain, String user, String password, String secpack)
at SourceCode.Workflow.Client.InternalConnection.AuthenticateWindows()
at SourceCode.Workflow.Client.InternalConnection.Authenticate()
at SourceCode.Workflow.Client.Connection.Open(String Server)
at K2.Services.Tasks.Task.GetInfopathUrl(String processName, String processFolio, String activityName) in C:TFS DT00MV49RWMProduction_CodeWorkflowsK2Service BrokersK2.Services.TasksTask.cs:line 1005
at Redirect.GetUrl()
at Redirect.Page_Load(Object sender, EventArgs e)
at System.Web.UI.Control.OnLoad(EventArgs e)
at System.Web.UI.Control.LoadRecursive()
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
Custom event details:
We first checked browser compatibility settings as that is a typical cause of this issue, however, they were set correctly, and changing document modes had no meaningful effect on the page display. We checked the K2 hostserver log, as well as windows event logs, and found the above errors present in the event logs:
In troubleshooting this issue, we found a similar issue relating to deadlocks caused by IIS worker processes not returning from web service calls, which caused the normal thread count of "20" to be exceeded. We noticed that there were 28 running processes, and setting one of those processes (the one mentioned in the error) to stopped would cause workspace to respond momentarily, however, as soon as another process instance grabbed the thread, we were again left in a situation where we were out of threads, identified by the same two error messages appearing in the event log. We went ahead and increased the thread count via the k2server.setup file as a test measure to see what the behavior was. In this case it resolved the issue.
Resolution
In this particular case, increasing the thread count was the solution, however, in many cases of this error, it is not a good practice to increase the number of worker threads. The customer in this case had a specific recommendation from K2 Professional Services as to what the thread count setting should be in their environment. The typical solution in this case is to find the root cause of the threads not returning, however, in this particular case given the recommendation, we proceeded to increase the thread count to the amount specified and the issue was resolved.
