Problems with deadlock on K2 database.

  • 6 December 2010
  • 10 replies
  • 18 views

Badge +1

Hi everyone,


I have encountered some problems with deadlock on K2 database.


Several incidents have been existed. Normally, in the wofkflows, in Activity A, some data is updated to a external database. But it seemed when the K2 database encountered deadlock, the commands to update data to the external database were not execucted. However, K2 Server still marked this Activity as Completed and consider it as there is nothing wrong with the system.


 


Has anyone encountered this error? And do you have any solution for this?


Need helps.


10 replies

Userlevel 4
Badge +14

Is this an IPC process? Can you please provide more detail on the deadlock form the K2logs?


Vernon

Badge +1

No, it's just a server event.


Moreover, it seemed the activity marked as Completed but in fact no code has been executed at all. It include code to update datafield value of K2 process as well as data value of a external database.


Here is the K2logs showed when this incident happened:


""17014466","2010-11-18 15:19:23","Error","General","1","GeneralErrorMessage","K2Sql.GetProcInsts","1 Transaction (Process ID 191) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.HasMoreRows()
   at System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout)
   at System.Data.SqlClient.SqlDataReader.Read()
   at SourceCode.Workflow.Runtime.K2Sql.GetProcInsts(Int32 serverid, Int32 batchid, Int32 count)","","","SGCSGMSNET06:c:program filesk2 blackpearlHost ServerBin","17014466","795da4a97d2c46cfb5bb6007671cec0b",""
"17017676","2010-11-18 15:26:48","Error","General","1","GeneralErrorMessage","K2Sql.GetProcInsts","1 Transaction (Process ID 85) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader()
   at SourceCode.Workflow.Runtime.K2Sql.GetProcInsts(Int32 serverid, Int32 batchid, Int32 count)","","","SGCSGMSNET06:c:program filesk2 blackpearlHost ServerBin","17017676","0ade0aa482674c9db1a3a028def86f53",""
"17018653","2010-11-18 15:28:14","Error","General","1","GeneralErrorMessage","K2Sql.Log","1 Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.    at SourceCode.Workflow.Runtime.K2Sql.Log(Int32 id, Int32 pid, ArchiveX ar)","","","SGCSGMSNET06:c:program filesk2 blackpearlHost ServerBin","17018653","5fdcb2fec0dd4497be0d03dd8039e454",""
"17018654","2010-11-18 15:28:14","Error","General","1","GeneralErrorMessage","K2Sql.Log","1 Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.    at SourceCode.Workflow.Runtime.K2Sql.Log(Int32 id, Int32 pid, ArchiveX ar)","","","SGCSGMSNET06:c:program filesk2 blackpearlHost ServerBin","17018654","95435a78ed8f4ba498d9a2362c6d1702",""
"17018657","2010-11-18 15:28:25","Error","General","1","GeneralErrorMessage","K2Sql.RemoveProcInst","1 Transaction (Process ID 174) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at SourceCode.Workflow.Runtime.K2Sql.RemoveProcInst(Int32 pid)","","","SGCSGMSNET06:c:program filesk2 blackpearlHost ServerBin","17018657","bfdc83d0765b463a8bf829f9106d7a4f",""

Userlevel 4
Badge +14

What version of Blackpearl are you running? 807 or 4.5 and with what KB applied?


Vernon

Badge +3

Issue may be related to the following


http://help.k2.com/en/KB000678.aspx


Please reference the URL and see if you agree with me.


*Deadlocks can happen due to low memory, worker threads
etc. There is a detailed explanation here if you are interested.


http://msdn.microsoft.com/en-us/library/ms178104.aspx


Kind regards,

Badge +13

Regardless of the reason it deadlocked, workflow should not mark the server event as complete if nothing was run.   It should go into an error state.


K2 should try to replicate the dead lock scenario (by forcing low memory/worker threads) and generate correct response.

Badge +3

It depends on the build the client use, because a lot of enhancement have been done regarding deadlocks. To reproduce a deadlock will be overkill because it can depend on a lot of reasons.


Best option for clients will be to apply the latest build (build 1230 [08 Dec 2010] )of K2 and see if deadlocks still occur, if so the client will have to log a support call for K2 support to investigate the issue


e.g. SQL trace, etc.


Kind regards,

Badge +1

Thank everyone for your help.


The version I'm using is 4.8210.3.690 (0807).


Although till now I'm not sure if the deadlock is the reason for that malfunction or not, which I'm confused here is why K2 still mark that Activity as Completed while no even single code line was executed? It makes things become so complicated cause we cannot detect those errors until users report to us

Userlevel 4
Badge +14

What Brink mentions is true there has been deadlock fixes in the latest build 4.5 KB1230. What are the chances for an upgrade? If not then yes you will have to go the K2 Support route and if this route is taken please ensure that you have deadlock graphs to send along.


Vernon

Badge +1

Hi everyone,

Our K2 Server version is 4.8210.3.670 (0807).

The hotfix article states it applies to

K2 blackpearl 0807 (4.8210.3.0)
K2 blackpearl 0807 (4.8210.2.450)

Does this hotfix still relevant/applicable to our current K2 Server version?

Userlevel 4
Badge +14

If you are referring to the article KB000678 then yes this applies to you. This fix is included in update KB000690. You can apply this patch to see if there are any improvements, if it doesn’t help I suggest logging a K2 Support ticket together with the Deadlock Graphs to exactly where the deadlock occurs.


HTH


Vernon

Reply