After K2 service restart all Default Client Event Notifications starts to fail with a "The specified string is not in the form required for an e-mail address" error
kbt156353
PRODUCTIssue
From time to time (not always) after a K2 service restart, all Default Client Event Notifications starts to fail with the following error message:
"61007 Permanent failure sending message System.FormatException: The specified string is not in the form required for an e-mail address."
Another service restart resolves this. The problem appears from time to time withouth any regular pattern.
Symptoms
Full error message text:
"Error","General","0","ErrorMessage","SourceCode.EventBus.Mapping.Resolver.ResolveXml","0 System.NullReferenceException: Object reference not set to an instance of an object. at SourceCode.EventBus.Mapping.DataService.GetUser(String fqn) at SourceCode.EventBus.Mapping.Resolver.ResolveXml(String destXml)"
"Error","MessageBus","61007","SendMessageException","MessageBusRuntime","61007 Permanent failure sending message System.FormatException: The specified string is not in the form required for an e-mail address. at System.Net.Mail.MailAddressParser.ParseAddress(String data, Boolean expectMultipleAddresses, Int32& index) at System.Net.Mail.MailAddressParser.ParseAddress(String data) at System.Net.Mail.MailAddress..ctor(String address, String displayName, Encoding displayNameEncoding) at SourceCode.MessageBus.Ews.Destination.EwsMessageDestination.GetEmail(AlternativeCollection alternativeCollection) at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext() at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext() at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source) at SourceCode.MessageBus.Ews.Destination.EwsMessageDestination.Send(String title, MessageExtendedInformation extended, IEnumerable`1 attachments, MessageBodyReader[] messageBodies) at SourceCode.MessageBus.Server.MessageBusRuntime.SourceCode.MessageBus.IMessageDestination.Send(String title, MessageExtendedInformation extended, IEnumerable`1 attachments, MessageBodyReader[] messageBodies) at System.Net.Mail.MailAddressParser.ParseAddress(String data, Boolean expectMultipleAddresses, Int32& index) at System.Net.Mail.MailAddressParser.ParseAddress(String data) at System.Net.Mail.MailAddress..ctor(String address, String displayName, Encoding displayNameEncoding) at SourceCode.MessageBus.Ews.Destination.EwsMessageDestination.GetEmail(AlternativeCollection alternativeCollection) at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext() at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext() at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source) at SourceCode.MessageBus.Ews.Destination.EwsMessageDestination.Send(String title, MessageExtendedInformation extended, IEnumerable`1 attachments, MessageBodyReader[] messageBodies) at SourceCode.MessageBus.Server.MessageBusRuntime.SourceCode.MessageBus.IMessageDestination.Send(String title, MessageExtendedInformation extended, IEnumerable`1 attachments, MessageBodyReader[] messageBodies)"
Resolution
This is a known issue which has been fully addressed in March 2018 CU FP21.
The problem is caused by the timing issue which in turn causes a URM object to be disposed, while another thread tries to call GetUser method. That’s why there is a ObjectReference error. This has been fixed in two fix packs for K2 4.7:
1. November CU FP 14 - Hotfix: Destination users not receiving Task Notification e-mails when a large number of workflow instances are running
2. March CU FP 21 - Hotfix: Destination users are not receiving task notification emails when many workflow instances are running
This was fixed first in Nov CU FP 14, but later another place was found that caused the same timing issue and this has been addressed in March CU FP 21. This latest fix fully closes this problem. Despite the aforementinoed KB articles describing the problem as "When running many workflow instances simultaneously, the destination users for tasks are not receiving their task notification emails" this fix is also valid for a scenario when notifications is not delivering shortly after a K2 service restart.
The error happens specifically when workflow sends notifications, and need to put things in the queues. To reproduce this at will you would really need to put a heavy load on the K2 server which gives you a better chance to reproduce this problem getting specific response timing that triggers the problem. This may also happen after a service restart.
Once the system enters into that error state only a service restart resolves this. It happens because internal counters stay out of sync, once the service restarts counters start again from 0 and it works again until the timing issue reoccurs causing the counters to go out of sync. The problem happens because an ObjectReference error appears but it is being ignored, and the email address field is populated with XML which then gets added to the queue, and when we try to send the email, and put the XML in the email address field, we get the string format error.
Apply March 2018 FP21 to fully address this issue.