Symptoms

Initially, default Client Events were not sending notifications out. After restarting the service, the errors regarding notification disappeared but users are still not able to receive the notification. It turns out these notifications were delayed by 1-2 days. When the queue is cleared, which happens over the weekend, the notification is received in ~3 minutes. But this does not last for long as messages pile up in the queue quickly
 

Diagnoses

We applied debug assemblies provided by our labs to determine which causes the slowness in notifications. Based on the logs generated, there were domains causing timeout issues resulting to an additional ~43 second delay per domain to resolve the user in AD.
 

Resolution

It turns out that these domains were not in use anymore so we had them removed from the domain configuration via Workspace which helped improve the performance of client event notifications