Symptoms
This problem has been intermittent for at least three weeks now, but I did not open a ticket since we thought it may have been related to our network connection issues. I am not so sure anymore, so here is the problem.
Several workflow steps are being marked as complete without errors, but they do not happen. These workflows steps write data back to the list item of reference in SharePoint. One can see that data has not been written to the list item of reference. As a result, subsequent workflow steps fail.
Since the workflows do not fail, as in error out, nor show any errors, this may be an application problem where the workflow seems to 'skip' steps at random.
I tried replacing the steps, which worked in some cases but not others. The procedure used is replacing a step with a similar one, then tie the new step with the preceding and subsequent steps, then delete the original step. After deployment, this seemed to work for only two or three executions, after which the same steps started to fail, or be 'skipped' again.
I talked to Jered about this, sorry, I don't mean to put anyone on the spot. I can show you the specifics over a shared screen session, and then attach information that you need. A lot can be showed here, so I am not sure what I should attach.
Diagnoses
During our troubleshooting session, full logging was enabled, we were able to reproduce the issue. The logs indicated an update list item operation before the list was updated from a previous smartobject update method. This leads us to the logic behind the workflow.
Resolution
Modifying the update events to the list and moving around some events seemed to resolve the issue.