The picture should explain things. Most of my trees are missing the "new" node. I don't know what you call it. This happens from time to time. It's really annoying to work around. I can't place any actions after the action set.
The only way to get things added to the branches when this happens is to add them above the last action on the tree. This works right up until you realize that you also can't drag/drop any of the existing branch items to reorganize it.
So now you have a bunch of actions out of order and you can't reorder them. Probably just need to delete them and start fresh right? Oh, can't do that either.
Nintex Workflow 2013 (188.8.131.52)
It's hard to say if it ONLY happens inside a SM. I use state machines for almost every workflow. For me, it has happened inside the SM every time. It could happen outside of one and I just haven't experienced it yet.
The only way I can resolve it is to delete the branch entirely.
One thing I noticed is that it happens when copy and pasting an action set, and then saving without any actions on the branch after the action set. If the last action is an action set then this happens. So far, if I put a blank notification action after the action set, it hasn't happened.
It's been a while (probably a couple of years ago), but I have noticed weirdness with copied and pasted actions. Seems like the XML configuration of the action sometimes does not carry over to the copied action properly.
Might be worthing testing with another browser to see if the issue still occurs.
This happened to me in an older version of Workflow (for On Prem 2013). I believe that we found that anything wrapped in an Action Set would not be given a proper continuing node.
I do not remember if we just upgraded, or if dropping in a different combo of action-set helped, but because I can't see the top of the image (beginning of the state node) it kinda looks like you might be running into the same problem.