I still have to absolutely confirm this in a test environment, but here's what I just witnessed and verified (just lost most of the morning looking at it)...
If you connect fields in a Workflow Task using a Nintex Form to workflow variables, it causes the document related to the workflow to silently check in when the task is submitted.
In my case I had a library workflow to do approvals of versioned documents - with a state machine with a "management" branch, where the user has a flexi task to choose the next route for the state machine to take. The flexi task uses a Nintex Form, and has an extra field on it connected to a workflow variable. Here's the clue to what was going on - when the task was submitted, the form took some time to submit - following that the workflow complained that the document was not checked out (even though it's approval status was still "Draft").
To resolve it, I added a conditional block after the rogue task, requesting a checkout if the "checked out by" field is empty (and then waited for the checkout to happen).
Another solution would be to use a "Request Data" task instead of the Flexi Task - but as we know, the Flexi Task has reminders and escalations that the other tasks do not.
Is the document check-in a bug ?
This does sound unusual. Does your workflow or form update any of the columns on the library list item? This would certainly require a checkout to be required before that action could take place.
Here's the weird thing - the document is already checked out, and has already been updated by the workflow several times. The workflow task does not update the document directly - only a workflow variable. I thought it might be that the workflow task obviously runs in the context of the assigned user, instead of the owner of the workflow, but that doesn't make sense because I have other (stock) tasks that don't increment the version.
I need to do more testing to narrow it down.