This error is usually associated with exceeding the size limitations in place for uncompressed workflow size in O365. This is going to be a hard limitation for compiling and processing within the Workflow Manager in O365. These scenarios will usually occur if there are hundreds (or more) of actions, or a large number of very large task forms in the workflow. I would recommend breaking the workflow out into multiple child / component workflows and attempting to publish again. The blog posts below will provide some additional information around size limitations and best practices.

Publishing Size Limit Within Nintex Workflow For Office 365 - Nintex Knowledge Base 

The workflow instance was too large to persist • Tomasz Poszytek  

https://www.nintex.com/blog/whats-new-component-workflows/