This error is usually associated with exceeding the size limitations in place for uncompressed workflow size in O365. This is going to be a hard limitation for compiling and processing within the Workflow Manager in O365. These scenarios will usually occur if there are hundreds (or more) of actions, or a large number of very large task forms in the workflow. I would recommend breaking the workflow out into multiple child / component workflows and attempting to publish again. The blog posts below will provide some additional information around size limitations and best practices.