how to increase nintex workflow size threshold

there is no such a threshold you could configure.

have a look on this document how to deal with "big" workflows -  

here is the quote from document:

There is no hard limit in the size of a workflow, and no definite way of measuring the size. The

symptoms outlined in the following section are indicators that a workflow is too large or complex