can you maybe share your JS code so we can start inspecting what might be the problem?
If you have the Nintex form coming up on a modal dialog, then the call that opens the dialog is passed a fixed size and width. I'm not saying it's impossible to resize the form on the fly afterwards but-
So, you're off on an adventure ;-)
A quick look in Firebug indicates that this element-
<div class="nf-outer ms-formbody " ...></div>
appears to form the outer boundary of the form. Resizing it (manually) has some effect, increasing the height moves the tracking data at the bottom of the edit form (the created by info etc.) down, but reducing the width has no effect, so presumably it's "wrapping" the form elements, so the form fields set the min width.
Presumably it would be possible to build your own layout engine to re-lay out the form fields on demand, but that's not a small task.
I'm in Nintex O365 Forms, but I have accomplished this by making the last panel hit the bottom of my form. Then the bottom is self adjusting as panels are hidden or visible.
Retrieving data ...