Please guide what am i missing.
I faced the same issue when trying to save a form for a list having "Calendar" view set as default. In that view SharePoint is not respecting the "modal setting" and is always opening form in a modal window. Anyway - when I was trying to save a form in that modal window, Nintex was showing an error: "* Invalid URL value. A URL field contains invalid data. Please check the value and try again."
I opened a ticket with Nintex Support, and after some time they wrote me back, that they very apologize, but currently Nintex Forms is not supporting opening forms for lists in "Calendar view", in a modal window. I see, this is not a problem only for a "Calendar view", but wider related to the modal window.
Anyway, as they suggested, I have opened a post on UserVoice: Support for Nintex Forms in list calendar view – Customer Feedback for Nintex , however they cannot any ETA on when this can be released.
I am not sure, but maybe this is your case too?