is there any way to upload an image in a nintex form to an image library in sharepoint?
Guess it is only possible the other way around at this moment.
Upload your image to an image library >> copy URL >> paste URL in image control in your form.
With a workflow?
Nintex attachment workflow
Attach the image to the form and then use Nintex workflow to get the attachment from the form using "Copy to SharePoint" action. This video by Dan Stoll that shows you step-by-step how to do this.
Nintex Workflow – Copy list item attachments into a library folder | The SharePoint Pub