I am not sure if there is any OOB Nintex way to get the file size of the attachment, (people can correct me here) .. I could think of couple options here ..
1) You could manage the file size restriction by setting the web App file upload property to 50 Mb. If this restriction is placed in SharePoint then Nintex will adhere it too.
2) Jquery does not have access to file system so I am not sure if you would be able to get any file size value for it, but you could use HTML5 as that has some file size property you could use, now again this would work only for newer verison of the browser and also you would not be able to use Nintex to customize your form.
I believe that you can reduce the number found here:
Central Admin -> Manage Web Application -> Select Web Application -> General Setting -> Maximum Upload Size
By default, I think it's at 250MB.