If the form is linked directly to list items you can’t do it because accessibility is governed by the item permissions set on Sharepoint’s side and Sharepoint does not support varying permissions on the individual column level as far as I can tell. Even if you lock certain parts out of the form people can still edit those columns on the Sharepoint side outside of the Nintex form, so locking them on the form isn’t going to do much in terms of security.

If you are talking about forms linked to workflow variables instead of list columns (for instance workflow start forms or task forms), you could use the inline function IsMemberOfGroup to hide/show fields depending on which Sharepoint group the user is a member of.

Is there a way to hide it, if the user is not a part of certain groups? 

Only on the form, you can put a condition on the visibility settings of the field, it’s under the control element settings you access by double-clicking on the field. Put a condition on the visibility of fn-IsMemberOfGroup(“Sharepoint group name”). It won’t do anything to prevent other users from editing the column directly through Sharepoint outside of the form though, any form settings only affect the form itself, they won’t secure anything on Sharepoint itself.