The rule I posted should work no matter what other groups people are in and have contribute access. As the rule below will only let one group see the fields when it is in edit mode. We use this method at my company for many things and it has always worked. Maybe there is just something I am not understanding regarding your requirements. Note, the rule in the image I posted is for the new responsive forms. If you are using one of the older formats, maybe that is the issue.
Also, when you say you have group 1 and group 2, are those SharePoint security groups or are you literally referring to two groups of people? If you are referring to groups of people, then you need to create two security groups (Group 1 and Group 2) and add the respective people to them. Then use those groups in the rule.