Remove 1 Permission Group without removing all Permission Groups

I am using NAC on SharePoint O365 lists.  A SharePoint group called “Visitors” has permissions to all of my lists.  I need to remove the “Visitors” group from the item, but do not want to remove any other group or people permissions.  I see I can remove all permissions, but I only want to remove 1 group.   

I can’t remove all permissions and then add some back in because this workflow is a component workflow and each list it runs on will have different groups.  

Is there a way to remove permissions of 1 group without removing all permissions?

Thanks!