I've 2 AD groups A and B. Some group B members also belong to group A. Can I add users to a SP group X using Nintex WF such that they are members of AD group B, but not A.
yes, it should be doable.
unfortunately there is not an OOTB action and will have to use UserGroup.asmx webservice.
if you search through the forum you will definitely find some examples how to use that service.
the service exposes two methods - GetGroupCollectionFromUser and GetUserCollectionFromGroup that might help you.
UserGroup Members (websvcUsersGroups)
Retrieving data ...