Known Issue: Filtering system groups when using the GetUsersByGroup SmartObject does not work.
KB003399
Issue
You cannot filter system groups, such as Domain Users, when using the GetUsersByGroup SmartObject of the Active Directory AD Service 2 service.
Cause
The methods of the AD Service 2 SmartObjects translate to a single Lightweight Directory Access Protocol (LDAP) query by design. System groups are stored differently in Active Directory and require multiple LDAP sub-queries.
Workaround
Depending on your scenario, you can use the GetUsers SmartObject and set Domain Users (or another system group) as an input parameter, and then filter by other criteria.