And I figured that out just now.
A userfield is a complex field. It contains ID - Displayname - Account - Presence - ...
All that info is stored in that field. And filtering on it doesn't work.
I believe if I make an extra (hidden) field which is just a textfield filled with the displayname, I can use that textfield as the filterkey. This field i'll have to fill with a workflow, but I'm pretty sure that'll work.
Thanks for the help all. couldn't have done it without you.