it's a bug.
I've reported it to Nintex Support and they confirmed it.
unfortunatelly (and typically ), they do not provide any estimate when it could have been fixed.
from my investigation, all the calculations that evaluate to NULL result cause the alert to pop up.
as far as I was able to track it down, they forgot(?) to check for a null-ness of a variable at a place of code
We've managed to come up with a workaround if you absolutely need your form working until a patch is released.
If you open each calculated field on the form that use the lookup function and simply add +"" to the end of the query (to add a null value to the result) it will stop the error from occurring. The downside is that the calculated field will display "null" if no value is returned by the lookup. To solve that we added a rule to all calculated fields with lookups to hide the control if it equals "null".
So to recap, change the lookup to: lookup("test","test","test","test")+""
Add a formatting rule to hide the calc field:
*Sigh* Updating Nintex forms, a roll of the dice to see what breaks this time.
Though I'd try Santoshi's fix first as that seems more elegant.