Which form type are you using? When using the new responsive designer you should be able to apply a text mask to ensure a valid phone number. This will automatically prevent users from entering characters and also ensure 11 digits:

Should have included that! We're using Responsive Designer rather than New Responsive Designer. Is it possible within there?

Not with the same fanciness but you should be able to use the regular expression validation on your text box control: ^[0-9]{11}$

It will not directly prevent the user from adding other characters than digits but when saving a validation error will pop up

Thank you! Do you know if it's possible to limit by 10 or 11 digits/characters? 

This should do the trick: ^[0-9]{10,11}$

Nintex Mobile/Tel Validation Control Settings - Single Line Textbox --> Validation -->

Use a regular expression : Yes

Regular Expression :    dddddddddd

Regular expression error message : please enter valid Mobile no

Code :   dddddddddd   

10 digits -->( d for digit)