Are the minimum and maximum rows configurable at all? This doesn't work:
fn-If(Is New Mode, 2, 1)
although documentation says this should be possible (https://help.nintex.com/en-US/o365/#o365/O365Forms/FormControls/RepeatingSectionControl.htm%3FTocPat...) fast experiment proves that it doesn't.
Even a valid expression put in that setting is being replaced back with a default value.
What you can try is to create a rule, added to repeating section, that invalidates it if number of rows is bigger than expected, e.g.:
If(And(Is New Mode, count(FieldInRepeatingSection)>2), "Number of rows for new form cannot be higher than 2", If(And(Is Edit Mode,count(FieldInRepeatingSection)>1), "Number of rows for edit form cannot be higher than 1"))