C:\Windows\Microsoft.NET\Framework\v4.0.30319\installutil.exe "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\16\BIN\ExternalPlatform\Nintex.External.RelayService.exe"

Location of the files may differ depending on your installation. I was then  able to start the service in Central Admin without errors.