The limit is there to prevent accidental endless loops from flooding the system. Do you have a workflow start rule that is preventing instances from starting?  That might still count against the limit.  You might consider opening a support ticket if the problem continues.

I added a start rule on another workflow, 

then I started some instances, there were some instances meet the start rule, then show the error like this:

and it will not populate a instance at k2 management, after I started 11 instance normally like this:

The limit error happed.

So I think it has nothing to do with the start rule.

@DavidL2 What is the limit and can this be configured to higher limit?

It can be changed. Go to your management site and drill down to Workflow Server -> Workflow.

On the right side, click the Settings tab.  On this tab you can configure these values.

If you are still having issues, go to Customer Central and open a support ticket.  They will assist you.

@DavidL2 Thank you.  What is the recommended max you should not go past?

It is hard to know in these types of settings what are reasonable settings.

That's a good question.  I don't think there is one answer for all cases.  Estimate how many workflows you think may be starting in an amount of time, then start with this setting and adjust if needed.  This setting is used to throttle workflows that accidentally start other workflows in an endless loop.  I would start with a conservative value and then adjust up in small increments.