If I were designing such a solution, here is how I would do it:

1. Allow as many process instances get kicked off as they want (just as you said).
2. First step in the process is to call the workflow instances smartobject and determine if there are other instances of the process currently running.  If there is, either exit the workflow or enter a loop state and check again later. 

Workflow Reports > Worklflow Solutions > Solution Name > Process Name - run a get list and filter on status to only show active processes

You could also use the Process Instance smartobject - Workflow Reports > Workflow General > Process Instance.

If it is important that the processes run in exact order they were started, you could build some type of queueing solution built on a SQL Server table.  Have the process assign itself a number with perhaps an identity column and by inserting a record.  Also use a "status column" perhaps that indicates whether or not that slot is active. Then on each instance once the process finishes, check to see if the next "slot" has been created, and if so, set it to active.  Then on the loop check have the process check to see if it's slot has been "Activated".  If it has, run.

Edit to my last post:

My solution would have a logic error if built as I explained it.  You would need to perform an additional check on this hypothetical queue table:  if I'm not the active process, is there an active process assigned?  If not, I'm active.

Otherwise, you could have a situation where the only currently running process completes, and the next submitted process wouldn't know what to do becuase it would be created with a default "inactive" status.