I have never used BCS so I cannot comment on how it runs or if it can kickoff the workflow. It seems to me your workflow would need to be running 24/7 to pickup new data added to it.
As for duplicates. The collection operation has an option to remove duplicates, so you could do that right after your GET collection so that all duplicates would be removed before ever writing. That is if they are being pulled into the collection as duplicate.
Another option if the list has duplicates in it is to use the Query List action with a filter (this can be set in the Query Action's configuration screen) set to check against the title (or whatever filled/variables you want to check - may want to use your collection variables) of the item you are trying to add. This way if your query returns a result, you know that an item with that title/variable already exists then you skip writing to the list, which could be done in a Run If statement. I would probably put the Query List inside the loop so you're basically checking 1 item at a time if you are using your collection variables as query filters. If the variable you store the query results in is NULL then you know it doesn't exist and you can then write it to the list. There are a lot of different ways you could do it.
Thank you for your reply, so you are saying to do the following ?
Then right after add another collection
And then create the item?
My title column will be blank so that column will not be used I will be removing that.
So basically I will end up with additional "remove duplicate" collection operations after each collection variable correct?
If you are getting duplicates from the BCS table then yes I think that would work. Although, you may have to remove duplicates BEFORE your get statement and maybe outside of the loop since you already have your collections before getting into the loop. You may have to play around with that.
If you already have the items on your list and you want to check the list first to see if that item already exist before writing to it again, then I would query the list using the collection variables as filters and if the query returned no results then you know the item doesn't exist on the list and you could write to it.