I figured out, that a SPD Custom Action works only if I disable opening items in dialog mode and the workflow starts with the Default page.

I've also tried to use a custom page (it must not be a wiki page).

Thanks for sharing, glad it was a simple solution. I just wanted to ask though, in the custom action, did you use javascript to start the workflow directly using a SP.WorkflowServices.WorkflowServicesManager on the button action? or did the custom action take you to an application page that did the same thing? I like to hear how others accomplish similar tasks to gain some insights.

I used a custom action offered by SharePoint Designer. There is still the Problem that my users have to confirm the workflow on its start page... I just wanted to click on the ribbon to start it...

If you are trying to do this without a custom installed solution, then you should be able to create a script to start the workflow and put the script on a new page in a page library. Have the script run on page rendering to then start the workflow. The Ribbon button will be to navigate to this page, (then the script will execute to start the workflow using javascript csom, and redirect the user back to any page - maybe a success page).