Can you confirm, do you mean an attachment in the email that is sent as part of the flexi task or an attachment to the list item?

Yes, the attachment sent as part of the flexi task.

This is nothing about Nintex. How files will be opened when you click on a link (e. g. attachment in taskform) depends on the settings of your browser adn your SharePoint server. Additionally you will need some word web app or similar to be able to open/edit documents inside the browser.

Set the default open behavior for browser-enabled documents (Office Web Apps when used with SharePoint 2013) 

But the mail attachment as a part of "To do task" action is opening in browser but not the "Assign Flexi task" and in the SharePoint 2010 library settings, "Default open behavior for browser-enabled documents:" is set to "Open in client application".

If the attachment is an actual email attachment (i.e. the same as me attaching a doc to an email I'm sending a colleague) rather thna a link, then it is not actually under the control of SharePoint or Nintex. It'll be an independent file in its own right and how it is opened is dictated by your windows default programs settings. If it's a link to an item in the email and the link is to an item in SharePoint then you can manage the settings in the Library settings, or even in the query string of the URL link.