Numbers retrieved from calculated columns by a workflow are being shown with more decimal points than the column is configured to show.
Round the number by calculation rather than the display setting using the ROUND() function. For example:
1/3 equals 0.3333333 . . .
=ROUND(1/3, 2) equals 0.33
Nintex Workflow is retrieving the raw value of the numbers in these columns.