Rounding Numbers

By: Tim Loftus

The analysis is complete, your technique was good, there were no interferences, and your equipment was operating at peak efficiency. How accurate is the result?

That depends on how accurate your glassware and instrumentation is and on the purity of the chemicals used. The final result cannot be more accurate than the least accurate part of the analysis. Therefore, how you record a result should reflect the accuracy you have achieved. For example, a total phosphorus value of 5.896 mg/L suggests that you have achieved a high degree of accuracy, but it’s also something that should raise a red flag for those who read it. This level of accuracy is generally unachievable because much of the laboratory equipment, especially glassware, does not have that level of accuracy. Again, the result cannot be more accurate than the least accurate part of the analysis.

This means that you will need to round numbers to reflect the accuracy of the result. The most common method for rounding numbers is as follows:

1. Drop all digits in a number that are not significant. (Refer to our article Significant Digits.)

2. If the number immediately to the right of the significant numbers is between 6 and 9, round up one unit. For example, if you wish to round the number 53.68 to three significant digits, round the 6 digit to 7 since the non-significant digit 8 is between 6 and 9. The result is 53.7. To round 53.68 to two significant digits, round the 3 to 4 since the non-significant digit immediately to the right is 6. The resulting number is 54.

3. If the number immediately to the right of the significant numbers is between 0 and 4, leave the significant numbers unchanged and drop the non-significant numbers. For example, 53.64 rounded to three significant digits is 53.6.

4. If the number immediately to the right of the significant numbers is 5, round to the nearest even number. For instance, 53.65 rounded to three digits is 53.6. For 53.75, the rounded number becomes 53.8.

When performing a series of calculations, round the final result rather than the intermediary numbers. For example, the calculations for BOD use several numbers, all having different amounts of significant digits. Perform the calculations, then round the result. For a real life example using the BOD equation, and inserting typical numbers, we have:

𝐁𝐢𝐨𝐜𝐡𝐞𝐦𝐢𝐜𝐚𝐥 𝐎𝐱𝐲𝐠𝐞𝐧 𝐃𝐞𝐦𝐚𝐧𝐝 (𝐬𝐞𝐞𝐝𝐞𝐝), 𝐦𝐠/𝐋 =

[(Initial DO, mg/L) − (Final DO, mg/L) − (Seed Correction, mg/L)] [300 mL]

Sample Volume, mL

Initial DO: 8.95 mg/L

Final DO: 5.25 mg/L

Seed Correction: 0.6952

BOD Bottle: 300 mL (assume three significant digits for this example)

Volume of sample used: 6.0 mL

[(8.95 mg/L) – (5.25 mg/L) - (0.6952 mg/L)] [300 mL]

6.0 mL

= 150.24 mg/L

Since the sample volume (6.0 mg/L) has the least number of significant digits, round the final answer to two significant digits. The rounded result is 150 mg/L.

If you incorrectly rounded each of the numbers in the calculation to two significant digits, you would get:

[(9.0 mg/L – 5.2 mg/L) – (0.70 mg/L)] [300 mL]

6.0 mL

= 155 mg/L

When the answer is rounded to two significant digits, the result is 160 mg/L.

The correct result is 150 mg/L, the incorrect result is 160. As you can see, there is quite a difference. And this difference may put you in violation of your discharge permit or place your analysis outside acceptable QA/QC ranges.

However, there are a few exceptions. For example, some numbers are assumed to have unlimited significant digits. If you wish to calculate the average of three duplicate final effluent BOD results, you will add the three results together, then divide by 3 (the number of BOD bottles tested). In this case, the number 3 used in the average calculation is assumed to have unlimited significant digits, not one significant digit.

The information in this article is very general. As usual, check your federal, state, and local regulations. You may have additional regulations or requirements that you must meet.

References:

Standard Methods for the Examination of Water and Wastewater 23rd edition 2017, part 1050 Expression of Results, section 1050 B. Significant Figures, #2 Rounding Off.

Wastewater Sampling for Process and Quality Control MOP #OM-1 WEF 1996

NEWEA Formula/Conversion Table for Lab Analyst Exam

May 20, 2024