Conceptions of precision


A topic in science concepts


Precision is an important notion when making, reporting or interpreting scientific measurements.

Precision and accuracy

Precision is sometimes confused with accuracy, although they are quite distinct. An imprecise measurement is simply not very fine-grained whereas an innaccurate mesurment is actually giving a false reading. (In any actual measurement there will always be some uncertainty.)

Precision is normally considered in terms of the number of signficiant figures in the reading: so the following readings would be to different levels of precision:

  • 0.3V
  • 0.30V
  • 0.300V

These measurments are different as one is given to the nearest tenth of a volt, one to the nearest hundreth of a volt, and the last to the nearest thousandth of a volt.

Avoinding ambiguity

Sometimes numbers may ambiguous in terms of precision. For example the mass

  • 30 000 kg

might seem to be to the nearest km, yet it is quite possible it is meant to be 'rounded up/down'. It would be possible to remove the ambiguity by writing the measurement differently – perhaps indicating the intended precsion in terms of significant figures, e.g.,

  • 30 000 kg (2 s.f.)
  • 30 000 kg (3 s.f.)
  • 30 000 kg (4 s.f.)

or, by using scientific notation ('standard form'), e.g.,

  • 3.0 ✕ 104 kg
  • 3.00 ✕ 104 kg
  • 3.000 ✕ 104 kg
A scientific measurement is not complete without its precision

A scientific measurment usually has 4 components:

  • a statement of what is being measured
  • a number
  • an indication of precision
  • a unit

Precise science

The precision of actual scientific measurements varies considerably. Here are some examples:

  • diameter of a human red blood cell: 8 µm
  • human gestation period: 40 weeks
  • atomic radius of helium: 0.208 nm
  • age of universe: 13.7 billion years
  • mass of Earth: 5.972 ✕ 1024 kg
  • density of lead: 11.34 g cm-3
  • volume of water on Earth: 1.386 billion km3
  • atomic mass of chlorine: 35.453 u
  • distance to the nearest star to our Sun: 4.2465 light-years
  • 'standard' gravity at Earth' surface 9.80665 m s-2 
  • Planck constant: 6.626 070 15 ✕ 10-34 JS
  • electronic charge: 1.602 176 63 ✕ 10-19 C
  • frequency used in atomic clocks: 9 192 631 770 Hz

 


Questionable precision

Meteors can heat the atmosphere up to 36 032˚F (22 222˚C)

"A rock travelling through an atmosphere has a buffeting and compressing effect on the molecules it encounters, causing them to pile up and dissociate into their component atoms. These atoms ionise to produce a shroud of incandescent plasma that is heated to extremely high temperatures – up to 20,000˚C (36,032˚F) – and envelops the space rock, causing it to become super-heated."

Natalie Starkey, Catching Stardust. Comets, asteroids and the birth of the Solar System

The figure of 20 000˚C is presumably meant to be an approximation, rather than a very precise number (see 'Avoiding ambiguity' above). However, someone (the author? a copy editor?) decided to convert from Celsius to the Fahrenheit scale and seems to have taken the 20 000 number to be precisely 20 000 (5 s.f.) or 2.0 000 ✕ 104 .