The resolution of a mass spectrometer depends on the efficiency of separating ions with different ion masses. The mass of an atom is approximated to the sum of the masses of protons and neutrons inside, considering the masses of protons and neutrons as equal. However, the masses of the proton (1.6726 × 10⁻²⁴ g) and neutron (1.6749 × 10⁻²⁴ g) are not truly equal. There is a minor error in the expression of atomic masses relative to the simplest atom of hydrogen. For example, the mass of helium (2 protons + 2 neutrons) is not precisely four times the mass of hydrogen (1 proton). Also, when multiple nucleons are confined in the nucleus, it causes an increase in the potential energy and a decrease in the mass (from the interconversion of mass and energy). This is the reason for a lower mass of ¹²C compared to the mass of six deuterium (²H) atoms, even though the total numbers of protons and neutrons are the same in both cases. Atomic masses are often expressed in unified atomic mass units (u), defined as the rest mass of a free, ground-state ¹²C atom divided by 12. (The unified atomic mass unit is also known as the dalton with the symbol Da. It is often called the 'atomic mass unit' with the symbol 'amu', which were the name and symbol of a similar unit based on oxygen that was replaced by the unified atomic mass unit.)

For all atomic nuclei except free, ground-state, at-rest ¹²C, the exact atomic mass is not a whole number. As a result, the molecular mass is not necessarily a whole number. Usually, the atomic and molecular masses are approximated to the nearest whole number (known as nominal mass). The mass of a given isotope can also be expressed as relative mass, which is the mass of the isotope divided by the atomic mass constant, which is equal to 1 u. Table 1 lists the relative mass, exact mass, and nominal mass for different isotopes.

Table 1: The relative mass, exact mass, and nominal mass of the most abundant isotopes of hydrogen, helium, carbon, nitrogen and oxygen.

Normal mass spectrometers can distinguish ions that differ by at least 1 u. This poses difficulty in distinguishing different molecules with the same nominal mass values. Consider the example of cyclopentanone and cyclohexane: both have the same nominal molecular mass of 84 u; however, they have different exact molecular masses, chemical structures, and numbers of atoms (cyclopentanone: (CH₂)₄CO, 84.0573 u; cyclohexane: C₆H₁₁, 84.0936 u). However, a high-resolution mass spectrometer can even differentiate ions that differ only by 0.0001 u. A high-resolution mass spectrometer employs minute magnetic field steps to scan the charged species in the analyzing chamber. The minute steps are efficient in separating ions with similar molecular weights by deflecting the paths of ions according to their exact molecular weight.