Prior to the end of the 18th century, guesses about the mass of the Earth had to be based upon assumptions concerning the composition of the Earth. For example, if one supposed that it had the density of granite, one could multiply the known volume of the Earth by the known density of granite.
On the other hand, according to Newton's theory of gravitation,
Now, the force exerted by one body upon another in the laboratory is very small, of the order of one (American) billionth of a pound. Therefore, if the effect was not to be obscured by potentially much larger forces, Cavendish had to isolate his equipment from such extraneous forces as those produced by friction, air currents, vibrations in the walls, etc. To this end, Cavendish used a torsion pendulum, suspending two small but heavy balls from a horizontal bar that was hung on a wire from a support above. Two even heavier balls he could remotely move closer to the small balls and watch through telescopes mounted in the walls of the room for a small but measureable deflection of the small balls.
Once the amount of deflection was known, the force could be calculated, for, according to Newton's laws of dynamics, one could determine the linear relation between torque and angular deflection by measuring the period of oscillations of the torsion pendulum.
In this simulation, the effect is exaggerated in order to make the principle clear.