by Rishidev Chaudhuri
One of the main aims of modern science is finding mathematical expressions to describe the relationships between observed quantities. For example, Newton's law of gravitation tells us that the force of gravity between two bodies depends in a certain way on their masses and the distance between them; thermodynamics tells us that the pressure of a gas depends in a certain way on its volume and temperature; and an economist studying income might conclude that income increases with educational level according to some functional form.
Sometimes these mathematical relationships emerge from an underlying model. We might model a gas as made up of molecules that collide with each other and the walls of the container, think that pressure is a measure of collisions with the walls and temperature a measure of kinetic energy, and then our functional form is a result of mechanistic insight into pressure and temperature. In other cases, the relationships serve to provide a summary representation of the data (instead of giving you a list of pressures at various temperatures, I could say pressure=3*temperature) and, even without providing an explanation of how the relationship came to be, allow us to make predictions about new data (for example, I might have observed the pressures at temperatures of 30 degrees and 60 degrees and want to predict the pressure at 90 degrees).
As we choose a relationship (or hypothesis) to explain a given set of data, the two goals of accounting for the existing data and making predictions for new data are often in conflict. Look at the graph below, which plots the simultaneously measured values of two quantities, X and Y.
Say we're trying to describe this relationship in a way that allows us to predict the values of Y at unobserved points (for example, we haven't measured the value of Y when X is 0.25 and we want to predict this). A common thing to do is to draw a line along the middle of this scattered cloud of points and use this as an approximation of the underlying relationship.