How an 1847 Formula Trains ChatGPT: Cauchy’s Gradient Descent

The formula that taught AI to write

ChatGPT and other modern language models are trained using an algorithm that appeared in 1847. It is gradient descent:

w ← w − η∇L

Without it, no modern neural network would exist. The method was not invented by OpenAI or Google, but in France almost 180 years ago, when mathematician Augustin-Louis Cauchy looked for a faster way to calculate the motion of celestial bodies.

Imagine standing on a mountain covered in thick fog. You cannot see the valley, but you feel the slope under your feet. You take a step in the direction of the steepest descent, measure the slope again, and take another step. Step by step, you reach the valley.

A neural network works the same way:

• The mountain represents the model’s errors, or loss.
• The valley is where errors are minimal and answers are most accurate.
• Each step is an update to the model’s parameters.
• The slope is the gradient, the direction in which the error grows fastest.

The only difference is that this mountain does not have three dimensions; it has billions of parameters. That is why training modern AI requires enormous compute and careful mathematical implementation.

Cauchy calculated planetary orbits, while his formula became the foundation for machines that write text, answer questions, and generate images.