TD-Gammon

Lore & Background

TD-Gammon was initially programmed 'knowledge-free,' using only a raw board encoding with no human-designed features. In early experiments, it reached a level comparable to Neurogammon, an intermediate-level human player. Tesauro then added hand-crafted features previously used in Neurogammon, such as 'advanced anchor' and 'blockade strength,' and the self-training program soon surpassed all previous computer backgammon programs. It stopped improving after about 1,500,000 games of self-play using a three-layered neural network with 198 input units, 80 hidden units, and one output unit.

During play, TD-Gammon examined all possible legal moves and their responses (lookahead search), feeding each resulting board position into its evaluation function. Its learning algorithm updated neural network weights after each turn to reduce the difference between evaluations of previous and current board positions—temporal-difference learning. The output layer had four neurons representing probabilities of White normal win, White gammon win, Black normal win, and Black gammon win. Backgammon wins were not represented.

TD-Gammon explored strategies that humans had not pursued, leading to advances in backgammon theory. For example, on opening plays with rolls of 2-1, 4-1, or 5-1, it found that splitting 24-23 was superior to the conventional 'slotting' move. Tournament players adopted this, and slotting disappeared from tournament play for several years. Backgammon expert Kit Woolsey found TD-Gammon's positional judgment superior to any human's.

Reader's Guide

TD-Gammon's significance lies in its demonstration that a neural network trained solely through self-play and temporal-difference learning could achieve a level of play just below that of top human backgammon players. In 1993, version 2.1 trained with 1.5 million games of self-play reached that level. In 1998, during a 100-game series, it was defeated by the world champion by a mere 8 points. Its unconventional assessments of opening strategies were accepted and adopted by expert players.

The program's legacy extends beyond backgammon. It is commonly cited as an early success of reinforcement learning and neural networks, and was referenced in papers for deep Q-learning and AlphaGo. TD-Gammon was never commercialized or released to the public, but it inspired commercial neural-network-based backgammon programs such as JellyFish (1994) and Snowie (1998). Its strengths and weaknesses were the opposite of symbolic AI programs: it excelled at intuitive positional play but was poor at endgame analysis and doubling strategies, the latter due to the neural network being trained without the doubling cube.

Did You Know?

• TD-Gammon's name comes from its use of temporal-difference learning (TD-Lambda) to train an artificial neural network.
• In 1998, during a 100-game series, TD-Gammon was defeated by the world champion by a mere margin of 8 points, mainly due to one doubling blunder that cost -32 points.
• TD-Gammon was initially programmed 'knowledge-free' and reached a level comparable to Neurogammon, an intermediate-level human player, before hand-crafted features were added.
• After 1992, TD-Gammon was trained with λ=0, degenerating into standard TD-learning, which saved compute by a factor of 2.

The Road to Mastery: Training and Version Evolution

TD-Gammon's journey from a simple 1-ply search in version 1.0 to the sophisticated 3-ply search of version 3.1 represents a remarkable arc of iterative improvement. The earliest iteration simply scored every legal next move with its neural network and selected the highest-scoring option. Versions 2.0 and 2.1 introduced a two-stage process: first pruning unlikely moves, then running a minimax analysis weighted across the 21 possible dice rolls, with non-doubles counted twice as heavily as doubles. Versions 3.0 and 3.1 expanded this to 441 possible dice combinations. The training itself was a feat of computational patience—version 2.1 required 1.5 million games of self-play to reach a level just shy of the world's best human players. Tesauro observed that the program stopped improving after roughly that many games when using a three-layer network with 198 input units, 80 hidden units, and a single output unit. The final version, 3.1, was specifically tuned for a high-stakes exhibition match, showing that even at its peak the program remained a work in progress shaped by a particular opponent.

Inside the Brain: Architecture and Temporal-Difference Learning

At its core, TD-Gammon is a three-layer neural network whose input layer carries 198 neurons—99 per color—encoding the raw board position as non-negative integers from 0 to 15 indicating checker counts at each location, plus additional hand-crafted features inherited from the earlier Neurogammon program, such as anchor strength, blockade quality, home-board strength, and the probability of a blot being hit. The hidden layer, which grew in size across later versions, processes these signals before passing them to four output neurons that estimate the probability of each possible game outcome: a normal win or gammon win for either White or Black. Backgammon wins, being exceedingly rare, were deliberately excluded from the model. What made TD-Gammon revolutionary was not its search during play—standard lookahead and evaluation—but how it learned. After every turn, the network adjusted its weights to close the gap between its own evaluation of the previous position and the current one, a process called temporal-difference learning. Tesauro discovered that a small lambda value performed nearly as well as a large one, so after 1992 he set lambda to zero, reducing the method to plain TD-learning and halving the computational cost.

Testing the Machine: Competitive Encounters

TD-Gammon's competitive record reads like a slow-motion collision between silicon and human intuition. In 1993, version 2.1, fresh from 1.5 million self-play games, reached a standard just below the top human players of the era. By 1998, the gap had narrowed to the point where a 100-game series against the reigning world champion ended with the human winning by only 8 points—a margin so thin it underscored how close the machine had come. The most dramatic showcase came at the 1998 AAAI Hall of Champions, where version 3.1 faced Malcolm Davis in a specially prepared exhibition. The program lost by 8 points, a result Tesauro attributed largely to a single catastrophic blunder: TD-Gammon chose to double and was gammoned for a 32-point swing. That one decision, a misjudgment of the doubling cube, erased what had otherwise been an extraordinarily close contest. These matches mattered not because the machine won, but because they demonstrated that a neural network trained purely through self-play could rival the deepest strategic instincts of a world-class backgammon player, all without being explicitly taught what good play looked like.

Reshaping a Game and a Field: Legacy and Influence

TD-Gammon's influence radiated far beyond the backgammon table. In the backgammon community, its unconventional assessments of certain opening strategies were so persuasive that expert players began adopting them, effectively shifting what was considered theoretically correct play. The program proved that a knowledge-free system, discovering its own evaluation criteria through millions of self-play games, could uncover insights that human experts had never articulated. In the broader AI landscape, TD-Gammon is routinely cited as one of the landmark early successes of reinforcement learning combined with neural networks. Its name appears in the literature of deep Q-learning and in the foundational papers behind AlphaGo, serving as a proof-of-concept that temporal-difference methods could drive a neural network to near-human performance in a complex, stochastic game. For Gerald Tesauro and IBM's Thomas J. Watson Research Center, the project demonstrated that the marriage of temporal-difference learning and artificial neural nets was not merely a theoretical curiosity but a practical engine for discovery, capable of generating new strategic knowledge in a domain long thought to require human intuition.