To a computer, words and sentences appear like data. But AI researchers want to teach computers how to actually understand the meaning of a sentence and learn from it.

One of the best ways to test the capability of an AI to do that is to see whether it can understand and follow a set of instructions for a task that it’s unfamiliar with. Regina Barzilay, a professor of computer science and electrical engineering at MIT’s computer science and AI lab, has attempted to do just that — teaching a computer to play Sid Meier’s Civilization.

In Civilization, the player is asked to guide a nation from the earliest periods of history through to the present day and into the future. It’s complex, and each action doesn’t necessarily have a predetermined outcome, because the game can react randomly to what you do.

Barzilay found that putting a machine-learning system to work on Civ gave it a victory rate of 46 percent, but that when the system was able to use the manual for the game to guide the development of its strategy, it rose dramatically to 79 percent.

It works by word association. Starting completely from scratch, the computer behaves randomly. As it acts, however, it can read words that pop up on the screen, and then search for those words in the manual. As it finds them, it can scan the surrounding text to develop ideas about what action that word corresponds with. Ideas that work well are kept, and those that lead to bad results are discarded.

“If you’d asked me beforehand if I thought we could do this yet, I’d have said no,” says Eugene Charniak, University Professor of Computer Science at Brown University. “You are building something where you have very little information about the domain, but you get clues from the domain itself.”

The eventual goal is both to develop AIs that can extract useful information from manuals written for humans, allowing them to approach a problem armed with just the instructions, rather than having to be painstakingly taught how to deal with any eventuality. Barzilay has already begun to adapt these systems to work with robots.