Story comprehension
The robots of Westworld are not programmed solely by software developers. The bulk of the work is done by professional writers, who give each character a unique backstory. These stories give them the memories and depth they need to seem real to the park guests. When asked who they are, what they’ve done or why they feel a certain way, they can consult their backstory to find out the answer.
Being able to answer questions about stories is a fundamental requirement for being able to pass the Turing test, which the show tells us started to happen “after the first year of building the park”. But Turing proposed his test as a kind of thought experiment, not as a useful yardstick for measuring progress in AI. A machine either passes or fails and that’s not very useful for figuring out how close we are.
To fix this, in 2015 Facebook’s AI lab introduced the bAbI tests in a paper called “Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks”. Quoting from the paper’s abstract:
To measure progress towards [building an intelligent dialogue agent], we argue for the usefulness of a set of proxy tasks that evaluate reading comprehension via question answering. Our tasks measure understanding in several ways: whether a system is able to answer questions via chaining facts, simple induction, deduction and many more. The tasks are designed to be prerequisites for any system that aims to be capable of conversing with a human.
In other words, before you can hope to pass the Turing test you must learn to pass bAbI.
The test is a large, auto-generated series of simple stories and questions, that test 20 different kinds of mental skill. Here’s one that checks the machine isn’t distracted by irrelevant facts:
Mary went to the bathroom. John moved to the hallway. Mary travelled to the office. Where is Mary? Answer: office
Here’s a harder one that tests basic logical induction:
Lily is a swan. Lily is white. Bernhard is green. Greg is a swan. What color is Greg? Answer: white
... continue reading