GoKawiilGame construction
We manually constructed the 121 two-player competitive strategy game variants played on M × N grids, where M is the number of rows and N is the number of columns. One goal in creating these games was to ensure there is enough diversity in board size and rule structure as well as systematic variance. To that end, we designed a series of variations on square boards: on 10 × 10 boards, we have K-in-a-row for K from 2 to 10; for 3-in-a-row, we have M by N = M boards from 3 to 10; we also include a few 5 × 5 boards with varying K. We additionally included a few other square boards varying in complexity, as well as rectangular boards ranging in size from from 1 × 5 to 5 × 10, and we have integer 2 ≤ n ≤ 6 for K-in-a-row. To assess how people reason about games that are not physically realizable, we included three ‘infinitely’ sized games with K = 3, 5 and 10 for K-in-a-row. These categories give us 41 games with typical ‘M–N–K rules’ (for example, where the players take turns and have the same objective: ‘make K in a row, where horizontal, vertical, or diagonal all count’). This set includes the standard tic-tac-toe as well as 4 × 9, 4 in a row wins from ref. 6. We then created a number of games with varied game rules. We kept the selection of board sizes and K-in-a-row fixed across categories (ten games within each category). The selection included 3 × 3, 4 × 4, 5 × 5 and 10 × 10 boards and n ∈ {3, 4, 5, 10}. We also designed games with more atypical rules, varying the winning conditions (for example, K-in-a-row loses and diagonal connections do not count as wins) and first-mover dynamics (for example, player 1 can place two pieces on their first turn). These categories give us an additional 80 games, totalling the 121 in our dataset. We manually implemented an automated win condition checker that permits flexible game assessment over all game types.
Game name codes
Games are expressed in abbreviated form throughout the paper. Games are described by their board size (rows × columns) and the number K in a row to win. For example, ‘4 × 4, 3’ means the game is played on a 4 × 4 grid and the first person to get 3 pieces in a row wins. Unless otherwise stated, horizontal, vertical and diagonal all count. If only horizontal and vertical 3 in a row count, the game would be written as ‘4 × 4, 3 HV’. If only diagonal counts, then the game will be written as ‘4 × 4, 3 D’. If the constraint only applies to one player (for example, P1 can only win horizontally and vertically, and P2 can win any way), that is represented as ‘4 × 4, 3 (P1 HV)’. If one player can go twice on their turn, for example, the second player (P2) can play twice, that is written as ‘4 × 4, 3 (P2 2p)’. A misère game (where first to K in a row loses) is written with an ‘L’, for example, ‘4 × 4, 3 L’. In our dataset, multiple rule modifications cannot co-occur, so each game can be expressed in this abbreviated way.
The full game natural language game descriptions were provided to participants (for example, as shown in Extended Data Table 1). The game codes are used only for ease of presentation in this paper.
Human experiments
All human experiments were conducted under prior approval from the institutional review board at the Massachusetts Institute of Technology through the Computational Cognitive Science Lab. All participants provided informed consent.
Zero-shot outcome evaluation experiment
We recruited 238 participants from Prolific49 to judge novel games. Each participant was randomly presented with 10 games sampled from our 121 diverse game stimuli, as well as regular tic-tac-toe (won by making 3 in a row on a 3 × 3 board) to set baselines for game judgements. We collected approximately 20 judgements per game stimulus for each game reasoning query. Participants were paid at a base rate of US$12.50 per hour with an optional bonus up to US$15 per hour; the full experiment approximately took 25 minutes.
Participants were instructed to evaluate likely game outcomes. Specifically, participants produced judgements on a continuous 0 to 100 probability scale to predict the likelihood of a first player win (“If the game does not end in a draw, assuming both players play reasonably, how likely is it that the first player is going to win (not draw)?”) and a draw (“Assuming both players play reasonably, how likely is the game to end in a draw?”). Judgements were made using sliders. Both game outcome question sliders appeared on the same page.
... continue reading