Are you winning because you’re good—or just lucky? AI has the answer

Tabletop games thrive on a delicate balance between skill and chance. Randomness can make a game thrilling—or frustratingly arbitrary. But how do we measure this effect objectively? Researchers James Goodman, Diego Perez-Liebana, and Simon Lucas from Queen Mary University of London introduce a technique to quantify randomness in games, analyzing 15 tabletop titles to determine how unpredictability impacts outcomes.

Measuring randomness in game outcomes

Games range from perfect information titles like Chess, where no chance is involved, to games like Poker, where randomness and hidden information define the experience. While randomness can level the playing field and keep games exciting, excessive chance can make player skill irrelevant.

In their study, the researchers tackled two key questions:

1. How much does built-in randomness affect game outcomes?
2. Can the sources of randomness—such as shuffled decks or randomized board setups—be isolated and analyzed separately?

They extended their research to examine additional aspects:

• How does player skill interact with randomness? Does expertise mitigate luck, or does it amplify its impact?
• Can knowledge of randomness help improve error analysis in AI-driven game experiments?
• How significant is randomness in Catan, a game often debated for its balance of skill and luck?

Skill vs. randomness: Do better players minimize luck?

A key part of the research examined whether increasing player skill leads to more deterministic outcomes. Intuitively, experienced players should be better at adapting to randomness, making the game feel less luck-driven. The researchers tested this by varying the skill levels of AI agents, measuring their performance across different levels of game randomness. The results showed a clear trend: in most games, skilled players were more effective at exploiting lucky events and mitigating bad ones. However, in certain cases, randomness still played a dominant role, regardless of skill level.

SENSEI: The AI that explores like a curious child

Breaking down randomness in specific games

The study analyzed multiple games in detail, isolating different sources of randomness:

• Seven Wonders: The biggest impact on outcomes came from the initial Wonder board assignment rather than the card decks shuffled later.
• Colt Express: The randomized assignment of characters had a much greater effect on gameplay than the shuffled train layout or round order.
• Dominion: Surprisingly, the first two shuffled hands of each player influenced outcomes more than subsequent deck reshuffles.
• Catan: The initial board setup had a far greater effect on game results than dice rolls during play, reinforcing long-held player debates.

The hidden impact of random seeds

One of the most intriguing findings from the study was how much the choice of a random seed—the number used to generate randomized elements in a game—could influence outcomes. The researchers tested games by fixing certain seeds while allowing others to vary, revealing how randomness interacts with player decisions.

For instance, in Colt Express, simply changing the seed that assigned player characters significantly altered the balance of power. Some characters had inherent advantages, and skilled players were able to exploit these differences more effectively. In Dominion, the shuffle of a player’s initial deck mattered more than later reshuffles, as early advantages compounded over time. Meanwhile, in Catan, the research confirmed a long-debated issue: the initial board setup was far more influential than the dice rolls themselves.

Researchers ran thousands of simulations with AI agents of varying skill levels to determine how they interacted with randomness. The results confirmed that stronger agents made better use of favorable randomness while mitigating the effects of bad luck.

However, an unexpected discovery emerged: some random setups created non-intuitive difficulty spikes. For example, in Poker, certain shuffles led to near-unwinnable scenarios for a player, even if they played optimally. In Seven Wonders, the AI found that some Wonder board assignments almost guaranteed victory when paired with specific neighbors. These findings suggest that not all randomness is created equal, and some design choices can lead to subtle but powerful imbalances.

Game designers can use these insights to fine-tune randomness and create a better balance between strategy and chance. For example, a board game publisher used the study’s findings to adjust the setup mechanics of an upcoming game to reduce unintended randomness.

For players, the study confirms what many have suspected: in some games, luck genuinely dominates, while in others, skill and decision-making allow players to navigate chance effectively. For designers, these findings highlight the importance of carefully tuning randomness to match their game’s intended experience.

One takeaway is clear: randomness is a tool, not a flaw—when used correctly, it creates challenge, excitement, and fairness.

Featured image credit: Kerem Gülen/Midjourney