Determining Sample Sizes

Overview

When deciding how many participants to include in a playtest or survey for a game, several factors should be considered. For surveys, it’s important to define both the target audience and the appropriate sample size. Ideally, participants should represent the game’s intended audience or a relevant subset of it. Depending on your research goals, the sample might include players with specific characteristics. Trying to reach everyone is unnecessary and unrealistic—but if your sampling method systematically excludes certain groups, you risk introducing coverage error.
 

While random sampling is the gold standard in scientific research, it’s often impractical in game user research, which tends to focus on smaller, more defined populations—the player base. In these cases, non-probability sampling methods are typically acceptable, including:

​

• Volunteer opt-in panels – players who willingly sign up for research.

• Self-selected surveys – links shared on forums, Discord, or social media.

• Convenience sampling – testers from easily accessible groups like coworkers, friends, or students.
   

These approaches are easier to implement but can introduce selection bias, making results less generalizable. Still, frequent playtesting, even with imperfect samples, is usually better than waiting for an ideal one. In many cases, testing regularly with internal team members or readily available participants offers valuable insights for small design decisions and helps drive iterative improvements over time.

Start With Your Goal

​Sample size depends on what you want to learn. If your goal is exploratory, such as identifying usability issues, understanding points of confusion, or testing clarity, you can often work with fewer participants. Validation goals, like measuring preferences, comparing options, or generating data for stakeholders, typically require larger samples.

High risk decisions, such as changes to monetization, major onboarding updates, or balance tuning, benefit from more data than low risk tweaks for the sake of safety.

There is no single “correct” sample size, only a size that is appropriate for the decision you are trying to make. 

Define Who You’re Sampling

Rather than trying to reach everyone, define a clear target audience or subset of players.
 

Ask:
• Who is this feature for
• What experience level matters here
• Are platform, genre familiarity, or playstyle relevant
 

For example:
• New players for FTUE testing
• Experienced genre fans for balance testing
• Mobile players versus PC players
 

Trying to sample too broadly often weakens insights. A focused sample is usually more useful than a large, unfocused one.
 

For instance, the creators of Elden Ring, a game intentionally designed for highly skilled and persistent players, are not aiming to appeal to audiences who prefer cozy or low challenge experiences. Including players who primarily enjoy cozy games and do not engage with action RPGs in a combat balance playtest would dilute the research and make the findings less actionable.

Understand Coverage and Bias

Most playtesting doesn't use true random sampling—and that’s okay!
 

Random sampling is the gold standard in academic research, but it’s often unrealistic in game development, where:

• Player populations are specific and niche

• Budgets and timelines are limited

• Recruitment happens through existing communities
   

Common non-probability sampling methods include:

• Volunteer opt-in panels - Players who sign up to participate in research.

• Self-selected surveys - Survey links shared via Discord, forums, social media, or newsletters.

• Convenience sampling - Easily accessible testers such as coworkers, friends, students, or internal team members.
   

These methods are faster and cheaper, but they can introduce selection bias, meaning results may not fully represent the player base.

The key is to be aware of this bias, not to eliminate it entirely.

Gold Standard Sampling

If you want to follow the academic gold standard, sample size is planned ahead of time rather than chosen by rule of thumb. This means clearly defining who you are studying, selecting participants as randomly as possible, and estimating how many people you need to confidently detect meaningful differences or trends. For surveys, this often means collecting a few hundred responses. For comparisons or experiments, it can mean hundreds of participants per group. This level of rigor is usually only necessary for high-stakes decisions or formal research, and is often more than most game teams need.
 

Learn more:

 

• Why You Only Need to Test with 5 Users — Nielsen Norman Group
   

• Calculating Sample Size — Qualtrics

Match Sample Size to Method

Different research methods need different numbers of participants:
 

• Usability testing / playtests:
  ~5–8 participants often uncover the majority of usability issues.

• Interviews / focus groups:
  4–8 participants per session is usually sufficient.

• Surveys:
  Sample size depends on desired confidence and variability, but trends can often emerge with 20–50 responses for directional insights.

• Analytics / telemetry:
  Larger samples are needed, but even small datasets can reveal early patterns.
   

More participants increase confidence, but diminishing returns kick in quickly for many qualitative methods.

Test Often, Not Perfectly

Waiting for the “ideal” sample can slow development more than it helps.

In practice:
 

• Frequent, small tests catch problems earlier

• Imperfect samples still reveal usability breakdowns

• Internal or convenience testers are useful for early iteration

• Later-stage validation can involve broader samples if needed
   

For most teams regular testing with available players is far better than infrequent testing with perfect samples.