Why Test Data Is So Hard – and How We Solved It at PrizePicks
The universal problem with test data
PrizePicks is a skill-based fantasy platform where users build and submit picks against player and game outcomes. Like many fast-growing companies, our quality control struggled to keep pace with the evolving complexities of our product features. More specifically, our automated end-to-end (E2E) tests needed data, and that data needed to handle a variety of very specific states.
To give you a sense of the type of test and data relationships we deal with, here’s just a few examples. A test that validates cashout eligibility which needs a user with an open entry that meets the cashout threshold. A test for responsible gaming enforcement which needs a user who has already reached their deposit limit. A test for social features which needs a user with followers and followees in the right configuration. All these states have to exist in the database before these tests run.
Most teams solve this with one of three approaches. None of them holds up at scale.
Static shared environments. A dedicated environment is seeded with a known set of test accounts. Teams maintain a spreadsheet of which account to use for which scenario. Tests fight over the same accounts, corrupt each other's state, and fail for reasons unrelated to the code being tested.
Database snapshots. A known-good database state is restored before each test run. This works for small teams but becomes expensive as the test suite grows: a snapshot restore can take tens of minutes, and the snapshot drifts from the real application schema as migrations accumulate.
Nightly batch seeding. A scheduled job recreates or resets test data every night. Fresh in the morning, degraded by afternoon. One job failure and the whole test suite starts the day broken.
All three approaches share the assumption that test data is infrastructure to be maintained separately from the tests that use it. That assumption is the problem.
We challenged those assumptions and built a self-seeding API. It fully provisions a valid, unique test user on demand from a single scenario identifier. It unblocks automated coverage for product states including specific eligibility tiers, limit thresholds, and reward promotion combinations. An approach that things like static accounts, snapshots, and nightly seed jobs could never reliably deliver.
Having this unblocked coverage within our flagship app now spans 127 of our 226 automated E2E tests and 225 distinct test cases, ensuring our app behaves as intended across experiences like our checkout, entries, payments, promotions, and social feed.
- Tests provision their own data instead of sharing static accounts, DB snapshots, or nightly seed jobs -- so they can't corrupt each other or fail from someone else's leftover state.
- Structural (shared) data seeds once and self-heals; behavioral (user-specific) data is created fresh on every call, so parallel runs never collide.
- Every seeded record is tracked in a lineage table, which is what makes idempotent re-seeding, ID-stable cross-referencing, and clean teardown possible.
A different mental model
Tests should own their own data. The union between the exact data state and the test requirements should be provisioned and contained within the test run itself. This approach reduces the overhead of managing additional setup and clean up steps.
When a test creates its own data, it knows exactly what state it is in. It cannot be corrupted by another test that ran before it. It can run in parallel with other tests using the same scenario. And when the test is done, the data it created can be abandoned without affecting anything else.
This is the principle behind the system we built at PrizePicks. We call it Self-Seeding.
How Self-Seeding works
The mechanism is a single API endpoint. Before a test flow starts, it calls the endpoint with a scenario identifier -- a human-readable string like user-with-cashout-eligible-entry or user-at-deposit-limit. The API provisions a complete test user in that exact state and returns their credentials. The test logs in and runs.
The user is real. It exists in the database with real entries, real picks, real responsible gaming settings, real follow relationships – provisioned on demand, with no setup script or fixture file for the test author to maintain.
Every call produces a distinct user with a unique email address. Two test runs using the same scenario identifier produce two entirely separate accounts that never interfere with each other. Parallel execution is safe by default, with no coordination required.
Scenarios as a versioned catalog
Every scenario is defined in a CSV file checked into the main application repo. A row describes the state the user should be in. The identifier is the stable key that test flows reference.
Adding a new scenario does not require changes to the API itself. A new row in the CSV is sufficient. The scenario is reviewed in the same pull request as the feature it supports and is available immediately after merge. Scenario definitions are versioned, diffable, and owned by the team that builds the feature, rather than a separate data-management team.
The identifiers also serve as documentation. user-with-open-entry-projected-loss tells a QA engineer exactly what state they are getting without consulting a spreadsheet or a Notion page.
Two tiers of data
In most applications, not all data has the same lifecycle. Some data defines the world users operate in: product catalogs, sports leagues, account tiers, feature configurations. It changes slowly and is shared across many users. Other data captures what individual users have done: transactions, sessions, settings, activity history. It is created constantly and belongs to specific users.
A seeder that treats these the same way creates unnecessary work. Recreating the entire "world" on every call is slow. Relying on world data that might be corrupted or missing makes tests brittle in a different way.
We split the data into two tiers.
Structural data is seeded once at the start of a test run. It is idempotent: if the records already exist, they are reused. If the test environment is wiped and rebuilt, the first seed call restores it automatically. This layer is self-healing and never becomes stale.
Behavioral data is always created fresh. The user account, their history, their settings, their connections -- all provisioned on every call, all unique to that request, all abandoned when the test ends.
The split keeps individual seed calls fast and removes the need to manually maintain any baseline data. The system recovers from a missing or corrupted structural record on its own.
Working with your domain rules, not around them
Any application complex enough to need sophisticated test data has domain validations -- rules that prevent the system from reaching invalid states. Order totals must match line items. Inventory cannot go negative. A transaction cannot reference a closed account.
When a seeder bypasses these rules -- writing records directly to the database, disabling callbacks, using factory shortcuts that skip validation -- it creates test scenarios that real users can never actually reach. The tests pass, but they are not testing what users experience. Over time the seeded states drift further from what production allows, and the test suite becomes less reliable at catching real bugs.
The better approach is to build your seeder to follow the same paths that production data follows. This means the seeder has to understand your domain's rules and create data in the correct sequence. It is more work to build, but every resulting scenario is a state that a real user can reach through normal product interactions.
In our case, user entries are created against live market data that can only move in one direction. The system enforces this to protect users from locking in stale information. That rule meant we could not simply create entries against pre-existing data. We built a staged approach: create the prerequisites in a valid intermediate state, perform the action that respects the rule, then apply the final resolved state.
This sequence – set up, act, resolve – maps cleanly onto any domain with ordering constraints. It is more steps than a direct database insert. But a test scenario seeded this way is faithful to what users actually do, which is the whole point.
It's also what makes it possible to seed states like early-payout eligibility thresholds or withdrawal limits at all – states that depend on live, one-directional market data and could never have been captured in a static account or a snapshot.
Respecting transaction boundaries
Seeding a user in a social scenario requires more than database records. Follow relationships need to sync to an external activity feed service. Promotion assets need to be attached via object storage. These are side effects that happen outside the database.
Running external calls inside a database transaction creates two problems. The transaction holds database locks for the full duration of the external call, increasing contention under parallel seeding. And if an external call fails, a transaction rollback leaves the external service in an inconsistent state: it received a follow that the database then cancelled.
The solution is a clean boundary. All external calls are deferred. The database transaction commits first. Follow syncs, storage attachments, and feed posts happen after the transaction is stable. If an external call fails with a transient error, the test user still exists in a usable state.
The general rule: side effects that cannot be rolled back should not run inside a transaction. Test data seeders that touch external systems need to respect this boundary or they introduce failure modes that are hard to debug and harder to reproduce.
Lineage tracking
Every record the seeder creates is registered in a lineage table that records the source identifier, the model type, and the resulting database ID.
Three things become possible.
Idempotency: Shared data seeding checks the lineage table before creating. If a record already exists from a previous call, it is reused. Duplicate seeding calls are safe.
Cross-reference resolution: When an entry needs to reference a specific projection by name rather than by ID, the seeder looks up the ID through the lineage table. Database IDs are environment-specific and change between resets. Source identifiers are stable.
Cleanup: Removing test data means deleting the lineage rows and cascading. There is no ambiguity about which records came from the seeding system versus other sources.
Without lineage tracking, cleanup has no reliable way to tell which records came from a seeded test versus real activity -- a naive delete either leaves orphaned test data behind or risks deleting records it shouldn't touch.
Integrating with a CI pipeline
In our E2E framework — built on Maestro, running on Maestro Cloud — the self-seeding API integrates at a single point in the CI pipeline: the test flow itself. Maestro's pre-flow hook, onFlowStart, is what makes this clean: it lets a flow call the seeding API and get its user back before any of the flow's real steps run, so the test itself never has to know how its data was created.
Each test flow calls a small helper script before it starts. The script authenticates with a short-lived JWT and posts the scenario identifier to the API in the same call, receives the user credentials back, and passes the email to the login step. The test runs against a user it just created.
For the test author, writing a flow that depends on a specific product state is a two-step process: choose a scenario identifier and pass it to the helper. Everything else -- authentication included -- is handled by the seeding infrastructure.
That adoption spread across teams rather than through a single migration effort: checkout, entries, social feed, promotions, and payments each moved their own flows to self-seeding separately, on their own timelines, over the following ten weeks.
What changed
Before self-seeding, test reliability was coupled to environment state. A test passing on Monday could fail on Thursday because another test had consumed the data it depended on. Environment resets were a recurring operational burden.
After self-seeding, tests are hermetic. Each run gets its own data. Parallel execution is safe. In the roughly two months since this shipped in our flagship app, the environment has not needed a full reset.
Entries went first — they were already deep into building out automation coverage for their features and felt the lack of on-demand data most acutely. Their early scenarios became the template the other teams borrowed from.
Getting the rest of the org to follow was slower than the technology itself. Two things stalled teams: scenarios they needed weren't in the catalog yet, and without someone on a team explicitly owning the migration, it never became anyone's priority. What broke the stall: in team-level demos, when we asked teams what was actually blocking their E2E coverage, the answer was almost always the same — not a lack of will, but a lack of consistent, on-demand data to build and test against. Once they saw the self-seeding API solve that problem live, and realized they could define their own scenarios without waiting on anyone else, teams started opting in on their own.
None of this happens without leadership backing it. Engineering leadership treated this as work worth protecting time for, not something squeezed in between other priorities, and once it started proving out team by team, backed pushing adoption the rest of the way.
Full adoption isn't finished. A few teams haven't migrated yet — some because their flows depend on third-party services the seeder doesn't cover, others because it's competing with other priorities on their roadmap. What's changed is that the remaining gap is now a backlog item, not a question of whether the approach works.
New scenarios ship in the same pull request as the feature they test. The scenario catalog has become shared vocabulary: when someone references user-at-deposit-limit, everyone on the team understands the exact product state it describes.
The nightly seeding job still exists for a small set of shared baseline data. But it is no longer the load-bearing pillar tests depend on. The tests carry their own weight.
Test data is not a QA logistics problem. It is an engineering problem, and like most engineering problems it rewards a clear mental model over a maintenance-heavy workaround. In our flagship app, that mental model scaled our test suite by two orders of magnitude, moving from zero to triple-digit test coverage, all without a dedicated data-management team to maintain them.
If your team is running a nightly seeding job and asking why tests keep failing Tuesday morning: ours don't anymore, because our tests stopped depending on it two months ago.
And if you are a free agent looking to get drafted to an Engineering team that solves real world problems like this in-house, explore our open positions below.
Rajesh Mannuru is a Staff Engineer on the Platform & DevEx Team at PrizePicks.