Agentic Market Simulator Part 2: Universe, Information and Leakage

This article covers one component of a larger system and makes no claim to replicate a real exchange. Realism will arrive cumulatively across the series: each part is held to its own scope only.

Nothing in this series should be understood as forecasts for prices or investment advice.

The Universe

In the context of this series, we will use mid and large cap US companies over 2019 to 2025. More precisely, we will look at every company which filed quarterly financial statements with the SEC (form 10-Q) and had a market capitalization of at least 10 billion USD as of January 1st, 2019. Using the beginning of the period to filter on market capitalization avoids creating a subtle forward-looking bias.

What Agents Observe

In Episode 1, the asset had a latent fundamental value that we generated ourselves following a geometric random walk, and each agent saw a private, noisy version of it. In reality, there is no observable fundamental value. This is where we leverage the added intelligence brought by the language models. Each agent will observe 4 types of data, and form its own beliefs accordingly:

It is important to note that the price history is not simply historical price data played back, as doing this would defeat the purpose of the simulator we are building. At the start, prices will be real, and every price after that will be the output of the auction system defined in Episode 1. It will therefore be shaped by whatever the agent population decides to do on each simulated day. This is a key point where the tool we are building in this series diverges from a traditional backtest.

What an agent can see as of one decision date. The price line is real up to the start of the window and auction-generated after it; filings and news become visible on the date they were released.

This information is represented with Python objects during the simulation.

@dataclass
class FinancialReport:
    fiscal_period: str        # e.g. "Q2 2021"
    filed_on: str             # the real SEC filing date, not the quarter end
    metrics: dict[str, float] # extracted figures, not the filing text itself


@dataclass
class AgentObservation:
    """Everything an agent may see on one decision date.

    Nothing else is available. Prices before the window start are the real
    historical record; prices after it are the auction's own output, shaped
    by every agent's past decisions, including this one's own.
    """
    as_of: str                          # the current decision date
    track_record: object                # agent's own track record
    price_history: list[float]          # daily closes since window start
    financials: list[FinancialReport]   # every report filed by as_of
    news: list[str]                     # exogenous events up to as_of

We enhance agents with the capability of computing statistics from the raw daily price time series. They will be able to compute these metrics by making a tool call, instead of performing arithmetic themselves. This gives them access to a small toolbox of quantitative functions, callable on demand, that turns price_history into the kind of derived figures an analyst would reason from in practice: historical return over a window, realised volatility, and other statistics. This is an effective way to reduce the reasoning load the agent has to perform itself, and allows for the use of smaller, faster language models.

How an agent forms decisions. Price, news, financial statements and track record flow into the agent one-way; the quant toolbox is called on demand and returns a result. The agent’s output is a belief over every asset in the universe.

The exact mechanism by which the agent produces the output belief vector and how this translates into orders will be the topic of the next episode. In the remainder of this article, we discuss how using language models can introduce new kinds of biases.

Biases & Mitigations

Using language models to represent the agents introduces a new class of bias that a traditional backtest does not have to deal with. A backtest runs a fixed, known rule against history. An agent here is a model trained on a large, mostly unknown share of the public internet, which very likely includes commentary, filings, and news coverage from across our entire 2019 to 2025 window. This raises an important question: when an agent makes good allocation decisions, is that because it reasoned well, or because it already knew what would happen next?

While this problem is central, it is often sparsely addressed or neglected. The purpose of this section is to illustrate through real experiments how this bias may arise and propose some mitigation strategies.

A Simple Example of Bias

As mentioned above, LLMs are trained on large corpora of data, mostly unauditable by the users. It is likely that this training data contains financial statements, financial news describing market trends, or other related information. When the model is then prompted to use historical information to decide which asset to buy and which to sell, we must ask: is it reasoning purely from the provided data or is it relying on prior knowledge? Answering this directly is not straightforward. However, we can reasonably assess whether or not prior knowledge exists inside the model.

Across a universe of 20 real companies, we gave a model real reported financial figures including revenue and net income, financial ratios, and the reporting fiscal period, with no company name and no sector, and asked it to identify the company. Each model was prompted 5 times for each company at a high temperature, with the model not allowed to perform any web search so that we are sure the answer comes from its own knowledge. We ran this across seven models, spanning both frontier and smaller-scale models: Claude Sonnet 5, Claude Opus 5, Claude Haiku 4.5, Gemma 3 27B, Llama 3.1 8B, Qwen3 32B, and Amazon Nova Micro.

If a model succeeds, it means that it is able to identify a company from its reported financial numbers only. With that capacity, it would be possible for the model to make its buy and sell choices, not from the numbers directly, but from inferring the underlying company first, then picking according to what it knows the real price trends were for that company.

Share of 100 trials (20 companies × 5 repeats) where the model correctly named the company, sorted highest to lowest.

Every model tested identifies at least some companies from figures alone, both frontier and smaller-scale. We observed that some companies are consistently correctly identified (e.g., Apple and Nvidia) and are not lucky guesses by the models. Other, equally known companies (e.g., Microsoft and Tesla) were never identified by any of the models. This suggests that the models are picking up on specific numerical patterns in the financials which identify companies, not a simple rule such as higher revenue, for example.

It is important to note that a backtest does not need every position to be contaminated to be biased. One ticker identified and positioned correctly for reasons that have nothing to do with the agent’s stated reasoning is enough to distort an aggregate return figure, especially over a long period of time where returns compound. A single Apple or Nvidia position moved by hindsight can be the difference between amazing performance metrics and poor ones.

Other Leakage Channels

Financial ratios are tested in detail here for illustration, but are not the only potential source of bias. Raw price levels are also identifying: a specific price time series with dates can be traced back to the underlying asset.

News events produce a different kind of potential bias. We give every agent the same fixed list of real events, Fed decisions, policy changes, elections, so that the population reacts to a shared shock the way a real market would. But a real, dated macroeconomic event is traceable to what actually happened next in the market. It is important to make the following distinction:

• A model leveraging historical patterns between past Fed decisions and market reactions is not a forward-looking bias.
• A model remembering the exact market reaction to a specific current (i.e., at time t in the simulation) Fed decision is a forward-looking bias.

Mitigation Strategies

News events do not fit easily into that list. Some aspects of it can be treated the same way as the other channels: dates can be replaced with a step count instead of a calendar date, and events can be written in flat, factual language rather than the narrated, market-reaction style a news article would use.

However, a global pandemic like COVID, or a specific central bank tightening cycle, has no substitute. Prior knowledge contained in the model is hard to adjust for. This series will not fully address this issue. Later episodes will instead run sensitivity experiments which turn this input on and off and analyze how the system reacts.

We measured the effect of the mitigation strategy using the same experiment as above. We show 3 variants: the raw financials, the cross-sectionally normalized, and the cross-sectionally normalized without mentioning that the numbers were normalized.

Share of 100 trials (20 companies × 5 repeats) correctly naming the company. Three frontier models, in descending order of the mitigation’s effect.

For every frontier model tested, each added condition reduces identification further. None of the three reaches exactly zero, landing between 1.1% and 2.1%.

What About Smaller Models?

The three models above are all frontier-scale, used here for illustration. Since our framework will rely on smaller models, we repeated the same three-condition comparison on four lighter models: Gemma 3 27B, Llama 3.1 8B, Qwen3 32B, and Amazon Nova Micro.

Same 100 trials per model. The highlighted cell is the one result across all seven models, at any condition, that reached exactly zero.

Gemma 3 27B is the strongest in identification when raw numbers are provided, but also the only one reaching 0% when we apply our mitigation strategy. We note that Qwen3 32B and Nova Micro barely move between the normalized and unlabeled conditions. However, we saw in our experiment that their identification was not consistent across different runs, indicating uncertainty in the guesses.

While our experiment is not a guarantee that no bias remains, it provides a methodology for identifying and assessing potential for bias. Furthermore, the analysis of different models under the lens of bias is a criterion that should be used to select the models used in the simulations.

The next episode covers the agentic decision and its implementation: how an agent turns price, financials, news and its own track record into an actual position.

Common Questions

Why not use fictional companies instead of real ones?

Simulating realistic company financials is a problem on its own and is not addressed in this series.

Does anonymizing the name fully solve the problem?

No. It partially addresses some potential for bias. Addressing bias fully cannot be done through a single trick when dealing with language models; it requires the combination of multiple mitigation strategies that will minimize bias.

Why not just use a model with an older training cutoff?

This could be done, but it highly limits the choice and quality of language models that can be used. Researchers have trained timed language models (one model per year), making sure the training data was point-in-time. This kind of approach is promising, but comes at a high computational and storage cost.

References

• Brown, Goetzmann & Ross, “Survivorship Bias in Performance Studies”, Review of Financial Studies (1992)
• Glasserman & Lin, “Assessing Look-Ahead Bias in Stock Return Predictions Generated by GPT Sentiment Analysis” (2023).