Backtesting: how it works and the four ways to fool yourself

Testing an idea against past data looks like the most objective way to know whether it works. It's also where fooling yourself is easiest without any intention of doing so, because three of the four ways to get it wrong are invisible from the inside: the result comes out well and nothing warns you.

How to test an idea against past data

A backtest means taking a rule, applying it to historical data, and counting what would have happened. It's the only way to get evidence about an idea without risking money, which is why it deserves doing properly.

The four steps

One: write the rule so it admits no interpretation. "Buy when it looks strong" can't be tested. "Buy when the price closes above its 50-session average" can. It's the same requirement that defined a method last session, and here it becomes mandatory: a computer doesn't interpret.

Two: define the universe and the period. Which assets, between which dates. Both decisions matter more than they look, and the first is where one of this session's four problems lives.

Three: subtract the costs. Spread, commission, and currency, using Chapter 3's figures. A backtest without costs isn't optimistic: it's simply something else.

Four: report the result with its three companions. Number of trades, expected value, and maximum drawdown. Without all three, the result can't be judged.

What a backtest can say and what it can't

It can: rule things out. If an idea wouldn't have worked across ten years of data, that's solid, cheap information. Ruling out is what it's best for.

It can't: confirm. That something would have worked doesn't mean it will, and the four sections that follow explain why the "would have worked" is more fragile than it looks.

With that asymmetry in mind, the correct attitude toward your own backtest coming out well is suspicion, not enthusiasm.

What look-ahead bias is

Look-ahead bias means using, to decide on a given date, information that wasn't known on that date. It's the first of the four and the easiest to commit without noticing.

The example that makes it obvious

Someone tests a rule buying companies whose annual profit exceeded some threshold. They take a sheet with each year's profits and each year's prices, and cross the two.

The problem: 2024's profit wasn't published in 2024. It was published sometime in 2025, months after the year ended. Using the 2024 figure to decide during 2024 means the rule is buying with information from the future.

The backtest comes out spectacular. And it's impossible to replicate, because nobody could know in March 2024 what would be published in March 2025.

The less obvious forms

Closing prices used to decide the same day. If your rule says "I buy when the price closes above X," the close is known once the market has shut. Buying "at that close" isn't possible: you buy at the next open, at a different price.

Revised data. Many economic figures get published and then corrected. Using the corrected figure to decide on the original date is using information that arrived later.

Index composition. Testing a strategy on the companies in today's S&P 500, applied to ten years ago, uses knowledge of which companies were going to enter. It's also the doorway to this session's third problem.

The check

One question per data point the rule uses: was this information publicly available on the date my rule uses it?

If the answer is "no" or "I'm not sure," the result doesn't count. And the correct answer to that question is what names the next section.

What point-in-time means and why it changes everything

A point-in-time data point is one that knows when it became known. It stores not just the value, but the date that value became public and every later revision.

The difference, with a case

Picture a company's fourth-quarter profit.

An ordinary database stores: Q4 2025 → 120 million. Full stop. It's the correct value, updated with the latest available revision.

A point-in-time database stores something longer: on February 15, 2026, 118 million was published; on April 30, 2026, it was revised to 120 million. It knows what was known at each moment.

With the first, any backtest using profits carries look-ahead bias by construction, and there's no way to detect it from the inside. With the second, you can reconstruct exactly what information was available on the day the rule decided.

Why this is so expensive and so rare

Because it requires having stored the state of the data at every moment, over years, and not having overwritten it as each correction arrived. It's a design decision that has to be made at the start: if it wasn't, it can't be reconstructed afterwards.

That's why point-in-time financial databases cost serious money, and why most backtests circulating free online don't use them.

How this applies outside a backtest

You don't need to write any code for this concept to be useful. It's exactly the principle of last session's logbook: recording what you thought at the time, without editing it later with what you know now.

A logbook where what was written before is untouchable is a point-in-time database of your own decisions. And it's the only record allowing a piece of reasoning to be judged without the outcome contaminating it.

It's also, for the same reason, why Volatly publishes each event's scenario before it happens and seals it afterwards. An archive built after the fact can look excellent and proves nothing, because it doesn't distinguish between having got it right and having known how to write it up later.

What survivorship bias is

Survivorship bias means studying only what still exists, ignoring what disappeared along the way. It's the third way, and it's the one that inflates results most with nothing to signal it.

The case you've already seen

Go back to the course's first session. The SPIVA reports from S&P Dow Jones Indices deliberately include funds that merged or liquidated during the period studied.

That isn't a methodological detail: it's what makes the figure mean anything. Funds doing badly tend to close or get absorbed, and a study looking only at survivors would erase precisely the worst ones.

With that correction applied, the result was that 98% of euro-denominated global equity funds failed to beat their index over ten years. Without the correction, that percentage would have been markedly more flattering, and false.

How it shows up in a homemade backtest

Testing against today's index. The S&P 500 of 2016 didn't hold the same companies as today's. The ones that went bankrupt or got bought left. Testing a strategy on the current list, applied to ten years ago, automatically eliminates every disaster.

It's exactly the case from the index chapter: Hologic left the S&P 500 in April 2026 on being acquired. Any test using the list from after that date, applied to before it, wouldn't include it.

Testing on companies still listed. The ones that delisted, went bankrupt, or got absorbed simply aren't in the database. And they are, by definition, the ones that went worst.

And the version that isn't a backtest

The same mechanism works with people, and it's last session's social feeds.

You see the results of whoever is still posting. Anyone who lost enough to quit posts nothing, and therefore doesn't appear in your sample. The impression you take away is of a success rate that doesn't exist — not because anyone is lying, but because the failures remove themselves from the count.

The course's first session gave the figure quantifying this: in the Taiwan study, more than 75% of those day trading quit within two years. That majority isn't on any forum explaining how it went.

How many trades a backtest needs to mean anything

The fourth way to fool yourself is the previous chapter's, applied here: testing on few trades and treating the result as a conclusion.

The reminder, with numbers

A method that got seven out of ten right has a 95% confidence interval running from 39.7% to 89.2%. With a hundred trades and the same 70%, the interval narrows to 60.4% – 78.1%.

A backtest with twenty trades says nothing, however spectacular the result. And reaching twenty without noticing is very easy: a restrictive rule across five years can generate exactly that many.

The problem stacking on the previous one

Here's the part making this the fourth way rather than a repetition.

The more conditions a rule has, the fewer trades it generates and the more it fits the ones it does. Both get worse simultaneously: the sample shrinks precisely as overfitting rises.

It's what we saw in the chart chapter: a strategy with very specific parameters and few trades is describing the past rather than explaining it. And it isn't an amateur problem: Park and Irwin flagged data snooping and ex post rule selection precisely as the reason the academic evidence favourable to technical analysis can't be treated as conclusive.

The three checks before believing a backtest

How many trades did it generate? Below fifty, the interval is so wide the result is consistent with almost anything.

How many configurations did I try before settling on this one? If it was twenty, finding one that works was guaranteed by chance. Almost nobody records that number, and it's the one that informs most.

Does it hold up in a period I didn't use to build it? It's the most useful check of all: set aside some years, build the rule without looking at them, and test there at the end. If it works in the period you never saw, something might be there.

What to remember

• A backtest serves to rule out, not to confirm: that something would have worked doesn't mean it will.
• Look-ahead bias means using information on a date that wasn't known on that date, and its most common form is not noticing when each figure was published.
• A point-in-time data point knows when it became known, and it's the same principle as the logbook: what was written before doesn't get touched.
• Survivorship bias erases the worst from the count, and it works identically with funds, delisted companies, and people who stopped posting.

Milestone reached

That closes Chapter 8. Your one-page plan is written, the third and last of the course's artifacts, and with it all three are complete: what each pot of money is for and when you need it, how much you risk per trade, and how you decide.

You also know why your own head sabotages everything above, what the studies say about overtrading, and the four ways to fool yourself testing an idea against the past.

Chapter 9 is the last and raises the level: how price forms underneath, what options are in plain language, and what the evidence says about the limits of machine learning applied to markets.

Published August 3, 2026. Last reviewed: August 3, 2026.

Related: what each indicator measures · drawdowns, streaks, and sample size