Nothing survived the backtest
I built a backtesting engine, gave an AI model a budget, and asked it to find a trading strategy that beats simply holding bitcoin.
It wrote 200. Every one of them compiled, and none of them worked. The interesting part is how they failed and how easy it would have been to think some of them had succeeded.
The same strategy, +89% or −94%
A backtest replays history and asks what a trading rule would have done. It's the first tool anyone reaches for, and it's very good at telling you what you want to hear.
Take a boring strategy: a moving-average crossover on bitcoin. Buy when the average price of the last hour rises above the average of the last four hours, and sell when it falls back below. Run it on every minute of 2024 the way a quick notebook would, and it makes +139.6%. That beats simply buying and holding, which made +120.4% in a very good year for bitcoin.
That first number cheats in a way that's easy to miss: it trades on each minute's closing price during that same minute, so it acts on information it couldn't have had yet. Remove that one-minute peek and it drops to +89.2%. Still very respectable.
Then add the things that happen when you actually trade:
| What changes | Return over 2024 |
|---|---|
| Notebook, one-minute look-ahead | +139.6% |
| Engine, trades are free and instant | +89.2% |
| + 0.1% exchange fee per trade | −90.0% |
| + 0.02% slippage | −94.5% |
| + 100 ms delay before orders arrive | −94.5% |
| + can't trade more than 10% of each minute's volume | −94.4% |
| Just buy and hold | +120.4% |
The strategy traded 2,956 times. A 0.1% fee sounds like nothing, but paid nearly three thousand times it came to over $10,000 on a $10,000 account. The strategy didn't get any worse between the second row and the third. We just stopped pretending trading is free.
This is the entire problem with backtests in one table. Every row is a real piece of software producing a real number, and most people stop at the second row.
An engine that's hard to fool
So the first thing I built was a backtester that doesn't let you stop at row two. It's called tape, and it's written in Rust. A few rules do most of the work:
- A strategy only sees the past. It receives each minute of data after that minute has closed, and it can't ask for anything else. Looking ahead isn't discouraged, it's impossible.
- Trading costs something. Fees, slippage, the delay between deciding and the order arriving, and a cap on how much of the market you can take are all in the default settings, not an optional extra.
- Money is exact. Prices and balances are whole numbers of tiny units rather than floating point, so cash always adds up to the cent.
- Same inputs, same answer. Every run produces a fingerprint. Run it again and you get the same one, on any machine. If a result can't be reproduced, it doesn't count.
Then I let the AI loose
With an engine I trusted, I set up a small research desk made of AI models, each with one job:
- A researcher proposes an idea: a hypothesis about why it might work and precise trading rules.
- A coder turns those rules into a real program.
- The engine backtests it on 2023 and 2024, with realistic costs.
- A critic reads the idea, the code and the results, and decides whether the evidence is convincing.
- Only strategies the critic accepts get one final test on data none of them had seen.
Two details matter more than they look.
The code the AI writes runs in a sandbox. It has no clock, no network, no files and a fixed compute budget, so a strategy can't cheat by looking something up, and a broken one can't hang the run. One strategy did blow through its compute budget a month into the backtest, and it was stopped with a clear error.
The final test period is June to August 2026, which is after the model's training cutoff. The model can't have memorised what bitcoin did in those months, because it had never seen them. Without that, any AI trading result is suspect: the model may simply remember what happened.
I used gpt-6-luna, one of OpenAI's cheaper models, on its highest reasoning setting. Two hundred attempts cost $1.47 in total.
What happened
| Ideas proposed | 200 |
| Compiled | 200 (190 on the first try) |
| Ran to the end | 199 |
| Accepted by the critic | 0 |
Code, in other words, was not the problem. The model wrote a working program on the first try 95% of the time. The strategies were the problem.
For comparison, buying bitcoin at the start of 2023 and holding it to the end of 2024 made +463%. Against that:
- The median strategy made +2% over two years.
- 86 of the 199 lost money. The median strategy traded 149 times and paid about $1,400 in fees on a $10,000 account.
- The few that roughly matched buy-and-hold did so by being buy-and-hold. The five best returns all made a single trade: buy at the start, then a filter that was supposed to step aside in risky conditions and never fired. The critic caught it. Its verdict on one of them: "effectively a buy-and-hold result".
The one that almost looked good
The strategy with the best risk-adjusted score was called Multiweek Cost-Basis Reclaim. It had a Sharpe ratio of 2.14 over the two years, slightly better than buy-and-hold's 2.02, with a much smaller worst drop. If you ranked all 200 by that number and published the winner, this is the one you'd publish.
It made four trades in two years.
The catch with trying 200 ideas is that some of them will look good by chance. One way to check is to ask what the best score would be if every strategy were pure noise. For the 188 strategies that actually traded, that luck threshold works out to a Sharpe of about 4.5. Our best was 2.14, less than half of what luck alone should produce. The probability that it reflects a real edge, using the standard correction for this (the "deflated Sharpe ratio"), is effectively zero.
The critic got there by a simpler route. Four trades isn't evidence of anything.
The trap I set for myself
The critic accepted nothing, so officially nothing reached the final test. I ran all of them on the unseen months anyway, to see what would have happened if I'd skipped the critic.
61 of them beat buy-and-hold (on Sharpe ratio, out of the 178 that traded in those months).
That sounds like a result, but it isn't one. Three months is a very short test, and the scores were all over the place: one strategy that was mediocre over two years scored the best of all over those three months. The in-sample and out-of-sample scores were correlated, but mostly for a dull reason: strategies that trade a lot pay a lot of fees in every period. What carried over was cost, not skill.
If I'd wanted a headline, I had one: "AI finds 61 strategies that beat the market". It would have been easy to write and it would have been false. The only thing protecting me from it was a set of rules I wrote down before I saw the results.
Try it yourself
The engine runs in your browser at tape.victorarowo.com. It's the same Rust code compiled to WebAssembly, and it produces the same fingerprints. You can pick a strategy, slide the fees and delay up from zero, and watch a profitable backtest turn into a losing one. Or write your own strategy in JavaScript and see whether it survives realistic costs.
I'd genuinely like to see one that does.