Ask an LLM for factor ideas and you get them quickly: a definition, a reason it might work, and the pandas code to compute it. Ask again and you get twenty more. Put it in a loop with a backtester and it will happily produce a few hundred candidates overnight.
None of that is the hard part any more. The hard part is deciding which of those candidates are real, and generated candidates fail in ways hand-written ones mostly don't. The model may already know how the test period turned out. The code may not compute what the description says. And because each idea costs almost nothing, the number of things you tried grows faster than your sense of how many things you tried.
These are the checks a generated factor or strategy has to pass in my quant system before it gets any capital. They run in this order, and a candidate stops at the first one it fails.
1. Count every trial, including the ones you threw away
A backtest Sharpe ratio means something only next to the number of things you tried to get it. With hand-written factors that number stays small because each one takes a day. With a generation loop it can reach hundreds without anyone deciding to run hundreds of tests.
To see how much this matters, I simulated strategies with zero true edge: daily returns drawn from noise, five years each, and took the best annualized Sharpe among N of them. Averaged over many repetitions:
A thousand tries of noise gives you a best-of-batch Sharpe around 1.5 over five years. Plenty of real strategies would be glad to have that number. The simulation assumes independent trials; generated factors are often near-duplicates of each other, so the effective count is smaller than the raw count, but it is never one.
What I do about it:
- Every candidate the loop produces gets a row in a log, including the ones rejected before a full backtest. The count is part of the result.
- The acceptance bar rises with the count. The deflated Sharpe ratio is the standard way to do this formally; even the rough version — "what does the best of N noise strategies look like here" — catches most of the problem.
- Near-duplicates are grouped first, so the count reflects distinct ideas rather than renamed copies.
2. The model may already know how the test period ended
This one has no equivalent for hand-written factors. A model trained on text up to some date has read about what happened before that date: which sectors rallied, which companies collapsed, which anomalies were written up. Ask it for a factor and test that factor on the same years, and the "discovery" may just be the model recalling what it read.
It doesn't need tickers or dates in the prompt for this to happen. A model that knows a certain kind of stock did well in a certain period will tend to propose factors that favor those stocks.
My rule: a generated factor is judged only on data after the model's training cutoff. Results from earlier periods are kept as a sanity check — does the code run, is the sign plausible — but they never count as evidence. This shortens the usable test window a lot, and that is the honest cost of using the tool.
A related point holds for any factor built from published ideas: anomalies that appear in papers tend to weaken after they are published. An LLM's sense of "what works" comes heavily from that literature, so its suggestions lean towards ideas the market has already had time to trade away.
3. Keep the holdout out of the generation loop
The usual loop is: generate a factor, backtest it, feed the result back into the prompt ("this one had a weak Sharpe, try a variant"), repeat. That loop is a fitting procedure. Whatever data its backtests run on is training data, even though no gradient was computed.
So the holdout period is never visible to the loop, not as a score, not as a pass/fail flag, not as a summary line. It is touched once, at the end, for the candidates that survived everything else. If I then change the candidate and look at the holdout again, that holdout is used up and the next one has to come from later data.
4. Check the code, not the description
Generated factors come with a confident English description and code that may not match it. The description says "20-day momentum, skipping the most recent week"; the code skips the week at the wrong end, or ranks across the whole history instead of per day. The backtest then evaluates the code, and the description is what you remember.
Three checks catch most of this:
- Known-answer tests. Build a tiny synthetic dataset where the right factor values can be worked out by hand, and compare.
- The future-change test. Take the input data, change every value after date t, and recompute. Factor values at or before t must not move. If they do, the code reads the future, through a centered rolling window, a full-sample normalization, or a join on the wrong date. This test is mechanical and catches leaks that reading the code misses.
- Point-in-time inputs. Fundamentals must be keyed to the date they became public, not the period they describe. Generated code almost always uses the period date, because that is what most example code online does.
The general list of backtest traps — look-ahead, survivorship, costs — still applies; I keep it in the backtest pitfalls lesson. What changes with generated code is the volume: nobody reviews three hundred factor functions by eye, so the checks have to be automatic.
5. Measure what is new, not what is good
Many generated factors are existing factors with extra steps: momentum computed on a slightly different window, value with an unusual denominator. They backtest well because the factor they copy works.
So the question is not "does this factor have a good Sharpe" but "what does it add to what I already hold". In practice: regress its returns on the existing factor set and industry exposures, and look at what is left over. A candidate with a strong standalone result and nothing left over gets dropped. A modest candidate with a clean, uncorrelated residual is more interesting.
6. Net of cost, at the size you would trade
Generated strategies drift toward high turnover, because short-horizon patterns give the loop the most signal to work with. A gross Sharpe tells you little: subtract the trading cost from each period's return first, using the strategy's turnover and an honest cost per trade, and then recompute the Sharpe on what is left.
I charge each candidate the cost model the live system uses, at the position sizes it would trade, and keep the gross and net numbers side by side. A large gap between them is itself a finding: it means the result depends on the cost assumption more than on the signal.
7. Paper trading before capital
The last check runs the candidate through the same code path as live trading, with orders recorded but not sent, for long enough to compare against what the backtest said that period should have produced. The comparison follows the same method I use for live books: split the gap between research and live into parts that add up exactly, so a shortfall can be traced to signal, execution or cost rather than described as "a bit worse than expected".
What the LLM is good for here
None of this is an argument against using LLMs in research. They are good at the parts that used to be slow: turning a vague idea into a precise definition, writing the first version of the code, listing the ways a factor might be contaminated, and drafting the tests above. The research survey chapter covers where they are being used more broadly.
What they do not do is reduce the evidence a strategy needs. If anything they raise it, because the number of ideas went up and the model has already read about part of the test period. The checks above are what a generated candidate has to pass before it gets the same treatment as one I wrote by hand.