A profitable backtest can disappoint in live trading because the signal was overfit, the market changed, the fills were unrealistic, or the accounting is wrong. A return gap alone does not identify which explanation is correct. Start with a matched comparison, not a new parameter search.
This guide focuses on execution and accounting. The examples are synthetic teaching fixtures, not Dnalyaw account results or evidence of profitability. An earlier version overstated how often costs explain the gap; costs are a testable hypothesis, not a universal diagnosis.
What should you compare first?
Use the same instruments, decision timestamps, starting capital, currency convention, and evaluation window. Keep three separate ledgers:
| Ledger | Inputs | What it can establish |
|---|---|---|
| Frozen decision replay | Information available at decision time; orders from a frozen policy; explicit fill/cost assumptions | What the strategy would have attempted under those assumptions |
| Actual-fill reconstruction | Recorded executions, fees, cash movements, and valuation marks | Whether your accounting matches what happened |
| Broker reconciliation | Statement records under a matched accounting convention | Which fills, charges, positions, or cash movements remain unexplained |
Replaying actual fills is useful for reconciliation. It is not an independent prediction of execution quality: those fills were not known when the decision was made. Never silently substitute them into the first ledger and call it an out-of-sample backtest.
Worked example: slippage without double counting
Suppose a synthetic strategy buys and later sells 100 shares. There are no dividends, financing charges, or cash flows; all amounts are USD.
| Item | Decision replay | Actual-fill reconstruction |
|---|---|---|
| Entry price | 100.00 | 100.02 |
| Exit price | 101.00 | 100.98 |
| Gross P&L | 100.00 | 96.00 |
| Entry + exit fees | 2.00 | 2.00 |
| Net P&L | 98.00 | 94.00 |
The four-dollar difference is a fill-price difference, not a commission error. Do not subtract another generic spread charge from P&L already calculated using actual executions. A midpoint replay, by contrast, needs an explicit model of crossing the spread, impact, latency, and non-fills.
One basis point of what? Name the denominator: order notional, portfolio equity, or traded volume. Four dollars is four basis points of the 10,000-dollar entry notional, not automatically four basis points of the account.
Delayed fees: unknown is not zero
The IBKR TWS API execution documentation describes separate execution and commission reports linked by execution ID. That page documents the older API; check callback names and correction behavior against your installed SDK and current IBKR Campus documentation.
The accounting requirement does not depend on a particular callback name:
- Record each execution using its broker identity.
- Mark the fee as pending until a fee record or documented zero-fee outcome is available.
- Join the fee to the execution; duplicate delivery must not charge twice.
- Preserve corrections and reconcile missing records after reconnect.
- Distinguish provisional P&L from reconciled P&L.
A late callback does not create an additional economic fee. It can create a temporary measurement error, or a lasting error if the fee is lost or misattributed. A synchronous backtest can still be correct if it books the correct costs and does not claim to reproduce intraday accounting availability.
Run the accounting fixture
Download the standalone fixture, save it as cost-reconciliation.py, and run the command below. It uses Python's standard library, synthetic executions, and no broker connection:
python3 cost-reconciliation.pyIt checks pending fees, fee-before-fill delivery, duplicate delivery, an explicit fee correction, missing FX, and the 98-versus-94 example. Passing it proves those small accounting cases only, not a broker adapter or investment strategy.
Currency and fee schedules: make units explicit
Store each amount with its currency. Record the FX rate direction, timestamp, and valuation policy. A fee of 7.80 HKD at an illustrative rate of 0.128205 USD per HKD is approximately 1.00 USD. Missing FX should leave a record unresolved, not default to one.
Transaction-time conversion and end-of-day translation can differ; reconciliation must compare like with like. Deposits, withdrawals, holdings, and valuation marks also matter for portfolio returns.
Do not hard-code a universal commission rate. IBKR publishes fixed and tiered stock pricing; applicable terms depend on the plan and market. Record the schedule and effective date used by the replay, and reconcile actual charges separately. Financing, borrow costs, exchange charges, and corporate actions need their own treatment where applicable.
Diagnose the residual in order
| Observation | Next check | What not to conclude |
|---|---|---|
| Actual-fill reconstruction disagrees with the statement | Missing fees, duplicates, FX, cash flows, corrections, valuation time | That the signal has decayed |
| Accounting matches, but replay fills differ | Latency, executable quotes, size, partial fills, queue assumptions | That matching the closing price proves executability |
| Signals or order counts differ | Point-in-time inputs, universe, calendar, warm-up, policy version | That costs explain missing decisions |
| Accounting and execution checks pass, but returns disappoint | Selection bias, sampling uncertainty, regime change, signal decay | That a small reconciliation error proves alpha |
Report attempted orders and the filled subset separately. Showing returns only for available fills can change the sample and make a weak strategy look stronger. Keep the decision policy frozen while diagnosing; retuning on the same disappointing period adds another search trial.
Acceptance checklist
- Identify the policy version, data snapshot, and evaluation window.
- Attach currencies to amounts and time zones to timestamps.
- Test pending, duplicate, corrected, and missing events.
- Do not count fees or spread twice.
- Keep non-fills and partial fills visible.
- Define tolerances before evaluation and tie them to strategy scale; no universal monthly basis-point threshold certifies a signal.
- Keep reconciliation, execution calibration, statistical validation, and capital allocation separate.
Read next
For the simulator side, read backtest system pitfalls. For uncertainty, read statistical traps of the Sharpe ratio. For verifying effects rather than trusting acknowledgements, see flatten verifiers.