Factor libraries grow on their own. Take momentum, change the lookback, change the universe, change the neutralization, and you have a "new factor." A library with 100 factors gets reported as 100 factors. How many independent signals are actually in there?
In The Geometry of Alpha I wrote about the square-root law: N independent signals of similar quality combine into an information ratio roughly √N times that of one. The law assumes independence. Count ten versions of the same bet as ten independent signals and √N will systematically overstate your portfolio.
This post is about computing that number and deduplicating.
The effective number of factors: one formula
For equally weighted, equal-volatility signals of similar quality, this special-case risk-equivalent count is:
where N is the number of signals and ρ̄ is the average pairwise correlation between them. Under those assumptions, the combined Sharpe ratio is roughly √N_eff times the Sharpe of a single signal. This is a useful diagnostic, not a universal portfolio formula.
| Signals | Average pairwise correlation | Effective number | Combined Sharpe / single Sharpe |
|---|---|---|---|
| 6 | 0 | 6.0 | 2.45 |
| 6 | 0.2 | 3.0 | 1.73 |
| 6 | 0.4 | 2.0 | 1.41 |
| 11 | 0.13 | 4.8 | 2.19 |
An average correlation of 0.2 sounds low, yet six signals shrink to three. Correlation hurts a portfolio far more than intuition suggests.
Running the numbers on public data
The data is the Ken French bundle shipped with linearmodels 7.0: US equities, January 1949 to March 2017, monthly, gross of trading costs. The repository keeps a frozen transformed snapshot so this article does not drift when the upstream library or source data changes. From it I built a typical "factor library":
- Four momentum factors: the standard momentum factor Mom, plus winners-minus-losers within small, mid and large caps.
- Four value factors: HML, plus high-minus-low book-to-market within small, mid and large caps.
- Three size factors: SMB, plus two differently sliced small-minus-big spreads.
Eleven factors, eleven different names. Each is scaled to the same volatility and then equally weighted:
| Metric | Value |
|---|---|
| Average pairwise correlation | 0.13 |
| Effective number | 4.8 |
| Average single-factor Sharpe | 0.46 |
| Combined Sharpe estimated with √11 | 1.51 |
| Actual equal-weight combined Sharpe | 1.00 |
These Sharpe ratios scale every factor to the same volatility using the full sample, and they are gross of trading costs. They describe the correlation structure. They are not a tradable backtest.
An average correlation of 0.13 looks well diversified, yet the effective number is only 4.8. Under this standardized full-sample setup, treating the series as eleven independent signals gives a Sharpe estimate of 1.5; the observed combination is 1.0, almost exactly 0.46 × √4.8. Here, the gap is mechanically explained by correlation among the series.
Deduplication: merge by correlation
The procedure is plain. Repeatedly merge the two groups with the highest average correlation, and stop when no two groups have an average correlation above 0.5. Each group is represented by the equal-weight average of its members.
The eleven factors end up in four groups:
- Momentum: Mom and the three size-bucket momentum spreads, average within-group correlation 0.80.
- Value: HML, mid-cap value, large-cap value.
- Small-cap value, on its own.
- Size: SMB and the two small-minus-big spreads.
| Metric | Before merging | After merging |
|---|---|---|
| Signals | 11 | 4 |
| Effective number | 4.8 | 4.1 |
| Equal-weight combined Sharpe | 1.00 | 1.02 |
Eleven signals become four and, in this frozen full sample, gross Sharpe does not fall (the small rise from 1.00 to 1.02 comes from reweighting, not new information). Under this construction, the seven additional series did not add an independent bet. That is not proof that grouping is information-lossless out of sample.
Group 3 is worth a second look. Both are called "value," but small-cap value and large-cap value correlate at just 0.25. Small-cap value correlates 0.64 with HML and 0.60 with mid-cap value. Its average correlation with the already merged value group is about 0.50, just under the 0.5 cutoff, which is why it stays alone. Loosen the cutoff slightly and this group folds back in. The same name does not make the same signal, and different names can hide the same one. Deduplicate on return series, not on labels.
Why you shouldn't weight by count
Suppose the library holds four momentum variants, one value factor and one size factor. Weight each signal equally and momentum gets two thirds of the weight:
| Approach | Effective number | Combined Sharpe |
|---|---|---|
| Equal weight on each of the 6 signals | 2.6 | 0.87 |
| Merge the 4 momentum variants first, then weight 3 equally | 4.0 | 0.90 |
Weighting by count lets the idea with the most variants run the portfolio. Merge signals from the same source first, then allocate: under the new allocation, exposure concentration falls and the effective count rises from 2.6 to 4.0. The merge has not created new independence; it has stopped variant count from setting the weights.
Is the correlation structure stable?
Split the sample in half and recompute with the same four groups:
| Period | Average pairwise correlation | Effective number | Sharpe, all 11 | Sharpe, 4 merged groups |
|---|---|---|---|---|
| 1949–1983 | 0.13 | 4.75 | 1.33 | 1.22 |
| 1983–2017 | 0.12 | 4.95 | 0.77 | 0.90 |
The measured correlation structure is similar in both halves, with an effective number close to 5 each time. What changes is the level of returns: Sharpe ratios are lower in the second half. The merged Sharpe is lower in one half and higher in the other. Because the groups were formed on the full sample, this is a descriptive reweighting comparison, not evidence that the merge preserves information out of sample.
Strip common exposures before measuring correlation
The same dataset has twelve industry portfolios. Measured directly, their average correlation is 0.65 and the effective number is 1.5: their risk is dominated by common market exposure. Remove each industry's market beta first and the average correlation is −0.006, with a risk-equivalent count of 12.8. A count slightly above the raw number is possible when average residual correlation is slightly negative; it is a diagnostic, not a claim that new assets appeared.
So before deduplicating, strip common exposures such as market, industry and style, or everything will look alike. It is the quantitative version of the "strip first, then talk about alpha" gate in The Geometry of Alpha.
What does not count as a new independent signal
These came up again and again in my own research and ended up written into my rules:
- The same risk source on a few more tickers. One logic run on several highly correlated ETFs is one leg, not several.
- A different trading clock. An afternoon strategy is not independent of a morning one just because it trades later; measure it on concurrent returns.
- Better execution. Lower slippage makes real money, but it is not a new signal and should not count toward the independent number.
- Recombining rejected weak signals. Putting rejected weak signals back together does not create a new independent signal. New independence comes from new data and new mechanisms, not from recombining leftovers.
- Merging several signals, then picking the final longs and shorts. That is not the same as adding independent portfolios; recompute the effective number on the final book.
For a more systematic treatment of portfolio construction, see the book chapters on portfolio construction and exposure management and strategy homogenization and capacity.
The same thing as repeated structure inside a model
If a trained model contains many repeated local structures, those might be mergeable in the weights without retraining. A factor library is a small-scale analogy: in this sample, eleven series grouped into four without lowering full-sample gross Sharpe.
My memory-model transfer experiment showed the other side: in a prior meant to transfer, structure shared across many relations is exactly the part that carries to new worlds. The two don't conflict. Both are the same move: count the independent part once. In a portfolio, the same bet doesn't pay you twice, so you merge it. In a prior, shared structure only needs to be stored once, so you share it.
Code
The script uses only the Python standard library: factor_dedup.py. The frozen CSV's source, transformations, date range, hash, and reuse caveats are recorded in its provenance note.
# Self-test: 3 hidden sources, 11 noisy copies; checks that clustering recovers the 3 groups
python3 factor_dedup.py --self-test
# Check this post's tables against the frozen series in the repo, independent of later linearmodels releases
python3 factor_dedup.py --article-check
# Recompute from the Ken French bundle in linearmodels 7.0 (through March 2017)
pip install 'linearmodels==7.0'
python3 factor_dedup.py --demo
# Your own data: a wide CSV, first column the date, one column of returns per signal
python3 factor_dedup.py --csv my_factors.csv --threshold 0.5
# If the CSV has a market-return column, strip its beta first
python3 factor_dedup.py --csv my_factors.csv --strip market
# Read a CSV downloaded from the Ken French site directly (first monthly block only)
python3 factor_dedup.py --french 12_Industry_Portfolios.csv
The output shows the average pairwise correlation, the effective number, the √N estimate against the actual equal-weight Sharpe, the clusters, and the same metrics after merging. Volatility scaling uses the full sample, so treat these Sharpe ratios as a description of correlation structure, not a tradable backtest.