The question nobody publishes an answer to

Matchup splits are published everywhere and validated almost nowhere. The implicit claim behind every one of them is that the split describes something durable about how two players interact. That claim is testable, and the test is not complicated.

Take every pairing with enough meetings. Sort their games, deal them alternately into two piles, and compute the scorer's rate in each pile independently. If a matchup split reflects something real, a pairing's first-pile rate should predict its second-pile rate. If it is mostly noise, the two will be unrelated.

The result

Correlation rises as the halves get thicker, which is exactly what should happen if the underlying quantity is real and being measured with error. That internal consistency is a reason to trust the test.

A split-half correlation understates the reliability of the whole sample, because each half is only half as long. The Spearman-Brown correction adjusts for that: 2r divided by 1 plus r. At r = 0.442 that gives roughly 0.61 for the full pairing.

What this does and does not license

It licenses treating a well-sampled matchup split as weak evidence worth combining with other things: film, scheme, personnel, health. It does not license treating it as a finding on its own, and it certainly does not license extrapolating a rate forward as though it were a projection.

It also puts a hard number on something usually argued by assertion. People who say matchup data is meaningless are wrong: 0.61 is not zero. People who present a matchup split as decisive are also wrong, by about the same distance.

Limits of this test, stated plainly

• It runs on 172 pairings, the ones with enough meetings to split. Those are the best-sampled matchups in the data, so 0.61 is an optimistic ceiling, not a typical case.
• Alternating games controls for order but not for context. Injuries, scheme changes, and midseason trades sit inside both halves.
• Reliability is not validity. A stable measurement can still be measuring the wrong thing, which is exactly what happens when this data is used to rank defenders against each other.
• One season only. A test across multiple seasons would be stronger and is not possible with the data currently loaded.

The third point deserves emphasis, because it is the one people skip. Showing that a number repeats does not show that it means what you think. A defender's points-allowed figure repeats fairly well and still mostly reflects where he is assigned to stand.