NLP · Python · 2026

Superforecasting at the Person Level

Do the words forecasters use predict whether they're right? Testing Tetlock's linguistic markers across 11,130 Metaculus comments.

View on GitHub ↗

The question

Superforecasting lore says good forecasters hedge, update their beliefs, and reason causally. If that's true, those linguistic fingerprints should show up in who is actually accurate. Do they?

What I did

A two-domain NLP pipeline (economy-business and geopolitics) across 11,130 modeled comments. I engineered 17 features capturing hedging, epistemic markers, causal reasoning, and belief-updating, then tested them against comment-level accuracy with permutation and DeLong tests.

What I found

• Linguistic features don't predict accuracy in either domain, permutation-confirmed.

• Track record is the only consistent signal, and it transfers across domains (r = 0.344, p < 0.001, 164 shared authors).

• Being liked is not being right: upvotes and accuracy are near-orthogonal (r = 0.051).

• Honest caveat: 65–70% of late correct comments arrived within 7 days of resolution, a partial timing confound.