Overview
I built a system that grades stock tipsters by what actually happened after they made a call. It collects StockTwits posts around the clock, decides which ones are real bullish or bearish calls, pulls the price history for every ticker mentioned, measures each call against the market over 1, 5 and 20 trading sessions, and ranks the people making them on a web leaderboard. It rebuilds a Twitter stock picker I wrote in 2017, keeping the idea and fixing the scoring.
The Challenge
Finance feeds are full of confident calls, and nobody keeps score. Grading a tipster fairly takes their whole history, the price when they posted, a market benchmark so a rising tide doesn't count as skill, and protection against three lucky calls looking like genius. About half of posts carry no bullish or bearish tag, so the text has to be read. My 2017 rule counted a buy as right if the price rose at any point in five days, which gives almost everyone 80%+. And the data had to come from free public endpoints, since StockTwits' firehose is partner-only.
What I Built
- Collector — polls StockTwits' public trending stream, trending symbols and a watchlist at about 1,400 requests an hour, with cursor tracking, gap back-filling and adaptive backoff. It estimates total platform volume from sequential message IDs, so coverage is a measured number.
- History backfill — pulls up to 3,000 past posts per user, queued from the leaderboard or picked automatically, so people are graded on their whole record.
- LLM call classifier — reads untagged posts as bullish, bearish or "not a call" with a confidence score. In a six-model bake-off on 300 posts with hidden tags, Qwen 3.8 Flash via OpenRouter hit 87% versus 68% for a TF-IDF baseline, at about 1.6 cents per 1,000 posts. Gemma is the fallback; a daily spending cap keeps costs fixed.
- Outcome grader — daily bars from Yahoo Finance; entry is StockTwits' stamped price at post time (or the next open if it fails a sanity check); returns are signed by direction and compared with SPY, or BTC for crypto. Unfinished bars and data gaps are excluded.
- Pluggable strategies — each scoring method is a small class the runner and leaderboard pick up automatically. Six ship today: average excess return over 5 and 20 days (shrunk toward zero until there's enough evidence), a 5-day hit rate scored at its Wilson lower bound, "called a +50% move early," excess return using only the poster's own tags, and the 2017 rule kept for comparison.
- Dashboard and leaderboard — a live feed with hand-labeling buttons, collector health, hourly volume, and a leaderboard with a tab per strategy and a per-user drill-down of every call.
- Bluesky volume test — a 24-hour run on the full Bluesky Jetstream firehose to see if that network had enough stock talk to grade.
Results
Four days in, it had stored about 1.68 million StockTwits messages from 20,500 users (most via backfill), extracted 290,056 distinct calls, graded 275,849 of them against real prices, and scored 14,033 users. About 900 users qualify for each leaderboard. The collector has hit zero rate limits, and the classifier has read about 274,000 untagged posts for $4.30 total. On the same users, the 2017 rule gave busy posters 80%+ while the market-relative score showed them trailing SPY. The Bluesky test scanned 3.38 million posts in 24 hours and found only 2,915 with cashtags, over half from 22 bot-like accounts. StockTwits carries roughly 105,000 messages a day, so it became the primary source.
Tech Stack
Python, PostgreSQL 16, psycopg 3, SQLite, asyncio and websockets, Bluesky Jetstream, StockTwits public API, Yahoo Finance chart API, OpenRouter (Qwen 3.8 Flash, Gemma 4), vanilla HTML/CSS/JavaScript, systemd services and timers, LXD containers on Ubuntu