Tidepool
A streaming recommender that ranks Bluesky's live post stream into a personal feed of recruiting posts.
- 85ms
- serving latency, p99
- <2s
- feature freshness, p95
- 19 min
- nightly training, down from 48
Solo project
Python, Kafka, Flink, FAISS, PyTorch, Spark, Delta Lake, Airflow, AT Protocol
What it is
During recruiting season, the useful posts (new grad and intern postings, referral offers, notes on interview processes) were buried in noise. Tidepool reads Bluesky's live stream of public posts and ranks it into a personal feed of recruiting posts worth reading. I built it for my own internship search.
Bluesky suited this for two reasons. Its whole network of public posts is streamed openly (the "firehose"), so there's real, high-volume data to work with. And anyone can publish a custom feed: a small service returns a ranked list of post IDs, and it shows up in the Bluesky app like any other feed. Tidepool appears in my app as a feed, and everything behind it is my own pipeline.
Architecture
- client
- service
- database or storage
- queue or stream
- model
- scheduled job
- external
- request
- async: event, message, webhook
- An ingest worker reads Bluesky's firehose, about 2,500 events a second.
- Posts land in Kafka, which absorbs bursts and lets consumers restart without losing anything.
- A Flink job filters, embeds and scores each post, writing embeddings to FAISS and features to the feature store.
- The Bluesky app asks the feed server for my feed.
- The server pulls the 200 most similar recent posts from FAISS.
- The ranker scores them, using features from the feature store, and the best come back as my feed.
There are three paths, each with its own time budget. Designing them separately kept each one simple:
| Path | Job | Budget |
|---|---|---|
| Streaming (Kafka, Flink) | Turn raw posts into features and embeddings | Seconds: p95 under 2s |
| Serving (FAISS, ranker) | Answer "what should I see now?" | Milliseconds: 85ms p99 |
| Batch (Spark, Airflow) | Retrain the ranker on history | Minutes, once a night |
From firehose to features
Kafka sits between the firehose and everything else. The firehose is bursty and external, and if the Flink job is redeployed, posts wait in Kafka instead of being dropped.
For each post, the Flink job drops non-English posts and obvious junk, computes a text embedding, and extracts features: keywords like "intern", "new grad" and "referral", company mentions, links to job boards, and the author's posting history. A small spam classifier in the stream removes engagement bait and repost spam before it reaches the index.
Freshness is the time from a post appearing on the firehose to its features being ready for ranking: p95 under 2 seconds. A recruiting post is most useful in its first few hours, so fresh matters more than complete.
Retrieve, then rank
Like most production recommenders, serving has two stages. A cheap one narrows thousands of recent posts to 200, and an expensive one orders those 200 carefully. Ranking every recent post on each request would blow the latency budget.
- client
- service
- database or storage
- model
- request
- The Bluesky app asks for my feed.
- FAISS returns the 200 posts nearest to a vector that represents my interests, in a few milliseconds.
- The server fetches features for all 200 in one batch.
- A small PyTorch ranker scores all 200 in one batched forward pass, using the embedding plus keyword, author and recency features.
- The server sorts the results, keeps one post per author so a prolific account can't flood the feed, and blends in recency.
Nightly retraining
Delta Lake stores the history of posts and features as versioned tables, so each night's training set can be rebuilt exactly. Spark joins posts with labels: the ones I engaged with or marked relevant, and the ones I scrolled past. Airflow runs the steps in order (build the data, train, evaluate) and promotes the new model only if it beats the current one on a held-out set, so one bad night of data can't quietly make the feed worse.
The first version of the nightly run took 48 minutes. Most of that was Spark rereading the full history and shuffling large joins. Partitioning the Delta tables by date, processing only new partitions, and tightening the joins brought it to 19 minutes.
Key decisions
| Decision | Instead of | Why |
|---|---|---|
| Kafka in front of processing | Reading the firehose directly | Absorbs bursts; consumers restart without losing posts |
| Flink for features | Hourly batch features | Recruiting posts are time-sensitive |
| Retrieve, then rank | Scoring every recent post | Stays inside the latency budget |
| One batched forward pass | Scoring posts one at a time | 200 candidates in one pass |
| Promote only if better | Always deploy the latest model | A bad night can't degrade the feed |
| Incremental, date-partitioned Spark jobs | Recomputing everything nightly | 48 → 19 minutes |
Results
- firehose
- ~2,500 events/s
- feature freshness
- p95 under 2s
- candidates scored
- 200 per request
- serving latency
- 85ms p99
- nightly training
- 48 → 19 min