Resume
← Work

CJSF Radio

Split the campus radio station's PHP app into services so the live stream stays up, fixed a race condition that broke about 15% of scheduled shows, and built archive search, live notifications and listener analytics.

300+
concurrent listeners
80K+
recordings searchable
90%
fewer analytics writes

Backend Developer · Apr to Jul 2025

Python, FastAPI, PHP, Laravel, PostgreSQL, Redis, Elasticsearch, WebSockets, Nginx, Docker, FFmpeg

What it is

CJSF 90.1 FM is SFU's campus and community radio station. Its website, show schedule, the job that starts each broadcast, the live stream and the recording archive all ran in one Laravel app on one server. A memory spike on an archive page, or a crash in the stream encoder, took everything down, including the live stream people were listening to.

Splitting the app

The main change was isolating streaming. The streaming service turns the live feed into HLS segments with FFmpeg and shares nothing with the website. If the web app is redeployed, overloaded or crashes, Nginx keeps serving the playlist and segments, and the 300+ people listening don't notice.

The station after the split
The station after the splitListeners and hosts reach Nginx, which routes by path to three containers: the Laravel web and scheduler service, a streaming service that turns the live feed into HLS segments with FFmpeg, and a FastAPI archive and realtime service. The web service stores shows and slots in PostgreSQL. The web and streaming services publish events to Redis, which the archive service consumes. The archive service searches 80,000 recordings in Elasticsearch.Docker containersListeners, hostsNginxroutes by pathWeb + schedulerLaravel (PHP)StreamingFFmpeg → HLSArchive + realtimePython · FastAPIPostgreSQLshows, slotsRedisStreamsElasticsearch80K+ recordings12SQL3events4search5/, /schedule → web/live → streaming/api, /ws → archive
  • client
  • service
  • database or storage
  • queue or stream
  • bold outline = what I owned
  • request
  • async: event, message, webhook
  1. Listeners and hosts reach Nginx.
  2. Nginx routes each path to its container: the website and scheduler, the live stream, or the archive API.
  3. The streaming service publishes listener events to Redis.
  4. The archive and realtime service consumes events from Redis.
  5. It searches the archive in Elasticsearch.

The migration was gradual. Nginx went in front of the old app first, then routes moved to the new services one at a time, streaming first because it mattered most. The old path stayed available as a fallback until the new one had proven itself.

The race condition

About 15% of scheduled broadcasts were failing, with nothing obvious in the logs. The scheduler ran as a periodic job, and when one run took longer than its interval, two workers overlapped. Both checked that a time slot was free, both saw yes, and both claimed it. I fixed it with a Redis distributed lock on each time-slot claim, so only one worker can ever own a slot.

Before and after
Before: two workers claim the same slotWorker A and worker B both check whether the 9pm slot is free, both hear yes, and both claim it. Both think they own the slot, and the broadcast fails.before: no lockWorker ARedisWorker Bis 9pm free?yesis 9pm free?yesclaim 9pmclaim 9pmtwo owners: the broadcast failsAfter: a Redis lock per slotWorker A takes the lock for the 9pm slot. Worker B tries the same lock, finds it taken, and backs off. Worker A claims the slot and releases the lock, so each slot has exactly one owner.after: Redis lockWorker ARedisWorker Block 9pmoklock 9pmtaken, back offclaim, unlockone owner per slot
  • service
  • database or storage
  • request
  • failure path

The archive has 80,000+ recordings with show, host, date, description and tags. SQL LIKE queries were slow and bad at relevance. Now a FastAPI service indexes the metadata in Elasticsearch, with full-text search on titles and descriptions and filters on show, host and date. Fuzzy matching means a search still finds the right record when a show or artist name is misspelled. It handles 500+ concurrent users.

PostgreSQL stays the source of truth. The index updates when a recording is added or edited, and it can be rebuilt from Postgres at any time.

Live notifications

Hosts wanted to know when their show went live, and listeners wanted "now playing" updates without refreshing. Services publish events to Redis Streams, and the FastAPI service reads them with a consumer group and pushes them to browsers over WebSockets in under 200ms end to end. With Streams, a consumer that briefly disconnects catches up on what it missed instead of losing it.

Archive search and live notifications
Archive search and live notificationsA web client calls a FastAPI service, which runs fuzzy search over 80,000 archive records in Elasticsearch. New archive items are published to Redis Streams, and a WebSocket gateway pushes them to connected clients in under 200 milliseconds end to end. Separately, the broadcast scheduler claims time slots through Redis locks.Web clientlisteners · staffFastAPI servicePythonconcurrent_users=500+Elasticsearchfuzzy searchrecords=80K+Redis Streamsarchive eventsWebSocket gatewaynotify=<200ms end to endBroadcast schedulerRedis lockstime-slot claimsbroadcast_failures=-15ppHTTPS1search2new item34push update5claim slot
  • client
  • service
  • database or storage
  • queue or stream
  • scheduled job
  • bold outline = what I owned
  • request
  • async: event, message, webhook
  1. A listener or staff member searches from the web client.
  2. The FastAPI service runs a fuzzy search in Elasticsearch.
  3. When a new item is added to the archive, the service publishes an event to Redis Streams.
  4. The WebSocket gateway picks the event up.
  5. It pushes the update to every connected client, in under 200ms end to end.

Listener analytics

The station wanted listener counts over time. The first version wrote one database row for every listener joining or leaving, which hammered the database during popular shows. Now the streaming service publishes those events over Redis pub/sub, an aggregator counts them in memory per show and interval, and it writes one batched insert per interval into a time-series table. That cut database writes by about 90% with no loss of detail the dashboards use.

DJLand

I also worked on DJLand, the platform CJSF's DJs and staff use to run the station: moving its old Laravel frontend to Next.js without touching the backend underneath.

DJLand: migrating CJSF's DJ platform

Key decisions

DecisionInstead ofWhy
Streaming in its own serviceKeeping it in the monolithThe live stream is the product; nothing else should take it down
Gradual migration behind NginxA full rewriteOne route at a time, with a fallback
Elasticsearch, with Postgres as the source of truthSQL LIKE queriesRelevance and speed; the index can always be rebuilt
Redis Streams for notificationsPlain pub/subConsumers catch up after a disconnect
Batched analytics writesOne row per eventAbout 90% fewer writes
The main choices, and what they replaced.

Results

concurrent listeners
300+
records searchable
80K+
search users
500+ concurrent
notifications
<200ms end to end
failed broadcasts
-15pp
analytics DB writes
-90%