OpenQQuantify

What it is

OpenQQuantify builds an AI design automation platform for hardware, including validation of FPGA configurations. Customers (tenants) submit design jobs, and each one produces a stream of events that other services and the UI consume.

A design moves through events like design submitted, validation started, validation finished and results ready. Four services handle them: the API, a design job service, a validation service, and a results and notification service. Traffic is bursty: about 50,000 events a day, with spikes around 10 times normal when a tenant starts a large batch job.

Architecture

Keeping events in order

Updates for the same design have to be applied in order. If "revision 3 saved" is processed before "revision 2 saved", the stored state is wrong. There were three ways to partition the Kafka topic:

With the tenant ID as the message key, every event for a tenant lands on the same partition, and each partition is read by one consumer at a time. Each tenant's events are processed in sequence, and different tenants run in parallel. There's no ordering between tenants, which is fine: nothing depends on whether tenant A's event came before tenant B's.

Retries can reorder writes too, so the producer is idempotent (enable.idempotence=true, acks=all): a retried send can't create a duplicate or land out of sequence. Consumers commit their offset only after the database write succeeds, so a crash means an event is read again, never lost. The result was zero out-of-order updates.

The downside of keying by tenant is that one very busy tenant fills one partition. The rate limiter below handles that at the front door.

Latency

During bursts, the API's p95 latency climbed to about 380ms. I profiled it with VisualVM under a burst load test. The thread view showed the service creating a new thread for each unit of work with no upper limit, so under a spike hundreds of threads competed for CPU and database connections. Context switching and lock contention were eating the time.

I replaced that with a fixed thread pool, two threads per CPU core, in front of a bounded queue. The work mixes CPU with waiting on the database and Kafka, so slightly more threads than cores keeps the CPUs busy without going back to heavy contention. When the queue is full, new work is rejected right away instead of everything getting slower. p95 dropped from 380ms to 270ms.

Overload protection

The 5xx spikes had one cause. The AI generation calls downstream are slow, and when one tenant launched a big batch job, their requests filled the worker pool and everyone else's requests timed out. More workers helped a little. The real problem was fairness, so I added a per-tenant rate limit at the API, before work reaches the pool.

Each tenant has a Redis sorted set of request timestamps. A sliding window means a tenant can't double its allowance by bursting at a window boundary. The check and the add run as one atomic Lua script, so two API instances can't both admit the last slot, and because the state is in Redis the limit holds across every instance. Over matching two-week windows before and after, server errors at peak fell from about 6 a second to about 3.3, 45% fewer.

Debugging in minutes

Before, diagnosing an incident meant SSH-ing into each box, grepping logs, and trying to line them up with recent deploys. It took 20 to 25 minutes to get from an alert to the first action. I added three things:

• Prometheus metrics on every service: request rate by tenant, error rate by endpoint, latency histograms and consumer lag.
• Correlation IDs, created at the API and carried in Kafka message headers into every service's logs, so one ID finds a request's whole path across four services.
• Grafana dashboards with deploy markers, so one screen shows which tenant or endpoint is the source and whether a deploy lines up with it.

In the three incidents I timed afterwards, the longest took 4.5 minutes from alert to knowing the cause.

Key decisions

Results

out-of-order updates

0

events

50K+/day

peak

10x normal

p95

380ms → 270ms

5xx at peak

~6/s → ~3.3/s (-45%)

time to diagnose

~25 min → under 5