RBC, Amplify

What it is

About 23 million calls a year reach RBC's contact centre. In some of them, a real client is on the line while a scammer coaches them to move their money. Every security check passes on those calls: the voice matches, the number isn't spoofed, and the login is genuine. The only evidence is what the client says, and nothing was reading it.

We built a two-stage pipeline that reads the transcripts, finds the fraud calls, and pulls out what the threat hunting team needs: the fraud type, plus any URLs, phone numbers and payment methods, each linked to the line in the transcript that supports it. It's patent-pending, I'm a named inventor, and I presented it to 1,000+ people at RBC's internal expo.

The team was two developers, a data engineer and a business analyst. I owned the cascade design, the backend orchestration service, the transcript ingestion path, auth, and the OpenAPI contract between the three services.

RBC Amplify, the program I built this in →

The gap

RBC's fraud tooling already covered fake websites and fraudsters pretending to be a client. Authorized push payment fraud, where the real client is tricked into sending the money, fell between the layers.

The threat hunting team had the opposite problem. They track phishing domains and scam campaigns, but nothing from the phone channel reached them. When a client said "I went to rbc-secure-verify.com", that domain never made it back.

Our two sponsors gave us a question, not a spec: how could the voice of clients help detect fraud faster? We spent the first weeks on stakeholder interviews and personas (client, advisor, threat hunter) and narrowed it to one buildable goal: turn call transcripts into structured fraud signal the threat team can act on.

Three services, one contract

The platform is three containerized services on OpenShift. The analyst console is React. The backend API, which I owned, is Python with FastAPI and async SQLAlchemy over PostgreSQL: it runs the pipeline, stores detections and analyst reviews, and serves the console. The ML service, also FastAPI, runs the two-stage cascade. Transcripts come from S3, Keycloak handles sign-in, and secrets come from Vault at deploy time.

The ML service is separate from the backend because the two fail and scale differently. The ML service is CPU-heavy and calls a paid external API. The backend mostly waits on Postgres. With them apart, the model can be redeployed without touching the API analysts use, and a slow LLM call can't hold up a dashboard query.

Early in the term I wrote and froze an OpenAPI v2.1 contract for every call between the three services. The frontend was built against it with mock data while the backend was still in progress, and any change after the freeze needed a version bump. That let four people build three services at once and integrate them without rewrites.

Why a cascade

An LLM is good at reading a messy transcript and pulling out "investment scam, the client mentioned crypto-wallet-xyz.com and was told to send an e-transfer." It's also slow and costs about $0.10 a call. Most calls are address changes, card activations and balance questions, and paying LLM prices to confirm that is waste.

So a cheap classifier decides which calls are worth the LLM. It lets about 35% of calls through, which brings projected spend from $2.35M to $822K a year.

The gate's threshold is 0.15, set low on purpose, because its two mistakes cost very different amounts:

Stage 2 corrects the gate's false positives, but nothing corrects its false negatives. When the two stages disagree, a fixed rule decides:

• The gate scores a call as clearly safe: it stops there, on the cheap path.
• The gate passes it and the LLM says fraud: it's a detection.
• The gate passes it with low confidence and the LLM says no: it's cleared.
• The gate is highly confident and the LLM still says no: it isn't dropped. It's flagged as suspicious for a human to review.

In that last case neither model overrules the other; an analyst decides.

What the LLM does

Calls above the threshold are batched and sent to the LLM through RBC's OpenAI-compatible gateway. For each call it does three things:

• Classifies it. A call can be more than one fraud type at once, such as impersonation and an investment scam, so this is multi-label.
• Extracts indicators from the text: URLs and domains, phone numbers, payment methods, amounts and claimed organizations.
• Points to the evidence: the exact span of the transcript behind each finding, so an analyst can check it in seconds.

Labels without ground truth

Nobody had ever marked which calls involved fraud, so there was nothing to train the gate on, and hand-labelling needs someone who knows the fraud taxonomy, which would have taken weeks of analyst time. So labels came from an LLM ensemble: several GPT model versions each labelled every transcript more than once, and a majority vote set the final label, so one inconsistent answer couldn't flip it.

Runs agreed with each other almost completely, so the process was stable. Stable isn't the same as correct, though: every run could be wrong in the same way. So we hand-labelled a separate set of transcripts and compared. The ensemble agreed with the human labels on the large majority of calls, and that's when we knew the labels were good enough to build on.

The gate is trained on LLM labels and stage 2 is an LLM, so the gate learns to predict what the LLM will say. That's intended: stage 1's job is to cheaply predict whether stage 2 will find something. The hand-labelled set is the independent check, and it's also the guard against the gate learning the LLM's blind spots.

Choosing the classifier

Keyword rules don't work here, because fraud calls don't share vocabulary. "He said he was from RBC security" and "the caller pretended to work for the fraud team" mean the same thing with no words in common. So every transcript is turned into a 384-dimension embedding with a SentenceTransformer model (all-MiniLM-L6-v2), which puts both phrasings in the same place. We tried shrinking the embeddings with PCA, but precision got worse, so we kept the full vectors.

We benchmarked SVM, XGBoost, CatBoost, LightGBM, random forest and logistic regression. The complex models were only marginally better. Logistic regression won on what the gate needs: its score is a real probability you can set a cutoff on, it runs in under a millisecond after embedding, and it's easy to explain to a bank's model-risk reviewers. With a small training set built on LLM-derived labels, extra model capacity mostly meant overfitting risk.

Fraud is a small minority of calls, so a model can look accurate by always saying "not fraud." We split train and test with stratified sampling so both kept the real fraud ratio, and handled the imbalance by moving the cutoff instead of resampling, since resampling distorts the probabilities the whole cascade depends on.

77% precision was a deliberate trade: a false alarm costs an analyst a few seconds, while a missed fraud is never looked at again. One caveat I'd raise myself: fraud is rare, so the test set has only about 40 fraud calls, and 94% recall on 40 cases has a wide confidence interval.

The daily pipeline

The pipeline runs once a day. Each morning a job processes the previous day's calls. The threat hunting team works campaigns over days, so a daily batch matched how they work, was far simpler to run than streaming, and fit how transcripts already landed in S3.

Every write is keyed on a natural key, such as the call ID, or the indicator value for indicators. If the job crashes halfway, or someone reruns yesterday by hand, nothing is written twice.

Auth

Auth was designed in from day one. Analysts sign in through Keycloak single sign-on, and a silent check signs them in without a redirect if they already have a session. The console uses OIDC with PKCE, because a browser app can't keep a secret and PKCE stops a stolen authorization code from being used by anyone else.

Every API call carries a bearer token. The backend checks its signature, issuer, audience and expiry against Keycloak's published keys (JWKS), cached so it doesn't call Keycloak on every request. Then role-based access control decides what each role can see. The backend passes the same token on to the ML service, which checks it too, so no internal endpoint is left open.

The API

The backend is async all the way down: FastAPI with async SQLAlchemy and asyncpg, so one worker handles many requests while it waits on Postgres and the ML service. Its routes are grouped by job:

The ML service has three endpoints, all behind the same JWT auth:

{
  "call_id": "c-2026-07-14-00421",
  "model_version": "cascade-lr-v3",
  "stage1": { "score": 0.62, "escalated": true },
  "stage2": {
    "labels": [
      { "fraud_type": "impersonation", "confidence": 0.91 },
      { "fraud_type": "investment_scam", "confidence": 0.74 }
    ],
    "indicators": [
      { "kind": "domain", "value": "secure-rbc-verify.com" },
      { "kind": "payment_method", "value": "e-transfer" }
    ],
    "evidence": [
      { "span": [1180, 1264], "excerpt": "they said to go to secure-rbc-verify dot com and..." }
    ]
  }
}

The analyst console

• Overview: how many calls were screened and escalated, and fraud types over time.
• Detections: each flagged call with its labels, scores and indicators, and the transcript with the evidence highlighted.
• Indicators: every extracted domain, phone number and payment detail, with how many calls mention it. This is the signal the threat team never had before.
• Campaigns: indicators grouped across calls into likely scam campaigns.

Early on we also designed a view for advisors on live calls, with keyword highlights and suggested next steps. It shaped the product direction, but the version we shipped focused on the threat team.

Running it

Every service is a Docker image deployed to OpenShift, with secrets (database credentials, the LLM gateway key, Keycloak config) injected from Vault. OpenTelemetry traces follow a request from the console through the API to the ML service, and Grafana shows latency, error rates and score drift: if the share of calls passing the 0.15 gate shifts, either the calls or the model have changed.

MLflow tracks evaluation runs and model versions, and every detection stores the model version that produced it, so any finding can be traced back to the exact model. There are tests across all three services: pipeline helpers, the transcript parser, endpoints, API schemas, classifier utilities and the frontend's data hooks. Database changes ship as Alembic migrations.

Key decisions

Results

recall

94%

precision

77%

f1

85%

test set

499 held-out calls

calls filtered before the LLM

≈65%

LLM spend

$2.35M → $822K/yr (projected)

fraud loss mitigated

~$18M/yr (projected)

clients protected

12K+/yr (projected)

demo

1,000+ attendees

The model was evaluated on 499 held-out calls, not run on all 23 million. The cost figures project that sample over the annual volume. The fraud-loss and clients-protected figures come from a separate business-case model built on historical fraud volumes, average losses and how often earlier detection allows an intervention. The $18M is the conservative end of that model.

What I'd change

• Label more fraud calls by hand. About 40 positives isn't enough for a tight recall estimate, and k-fold cross-validation would help too.
• Mask personal information before transcripts reach the LLM gateway.
• Add a model registry, so retraining and rollback aren't manual.
• Close the loop on drift. We could see it, but nothing acted on it automatically.
• Finish replacing the mock data fallbacks in the console with live data.
• Connect straight to the call platform instead of exported transcripts, and add streaming for real-time alerts to advisors.
• Finer-grained roles for each analyst team.