Inference at Scale: Transcribing Millions of Insurance Calls with Whisper and Azure ML

Transcribing a few audio files with Whisper is easy. Transcribing millions of recordings efficiently, reliably, and cost-effectively is a very different problem.

In this talk, I'll dive into the Python code and infrastructure behind a large-scale speech transcription platform built for the insurance industry. Starting from a notebook prototype, we'll explore how the system evolved into a distributed inference pipeline running across thousands of GPU workers.

Rather than focusing on machine learning theory, we'll focus on inference engineering: …