A multi-voice audiobook engine with a swappable AI stack
Raw book text in. A performed, multi-character audiobook out, for about a dollar.
5 voice and LLM models integrated or evaluated. 3 interchangeable LLM backends (cloud API, local, cloud GPU). ~$1 per book to render, cached.
The problem
Off-the-shelf text-to-speech reads a book in one flat voice. I wanted a performed audiobook: a consistent narrator, distinct character voices, and emotion that tracks the prose, generated automatically from plain text, at a cost that makes rendering a whole novel practical.
That is three hard problems stacked. Something has to read the prose and decide, line by line, who is speaking and how. The voice models each win on different axes, so work has to be routed to the right one. And the whole thing has to be resumable, cacheable and cheap enough to actually use.
The insight
The interesting engineering was never a single voice model. It was the layer that decides what to say, in which voice, with what emotion, and the clean seams that let every model behind it be swapped. That is the same pattern a production voice product needs to swap one provider for another without touching the rest of the system.
What I built
A pipeline that runs a book from raw text to a finished, chaptered audiobook: capture, clean, attribute with an LLM, cast, render through the right voice engine, package and deliver. An LLM reads the prose and decides who speaks each line and with what emotion. I built that three ways behind one interface: a top-tier cloud model via the Anthropic API with context caching and schema-constrained output, a fully local model for free runs, and a cloud GPU endpoint when local is too slow. Switching brains is a config change, not a code change. The voice engines sit behind one protocol too, with a router that sends narration to a free local engine and expressive dialogue to a cloud GPU one, per segment.
Judgement calls, made on evidence
I ran a model bake-off and proved expressive voice cloning was too slow on the local GPU, which drove the hybrid design instead of guesswork. The first multi-voice render sounded jarring on listen-back, so I pivoted to one consistent cloned narrator who also voices the protagonist, with a small library for everyone else, and that was approved. When a character needed a non-American accent the cloning model kept fighting, I stopped tuning and switched that voice to a clean synthetic one rather than burn days on it. Each pivot is written down with the evidence behind it.
Quality
Around ninety tests, green and clean, including a contract test that runs the cloud engine against a fake worker with zero spend, and a preflight that exercises the real backend through a mock before any paid render. Attribution coverage is measured per chapter, not assumed. Full pipeline proven on real books: one novel at 29 chapters and about 7.4 hours of audio, another at 43 chapters through the local path.
What I would bring to your product
Provider-swappable AI architecture, so you are never locked to one vendor. Real cost and reliability engineering on serverless GPU. And LLM integration done properly, with caching, structured output, and honest trade-off analysis behind every choice.