The game

Listen to a word and try to spell it.

It evolved to include hearts, clues, streaks, single player, multiplayer, replay mode, and all sorts. It’s wild what you can do when you have AI at your side.

It was fun to make and fun to play.

The audio was more uneven. Some words sounded great. Others had lost their beginning, stopped too early, or brought a fragment of the preceding word along with them. For a game where the player has to work out what they heard, that is a fairly central problem.

I wanted a WAV file of each word, spoken clearly from beginning to end.

The voices

Using audio.cpp, I created AI voices for English, French, and Norwegian.

Initially I used OmniVoice, before moving to VoxCPM2 for voice creation. That move was driven by the licensing requirements.

I used VoxCPM2 to create a reference recording, then used that recording with Chatterbox (model running thru audio.cpp) to generate the sentences. The reference gave Chatterbox an example of the voice and delivery I wanted.

Creating a voice

After my first attempts, a Reddit comment about helping children learn to spell made me think more carefully about the voice. I wanted clear articulation and a patient delivery.

A back-and-forth with AI led to a teacher prompt. Alongside it, I supplied a short reference sentence: enough speech to establish the voice and accent without making the model follow a long passage’s rhythm.

Here is a complete command example, using a reference sentence from my experiments. Run it from the audio.cpp directory after building the CLI, and adjust the model path for your setup.

TEACHER_PROMPT=$(cat <<'PROMPT'
(
A professional female Norwegian literacy teacher in her thirties.
Her voice is warm, calm, clear, and exceptionally articulate.
She has precise consonants, clean vowels, excellent diction, and natural Norwegian pronunciation.
Her voice is friendly and reassuring without sounding childish.
She speaks with a composed, confident teaching presence.
Her pitch is natural and comfortable.
Her voice is clean, with no raspiness, breathiness, vocal fry, or exaggerated emotion.
)
PROMPT
)

REFERENCE_TEXT='Når vi snakker tydelig og rolig, blir det lettere å høre hver enkelt lyd. Vi bruker stemmen naturlig, med klare vokaler og presise konsonanter.'

./build/macos-metal-release/bin/audiocpp_cli \
  --task tts \
  --family voxcpm2 \
  --model "$HOME/models/VoxCPM2-GGUF/voxcpm2-q8_0.gguf" \
  --backend metal \
  --text "${TEACHER_PROMPT} ${REFERENCE_TEXT}" \
  --out teacher.wav

I then slowed the recording to 85% of its original tempo:

ffmpeg -i teacher.wav \
  -filter:a "atempo=0.85" \
  teacher-slow.wav

That gave me a more deliberate reference recording to use for sentence generation. The next job was making the individual words reliable and clear.

The audio

We went on a journey, AI and I. Some experiments moved us forward; others sent us back to listening.

Add knobs

I started by adjusting the controls available to me: generation steps, guidance, speed, seeds, and word-control tokens.

Generate a word, listen, change something, try again.

There were encouraging results. There were also results that made it hard to know what I had learnt. A different seed could rescue a recording, but it did not explain why the previous attempt had failed. A setting that helped one word did not necessarily help the next.

It wasn’t going well.

Give the word a sentence

The models I tried struggled with isolated words, so we added a sentence template to give them more context and encourage a natural delivery.

This gave the model some context and gave me a pattern to look for. The last word should follow a pause. Find that pause, cut there, and save the result.

This is the word ‘acquaintance’. Acquaintance.

The problem

Where should I cut the sentence to capture the final word?

It sounded like something FFmpeg could do in one line. That turned out to be wishful thinking.

The first useful distinction was between detecting silence and deciding what that silence meant. FFmpeg could report quiet intervals. It could not tell me that a particular interval separated the introductory sentence from the final word.

Even reading the settings involved some learning. A minimum silence of 0.12 meant a quiet interval had to last 120 milliseconds to qualify. It was not how long to record. If the useful pause was shorter, my script could miss it completely.

Reducing that threshold found more pauses. It also gave the script more opportunities to pick the wrong one.

A plausible cut can still be wrong

I tried rules based on the last pause, the longest pause, and how much sound remained afterwards.

Each had a reason behind it. Each met a recording that exposed the assumption.

With kjole, an early cut kept almost the whole sentence. Rejecting an output that retained 80% or more of the recording caught that sort of mistake. But a file could be much shorter than the original and still contain the wrong audio.

At the other extreme, one candidate left only about 26 milliseconds of sound above the threshold. That was a useful clue: perhaps I had found a gap inside the word and kept only its ending.

I added a rough minimum based on the number of letters. That helped reject implausibly small fragments, but letters are an imperfect measure of spoken duration. Quiet sounds also complicate any rule based on time above a volume threshold.

Acquaintance kept coming back. Its ending sounded troublesome even when I could hear the sentence pronounce it properly. But a check of the trial clip showed it contained the complete, unchanged ending. My first description of the problem was not enough to diagnose it. Jente exposed a different difficulty: the gap before the repeated Norwegian word could be tight enough that a seemingly reasonable cut still sounded wrong.

I needed to see what the rules were choosing.

Visualise

I asked AI to build a small browser tool for opening a WAV, selecting a region, and playing exactly that region. It became audio-lab. As it evolved, it became much easier to show the AI what I meant and check the changes we made.

I could drag the boundaries, zoom in, slow playback, and listen again. Then we added silence detection, with all the detected gaps highlighted together over the waveform.

The decibel threshold made more sense when I could change it and see which areas qualified as quiet. These bands represented sound below a chosen threshold for a chosen duration. They were not proof of empty space between words. A quiet part of speech could end up inside a band too.

The next useful addition was selection JSON: a small record of the file and the start and end times of a region. The extractor could write its proposed selection, and audio-lab could load it over the original recording.

That made the script’s decision inspectable. I could load its JSON, hear exactly what it proposed, and compare the boundary with the detected gaps. A file fingerprint helped make sure the selection belonged to the recording I had opened.

I could also move the selection by hand and copy the new coordinates back into the discussion. Instead of describing a vague problem at the beginning, I could point to an interval and explain what I heard there.

Hear the difference

Here’s what those decisions looked and sounded like. This recreated comparison uses one saved acquaintance sentence, made with a later template. Both cuts use that same recording, so the comparison demonstrates the cutting failure, not a change in generated speech.

The simple rule chooses a short gap after the final word has begun. The later rule chooses the longer gap before it. Both keep the original ending.

Download WAV · Download selection JSON

Download WAV · Download selection JSON

The controls below each screenshot play its extracted WAV; click the image to inspect it at full size. To reproduce the selections in audio-lab, open the complete sentence WAV and then load either selection JSON. The reproduction commands include the extraction settings.

This was a particularly useful role for AI. It could make the interface I needed while I was still discovering what I needed to inspect.

Estimate the sentence, then look nearby

We knew more than the waveform alone. We knew the word and the sentence template.

That suggested estimating where the introductory sentence should end, then looking for a nearby pause. I tried counting letters and assigning a rough number of milliseconds per letter.

There was a mistake hidden in that too. The small number I had used as a minimum-duration check for a word was unsuitable as an estimate of normal speaking time. Those calculations served different purposes, even though both used milliseconds per letter.

A larger estimate got us into a more useful part of the recording. Then leading silence, speaking pace, and trailing silence became important. A fixed rate was still a guess, and different voices did not all speak at that rate.

The next step used the recording itself to estimate the rate. Find the sentence’s start and the last sound, account for the known text, and evaluate candidate gaps with their duration taken out of the timing calculation. Use that estimate to help choose a plausible boundary.

This was still a collection of heuristics. I was learning from examples and improving rules; I was not training a model to recognise word boundaries.

Audio-lab made that distinction useful rather than frustrating. I could inspect a guess and work out why it had failed.

Change the sentence, too

After all that attention to the cut, I learnt something I wish I had tried earlier: changing the sentence template could improve the generated audio itself.

I experimented with the wording, quotation marks, and punctuation:

This is the word ‘{word}’. {spoken}.
This is the word "{word}". {spoken}.
This is the word '{word}'. {spoken}.
This is the word '{word}.' {spoken}.

This version worked best in my English tests:

Please say the word '{word}.' {spoken}.
Please say the word 'acquaintance.' Acquaintance.

Notice the full stop inside the quotation marks and the capitalised first letter of the repeated word. The combination helped in my tests. I hadn’t isolated each change, so I couldn’t say how much the wording, punctuation, or capitalisation contributed individually.

Acquaintance worked cleanly with this template. The source sentence in the audio comparison above uses it. I had been asking where to cut the recording; changing what I asked the model to say was another way to improve the result.

Give cleanup its own job

A later acquaintance recording had a little hiss before the word. The main extraction could get me close while still leaving an unwanted opening sound.

I could have kept adding conditions to the extractor. Instead, I added a separate cleanup step operating on the extracted word.

It examined the beginning for an opening sound followed by a qualifying quiet gap, then proposed a later start with a little padding. It wrote a preview WAV and JSON so I could inspect the result in audio-lab before accepting it.

This was trimming an unwanted lead-in, rather than removing noise throughout the recording. Finding a quiet gap did not by itself establish that everything before it was disposable.

Separating the steps helped. I could keep the generated sentence, repeat extraction, and try cleanup settings independently. Some recordings that still sounded rough after extraction became useful after cleanup.

A second opinion, and a better listening desk

I used Whisper and nb-whisper to compare recognised text with the expected word. A mismatch gave me somewhere to look.

There was a configuration mistake here too: sending language=no did not mean I was using nb-whisper. It selected Norwegian transcription on the server receiving the request. The regular Whisper and nb-whisper models were running separately; I had to use the correct endpoint.

Recognition results also needed listening judgment. In one Norwegian batch, seven recordings were flagged, but after listening I considered only one wrong. In another English check, I agreed with the seven flagged results. A mismatch was useful evidence, not a verdict.

review.py made this manageable. It turned the batch JSON into a listening page, with the source sentence, extracted audio, available cleanup previews, word IDs, transcripts, and expandable metadata. Filters let me concentrate on mismatches or recordings with cleanup versions.

That little page saved a lot of opening files and remembering which version I had just heard.

Licence

I wanted to keep the possibility of commercial use open, and the specific OmniVoice Word-Control weights I was trying were marked CC-BY-NC-4.0. Checking the licence on the code alone would have missed that. OmniVoice Word-Control model card.

That pushed me towards Chatterbox for the sentence-generation workflow and VoxCPM2 for voice experiments. Their published model cards list MIT and Apache-2.0 respectively. The particular model and weights mattered when making that decision. Chatterbox, VoxCPM2.

Tooling

The two small tools made a big difference: audio-lab let me inspect a cut, and the review page let me move quickly through a batch, listening and comparing versions.

Before AI, I would have weighed up the time needed to build a tool against the time it might save. That was harder when I didn’t yet know what the tool needed to do. With AI at my side, I could get a first version in minutes, then improve it in small steps as I learnt.

I still had to listen, notice the mistakes, and decide what was worth trying next. But I could turn a question into a waveform selection, a comparison page, or a repeatable command while the problem was fresh in my mind. Without those tools, I don’t think I would have reached my virtual finish line.

The goal stayed simple: press play and hear one complete word, clearly enough to have a fair go at spelling it.

References

Projects and models

• audio.cpp
• OmniVoice model card
• OmniVoice Word-Control model card and licence
• VoxCPM source code
• VoxCPM2 model card
• Chatterbox model card
• Whisper source code and documentation
• nb-whisper-small model card

Reading and notes

• Reddit comment about helping children learn to spell
• Licensing notes in this article

Examples and downloads

• Game contact sheet
• Complete acquaintance sentence (WAV)
• Recreated bad cut: WAV, selection JSON, audio-lab screenshot
• Corrected cut: WAV, selection JSON, audio-lab screenshot
• Review-page screenshot
• Reproduction commands