Following the voice, word by word
Highlighting each word as it is spoken looks trivial and is not. A short tour of why aligning audio to text is hard, and the two ways to do it.
In short: Good highlighting follows the audio timeline, stays anchored inside each sentence, and survives reflow when the reader changes the page layout.
Watching a reading app highlight each word exactly as it is spoken feels like the least remarkable feature imaginable. The audio and the text obviously correspond; surely you just colour them in as you go.
You do not, and the reason is that speech synthesis does not hand back the one thing this needs: a list of which sample each word starts at.
What the model actually gives you
A neural text-to-speech model takes a sequence of tokens and returns a waveform. That is the whole contract. There is no per-word index in the output, because the model has no concept of a word. It works in phonemes, or sub-word units, mapped through an internal representation to audio frames.
So you have text on one side, and a few hundred thousand audio samples on the other, with nothing joining them. Building the join is the problem, and there are two families of answer.
Estimated alignment
The cheap approach: assume speech time is roughly proportional to written length, and divide the duration up accordingly.
You take the total length of the audio for a chunk of text, weight each word by its character count, and assign each a share of the time. The eleven-character word gets roughly twice the slice of the five-character one.
This is obviously wrong in the specifics. "Through" and "thorough" have almost the same length and noticeably different durations. Punctuation adds pauses that belong to no word. A rushed clause and a slow, emphasised one get the same treatment.
What saves it is that the errors do not accumulate. Because each chunk is anchored at its own start and end, drift is bounded within a sentence and resets at every boundary. The highlight can be a little early in the middle of a long sentence and still be exactly right at the full stop. For most listening, sentence-accurate with a slight wobble inside is genuinely fine: you are following along, not conducting.
Timed alignment
The accurate approach: get real timings out of the synthesis pipeline.
Some models expose per-token durations as a side effect of how they work. Any architecture with an explicit duration predictor, a component that decides how many audio frames each phoneme should occupy, already computes exactly the number you need. If you can read that out and map phonemes back to the words they came from, you get true timings rather than estimates.
The catch is that not every model exposes it, and the phoneme-to-word mapping is fiddlier than it sounds. The text has been normalised (numbers expanded, abbreviations rewritten) before reaching the model, so the tokens the model timed do not correspond one-to-one with the words on screen. "Dr." became "doctor", "1996" became four or five spoken words, and the highlight has to land on the characters the reader is actually looking at.
That mapping, from displayed characters through normalisation to phonemes to audio frames and back, is where the real work is.
Why the text side is just as hard
Even with perfect timings, you still have to highlight the right characters, and in a reflowable document that is not obvious.
The narration index and the visual layout are separate things. The index is a list of sentence-sized chunks with character offsets. The rendered page is a web view with its own line breaks, columns, and page boundaries that change whenever the reader changes the font size. Highlighting means turning "characters 412 to 419 of chunk 37" into a rectangle on screen, and that answer changes every time the layout does.
It also has to survive paging. If the word being spoken sits on the next page, the reader has to follow, but only when the reader has not deliberately scrolled somewhere else to look at something.
Why bother
Because for anything you are trying to understand rather than merely consume, reading and listening together is better than either alone. The voice sets a pace you would not otherwise hold, and the highlight keeps your eyes from wandering off. For dense technical material, or for anybody who finds a page of small text tiring, it is the difference between finishing the paper and abandoning it.
That is a lot of machinery for a moving green box. It is worth it.