Beyond Text: How Multimodal AI Works

One architecture, several kinds of data

The interesting fact about multimodal AI is how little had to change. The transformer does not care that text is text. It operates on sequences of vectors. Anything you can turn into a sequence of vectors, it can process.

So:

• Text → tokens → vectors.
• Images → patches → vectors, via a vision encoder.
• Audio → short spectrogram frames → vectors.

Crucially, these land in the same representation space. The model attends across an image and a question about it in one pass, exactly as it attends across two sentences. There is no separate captioning step in the middle.

That is why “what is wrong with this diagram?” works at all — and why the model’s reasoning about an image inherits the same strengths and weaknesses as its reasoning about text.

Two very different ways to generate

Understanding this split explains most of what you observe.

Autoregressive — one piece at a time

Text generation predicts the next token, then the next, each conditioned on everything before. Sequential, and it can only look backwards.

Characteristic failures: drift over long outputs, and no way to revise an earlier commitment. Once a sentence is written it stands.

Diffusion — refine the whole thing at once

Image generation usually starts with pure noise and repeatedly removes a little of it, steered by your prompt, until an image emerges. Every step sees the entire canvas.

Characteristic failures: global structure lands well — composition, palette, mood — while precise local detail goes wrong. This is exactly why the classic artefacts are hands, counts of objects, and small text. The model is optimising for “looks like the right kind of image”, and local precision is not what that objective rewards.

It also explains why “exactly four people” is unreliable while “a crowded market at dusk” works beautifully. One is a discrete constraint; the other is a global impression.

The boundary is blurring — some systems now generate images token by token, and some text systems use diffusion. But the failure signatures above still describe most of what you will encounter.

What is reliable today

Worth being blunt about the gradient, because it is steep.

Reliable enough to build on:

• Speech-to-text. Mature, measurable, near-human on clear audio. Accents, overlapping speakers and jargon remain the weak spots.
• Document understanding. Scanned invoices, forms and receipts into structured fields. Well-constrained, and the output is checkable — the numbers reconcile or they do not.
• Image understanding for description and classification. Solid for “what is in this picture” and “does this photo show damage”.

Useful with a human in the loop:

• Image generation for concepts, mood boards and drafts. Excellent at style, unreliable at specifics.
• Text-to-speech. Genuinely good now; still occasionally mangles emphasis and unusual names.

Impressive, not yet dependable:

• Video generation. Improving fast. Physics, object permanence and character consistency across a long clip remain hard.
• Precise text inside generated images. Better than it was, still a coin toss on anything long.
• Exact spatial and numeric constraints. “Three items on the left” is not a promise.

The pattern: constrained tasks with a verifiable answer are reliable; open-ended generation is not. That is a good heuristic for anything new you encounter.

Prompting for images differs

Text prompting is instruction-following. Image prompting is closer to specifying a coordinate in style space. Four dimensions do most of the work:

Subject — what, and in what state. Setting — where, and the light. Style — medium, era, technique. Composition — framing, angle, depth of field.

“A fluffy orange tabby on a windowsill at sunset, watercolour, warm palette, soft light, shallow depth of field, close crop.”

Two things that carry over from text prompting: iterate rather than rewrite, and negative constraints are weak. Asking for no text often produces text, because the concept had to be represented in order to be negated. Describing what you do want is more effective than listing what you do not.

The things worth thinking about

Not a moral lecture, just what is actually contested:

• Training data and consent. Most large generative models learned from work whose creators did not agree to it. This is unsettled in law and varies by jurisdiction.
• Commercial safety. Some providers indemnify commercial use of outputs; most do not. Check before shipping client work.
• Synthetic media of real people. Voice cloning needs a few minutes of audio. Impersonation is the obvious harm, and provenance standards are being adopted but are far from universal.
• Disclosure. Norms are converging on labelling generated media. Increasingly it is a legal requirement, not just courtesy.

Try this: Give the same image prompt to a generator twice, once asking for a precise count of objects and once describing an overall impression. The gap in reliability between the two is the diffusion failure mode, and once you have seen it you will predict it everywhere.