AI Simultaneous Interpreting: When the Machine Listens, Translates and Speaks

A speaker at the podium begins a long German sentence, and the crucial verb comes at the very end. A human simultaneous interpreter listens, interprets the context and begins speaking before the sentence is complete. An AI interpreter faces exactly the same task: it has to listen, translate and speak at the same time. How it does this depends heavily on its architecture. Until now, many systems for simultaneous interpreting have worked in several consecutive steps. New research approaches are trying to integrate this processing more closely. In this article, we explain the most important approaches, their advantages and disadvantages, and why the question of architecture is also relevant to you as an event organiser.

The classic pipeline: ASR, translation, speech output

The proven structure of an AI interpreting system is a chain of three building blocks:

• Speech recognition (ASR, Automatic Speech Recognition): A model converts the spoken word into text.
• Translation: A second model renders this text into the target language.
• Speech output (TTS, Text-to-Speech): An artificial voice speaks the translation.

Each building block performs a clearly defined task and passes its result on to the next. This is therefore referred to as a modular architecture or a pipeline.

The case for a modular structure

The greatest advantage lies in transparency. Between the steps, results are produced that can be viewed and checked, above all the transcript after speech recognition. If a word is translated incorrectly, it is possible to trace whether it was already recognised incorrectly or only altered during translation.

In addition, the building blocks can be optimised or replaced individually. Technical terms and proper names can be introduced precisely where they are needed, for example when recognising a product name or translating a technical term. And when a better model becomes available for one step, the entire system does not have to be rebuilt.

Where the pipeline reaches its limits

The chain also has weaknesses. Errors carry forward: whatever speech recognition misunderstands, the next stage translates as it was recognised, not as it was meant. In addition, the processing times of the individual steps add up.

A third point concerns the manner of speaking. When speech is converted into text, information such as emphasis, speaking pace or irony is easily lost. What is no longer in the text cannot be reproduced by the speech output at the end either. We have dedicated a separate article to this topic: Can AI translate emotions?

End-to-End Speech-to-Speech Translation

This is where a field of research comes in that has attracted considerable attention in recent years: End-to-End Speech-to-Speech Translation. The idea: a single model takes in spoken language and directly produces spoken language in the target language, without necessarily taking the detour via written text.

This offers potential advantages. Fewer handovers between individual models can speed up processing. And because the model works with the sound itself, there is a chance of carrying over characteristics of the original manner of speaking.

These are offset by challenges. An integrated model is harder to see into: if there is no visible intermediate step, it is harder to attribute an error to a cause. Targeted control, such as specifying certain technical terms, is also not as readily possible with a closed model as it is in a pipeline.

Audio-native AI models: sound as a source of information in its own right

Closely related to this approach are so-called audio-native AI models. They process audio signals directly instead of working exclusively with text. As a result, they can in principle use information that is contained only in the sound, such as pauses or emphasis.

The boundaries between the approaches are becoming increasingly blurred. Research also includes hybrid forms that combine elements of a modular chain with integrated models. The question is therefore increasingly not "pipeline or end-to-end?" but rather: which parts of the processing should be closely linked, and where is a visible, controllable intermediate step useful? Neither approach is superior across the board. Which architecture makes sense depends on what matters most in the particular use case.

Regardless of the architecture, one thing applies: every model can only work with the signal it receives. To find out why sound is the starting point of every translation, read our article AI interpreting and audio technology: why everything starts at the microphone.

Streaming: simultaneous interpreting before the sentence is over

For simultaneous interpreting at a conference or congress, translating well is not enough. The system also has to do it continuously. It cannot wait until a speaker has finished their presentation, but has to process the flow of speech in small segments. This principle is called streaming processing.

This results in a fundamental trade-off:

• Translating early means a short delay, but less context. The system may have to translate before it is clear where the sentence is going.
• Listening longer means more context and thus a better basis, but also a greater lag behind the original.

The example from the beginning shows why this is so demanding: in German, the verb often comes at the end of the sentence, whereas in many target languages it comes early. A streaming system therefore has to decide how long to wait. In addition, text can still be corrected in the background, but a word already spoken in the target language cannot.

Which developments are particularly relevant for real-time systems

For live use at events, we consider four developments to be particularly important:

• Streaming-capable models that handle incomplete sentences and strike a good balance between delay and context.
• Controllability, so that technical terms, product names and proper names from the programme reliably come across as intended.
• Monitorability, so that it is possible to see at any time during the event what the system is currently recognising and outputting.
• Efficiency, so that powerful models can also run on on-site equipment instead of depending on remote data centres.

What the architecture means for your event

You do not need any specialist knowledge of language models. However, the architecture of an interpreting system determines how flexibly it can be optimised, monitored and developed further. And that is exactly what you notice in practice: when the name of your main sponsor is rendered correctly, when a technical term from your industry is spot on, or when someone can step in quickly if there is a problem.

At MILA, we therefore see simultaneous interpreting not as an interface that you simply plug in, but as the entire processing chain from microphone to headphones. In concrete terms, this means:

• We set up technical terms and proper names from your programme in advance.
• The AI simultaneous interpreting runs locally on our own equipment at the venue, without the cloud. What is said does not leave your event.
• A system technician is always on site. He plans with you, sets up, supervises the interpreting during the event and dismantles everything afterwards.
• Each language is fed into your existing interpretation system or conference equipment on its own channel; on request, we provide receivers and headphones for the guests.

Research into speech-to-speech systems is developing rapidly, and we follow it closely. If you would like to know what simultaneous interpreting could look like for your congress, conference or corporate event, you will find details on the technology and process on our Services page. We have compiled answers to frequently asked questions in our FAQ.