Speech-to-Speech Translation: How Modern AI Speech Translation Works

A speaker talks in German while guests in the hall follow the presentation through headphones – in English, French or Polish. What seems to the audience like a single process actually consists, technically, of several tasks that have to mesh closely. This is exactly what Speech-to-Speech Translation is about: translating spoken language into spoken language. In this article, we explain how the classic processing chain is structured, where its limits lie, which new AI architectures researchers are testing – and why these questions are also relevant to you as an event organiser.

Three tasks, one goal

A professional AI interpreting system has to combine several complex tasks. It must:

• recognise spoken language,
• translate the content into the target language,
• and generate intelligible speech output from it.

A human simultaneous interpreter does all of this in one head: listening, understanding, rendering, speaking. In machine simultaneous interpreting, these tasks have traditionally been distributed across different models, each specialised in its own part. We look at the fundamental differences between AI interpreting and human conference interpreters in our article AI interpreter or human: Will the simultaneous interpreter become obsolete?

Splitting the work across several models has advantages, but also technical limits – both are worth a closer look.

The classic processing chain of Speech-to-Speech Translation

In its classic form, Speech-to-Speech Translation runs in three consecutive steps. The output of each step is the input for the next.

Step 1: Automatic Speech Recognition (ASR)

The process begins with automatic speech recognition, or ASR. It converts the audio signal – for example from the microphone at the lectern – into text. Spoken language becomes words and sentences.

Step 2: Machine translation

Next, a machine translation model translates the recognised content into the target language. It works with the text supplied by speech recognition – not with the original audio.

Step 3: Speech synthesis (Text-to-Speech)

In the third step, a Text-to-Speech system, or TTS, generates the spoken output from the translated text. This is the voice the guests ultimately hear in their headphones or through the conference system.

The advantage: each component can be improved individually

This modular architecture has a clear advantage: the individual components can be optimised independently of one another. If a better model becomes available for one of the steps, it can be deployed without having to redevelop the other building blocks. Each part of the chain can be specifically tested, adjusted and replaced.

Where the challenges lie

Clear as the chain is, it brings fundamental challenges with it.

Every processing step takes time

Speech recognition, translation and speech synthesis run one after the other, and each of these steps takes time. In simultaneous interpreting, where guests should be able to follow the programme with as little delay as possible, this is a key factor.

Errors travel through the chain

Another challenge is less obvious: errors can be passed from one component to the next. If speech recognition misinterprets a term, the translation model is already receiving flawed information. It may then translate cleanly – but translate the wrong thing. At the end, speech synthesis makes this error audible.

This is particularly tricky with technical terms and proper names, which come up constantly at congresses and specialist conferences. A single misrecognised term can give a sentence a different meaning. That is why the first step is so important – and with it the quality of the input signal. We explain why the sound makes such a difference in our article AI interpreting and audio technology: Why everything starts at the microphone.

End-to-End Speech-to-Speech Translation: the more direct route

An alternative field of research deals with models that process speech information more directly. This is known as End-to-End Speech-to-Speech Translation. Such systems can reduce certain intermediate steps or link them more closely internally – instead of strictly separate stages, a more integrated processing path emerges.

What integrated models can preserve

The appeal of this approach lies in what can be lost in the classic chain. As soon as spoken language is converted into text, mainly the words remain. Voice characteristics, prosody – that is, stress, rhythm and intonation – and parts of the linguistic context are not contained in the text.

Integrated approaches open up new possibilities for carrying such information across. We have summarised how far research has come in conveying emotions, for example, and where the limits lie, in our article Can AI translate emotions?

No automatic shortcut

However, integrated systems are not automatically faster or more reliable just because they have fewer separate intermediate steps. Their controllability and adaptability must also be taken into account.

For use at events, this is no minor matter: anyone who wants to set up technical terms and proper names in advance and supervise the interpreting during the event needs a system that allows targeted intervention.

Modular or integrated architecture?

Both approaches have advantages: modular systems can be replaced and optimised building block by building block. Integrated models can preserve information such as prosody and voice characteristics that is lost in text-based intermediate processing. Which architecture is better suited depends on the requirements – there is no one-size-fits-all answer.

What this means for your event

As an event organiser, agency, company or association, you do not need to evaluate model architectures. What matters to you is that your guests can follow the programme in their own language – whether at a congress, an annual general meeting or a public session. Even so, understanding the processing chain helps when you are planning simultaneous interpreting for your event. Because some factors can be influenced in advance:

• The sound: Because everything begins with speech recognition, a clean microphone signal helps determine what arrives in the subsequent steps.
• The programme: Technical terms and proper names that are known in advance can be prepared. At MILA, we set them up before the event.
• The supervision: AI-assisted simultaneous interpreting should not run unattended. With us, a system technician supervises the interpreting on site during the event.

The MILA perspective

For MILA, the architecture of a speech system is a key technical decision. We are exploring how translation quality, real-time capability and local processing can be combined.

Local processing is an integral part of our service: our AI simultaneous interpreting runs locally on our own equipment at the venue, without the cloud. What is spoken does not leave the event.

In doing so, we consider both specialised individual models and new integrated approaches. Because the quality of an AI interpreting system does not come from a single model. It comes from the interplay of the entire processing chain – from the input signal and the preparation of the languages through to on-site supervision.

At the end of the chain, each language is fed on its own channel into the existing interpretation system or conference equipment; on request, we provide receivers and headphones for the guests. You can find details of the technology, workflow and additional services on our Services page. If you are planning a multilingual event, we would be happy to discuss what your programme needs.