Can AI translate emotions? What AI interpreters can do today

“That’s just great.” A short sentence, but depending on the intonation it can express genuine enthusiasm or the exact opposite. People often recognise such differences intuitively. For an AI interpreter, however, emotions are a particular challenge. In this article, we show how conventional speech translation works, what research is currently focusing on and where the limits lie.

Language conveys more than information

Listeners pay attention to more than just words. Pitch, speaking pace, pauses and the situational context carry part of the meaning. Irony, enthusiasm or restraint are often found precisely in this additional layer.

For AI systems, this layer of meaning is hard to grasp. It is contained in the spoken signal, but not necessarily in the words themselves.

How AI interpreters conventionally process speech

Many AI interpreting systems work with multi-stage processing:

• Speech recognition: A model recognises the spoken content.
• Translation: The recognised text is rendered into the target language.
• Speech synthesis: An artificial voice reproduces the translation.

This approach is technically flexible, but it has one weakness: when speech is converted into text, information about emotional expression can be lost. What is no longer in the text cannot be reproduced by the speech synthesis at the end.

Expressive Speech-to-Speech Translation: carrying emotions across

One important field of research is so-called Expressive Speech-to-Speech Translation. The aim is to convey not only words, but as far as possible also characteristics of the original manner of speaking – such as stress, rhythm, speaking pace and features of emotional expression.

Modern audio models are increasingly able to process information directly from speech signals. This opens up possibilities that go beyond conventional text-based translation processes.

Recognising emotions is not the same as understanding them

One crucial limitation remains. AI can analyse certain acoustic features and derive probabilities for emotional states from them. However, this does not mean that it reliably recognises a person’s actual feelings.

A loud voice can signal enthusiasm, anger – or simply poor microphone placement. We explain in detail why sound is the starting point of every translation in our article AI interpreting and audio technology: why everything starts at the microphone.

In addition, cultural differences also influence how emotions are expressed.

The risk of artificial interpretation

An emotionally enriched translation can become problematic if the system adds feelings that the speaker never intended to express.

The development of such systems must therefore carefully distinguish between authentic transfer and artificial interpretation.

The MILA perspective

For us, the transfer of emotion in speech is one of the most exciting areas of research in modern audio AI. The future of professional interpreting systems lies not only in the accurate translation of spoken words, but also in conveying human communication as authentically as possible.

Until then, it remains essential to assess available capabilities realistically and to be transparent about technological limits. At MILA, we therefore focus on what can be planned for events: we set up technical terms and proper names from the programme in advance, the AI simultaneous interpreting runs locally on our own equipment at the venue, and a system technician supervises the interpreting on site.

If you would like to know how this could work for your event, you will find full details on equipment and procedures on our Services page.