Latency-controllable Machine Translation Integrated with Speech Transcription

As civilization advances, the need for fluent communication and understanding across languages becomes more and more crucial. And human prefer instant input from vocal information in the most direct scenario. There are two types of human interpreters: a consecutive interpreter waits until the speaker pauses (usually at sentence boundaries) to start translation, thus doubling the time needed, while a real-time interpreter performs translation concurrently with the speaker's speech, with a delay of just a few seconds. Machine translation (MT), the task of teaching machines to learn to translate automatically across languages, as a result, is an important research area. Real-time translation (also known as simultaneous translation), which translates sentences before they are finished, is useful in many scenarios but is notoriously difficult due to word-order differences and simultaneity requirements. This thesis introduce a very simple yet surprisingly effective wait-k model trained to generate the target sentence concurrently with the source sentence, but always k words behind, for any given k. This framework seamlessly integrates anticipation and translation in a single model with only minor changes to the seq2seq framework. We also present a speech transcription module (also known as ASR, Automatic Speech Recognition) to feed our translation system with incremental text from audio streaming. Moreover, we formulate a new latency metric that addresses deficiencies in previous ones.

Major Advisor: Liang Huang
Committee: Prasad Tadepalli
Committee: Xiaoli Fern
Committee: Lizhong Chen
GCR: Bo Zhao