Machine Learning Hallucinates?

Intriguing thought.  Helpful explanatory images at the link.

A machine-learning method hallucinates its way to better text translation   by Lauren Hinkel, Massachusetts Institute of Technology

Overview of VALHALLA Architecture for Machine Translation. Left: Training pipeline of VALHALLA. Translation outputs are gathered from two streams of input, either with ground-truth visual tokens z or hallucinated representation zˆ, and optimized on a combination of hallucination, translation and consistency losses.  

As babies, we babble and imitate our way to learning languages. We don't start off reading raw text, which requires fundamental knowledge and understanding about the world, as well as the advanced ability to interpret and infer descriptions and relationships. Rather, humans begin our language journey slowly, by pointing and interacting with our environment, basing our words and perceiving their meaning through the context of the physical and social world. Eventually, we can craft full sentences to communicate complex ideas.

Similarly, when humans begin learning and translating into another language, the incorporation of other sensory information, like multimedia, paired with the new and unfamiliar words, like flashcards with images, improves language acquisition and retention. Then, with enough practice, humans can accurately translate new, unseen sentences in context without the accompanying media; however, imagining a picture based on the original text helps.

This is the basis of a new machine learning model, called VALHALLA, by researchers from MIT, IBM, and the University of California at San Diego, in which a trained neural network sees a source sentence in one language, hallucinates an image of what it looks like, and then uses both to translate into a target language. The team found that their method demonstrates improved accuracy of machine translation over text-only translation. Further, it provided an additional boost for cases with long sentences, under-resourced languages, and instances where part of the source sentence is inaccessible to the machine translator.

As a core task within the AI field of natural language processing (NLP), machine translation is an "eminently practical technology that's being used by millions of people every day," says study co-author Yoon Kim, assistant professor in MIT's Department of Electrical Engineering and Computer Science with affiliations in the Computer Science and Artificial Intelligence Laboratory (CSAIL) and the MIT-IBM Watson AI Lab. With recent, significant advances in deep learning, "there's been an interesting development in how one might use non-text information—for example, images, audio, or other grounding information—to tackle practical tasks involving language," says Kim, because "when humans are performing language processing tasks, we're doing so within a grounded, situated world." The pairing of hallucinated images and text during inference, the team postulated, imitates that process, providing context for improved performance over current state-of-the-art techniques, which utilize text-only data.  ....