Exploring Dreams at MIT Media Lab Fluid Interfaces Group

Way back I worked on some related topics, just happened on this.   A new interface? A way into the brain?  Anyone know other groups doing related things?

Youtube presentation: https://www.youtube.com/watch?v=fW0yxZ7Gp8E

DayDreamNight: A lucid evening at MIT exploring dreams

MIT Media Lab

950 views  Feb 1, 2023  #daydreamer #dreams #cultivatingcreativity

DayDreamNight was a gathering for the MIT community to explore the collaboration between CAST Visiting Artist Carsten Höller and the MIT Media Lab’s Fluid Interfaces group.

The selected projects open up avenues for understanding and shifting our sleep, dreams, and daydreams. 

More information at: 

Sleep & Dream theme page: https://mitmedialab.info/SleepDream 

Dream Hotel with Carsten Höller: https://mitmedialab.info/DreamHotel 

MIT Museum Studio: https://mitmedialab.info/mitmuseumstudio 

Flying Dreams: https://mitmedialab.info/FlyingDreams 

#dreams #daydreamer #daydreamnight#cultivatingcreativity

Creating Text Out of Thin Air

New means of interaction with data

Creating Text Out of Thin Air

By IEEE Spectrum

Researchers at China's Zhejiang University have developed and tested a smartwatch application in which users write text in mid-air with a finger, and the text is captured by the app.

"The goal of AirText is to infer the texts written by the fingertip in the air, using only the IMU (inertial measurement unit) readings, for example the accelerometers, gyroscopes, and magnetometer, from the smartwatch on the wrist as input," said Zhejiang's Wei Dong.

The researchers' Leap Motion program fed wrist- and hand-motion data to an artificial intelligence model to correlate wrist and fingertip movements in spelling out letters

Volunteers used AirText to realize an average typing speed of 8.1 words per minute, and an average word error rate of between 3.6% and 11.2%.   .... ' 

From IEEE Spectrum     

Bayesian Keyboards

Nice piece of how keyboards are made smart.

A Look Inside the Bayesian Keyboard

A visual tour of the data-driven features that make modern smartphone keyboards smart

Daniel Buschek  in TowardsDatascience

What you type is what you get? Not with modern touch keyboards. This article visually explains four features at the heart of your smartphone’s keyboard — including personalisation, auto-correction, and word prediction. Based on material I created for my “Intelligent User Interfaces” lecture, we examine the inner workings of these features in our daily typing, and conclude with takeaways for inspiring, evaluating, and critically reflecting on data-driven and “intelligent” user interfaces.  ... '

Reimplementing Software Interfaces is Fair Use

 Somewhat surprising,  and Beyond the US?   Good overview of the technical implications.

Reimplementing Software Interfaces Is Fair Use

By Pamela Samuelson  Communications of the ACM, July 2021, Vol. 64 No. 7, Pages 24-26

10.1145/3466607

A long-standing, generally accepted norm in the computing field distinguishes between software interfaces and implementations: Programmers should have to write their own implementing code, but they should be free to reimplement other developers' program interfaces. This norm, of which Sun Microsystems, the developer of Java, was once the software industry's foremost proponent, is now the law of the land in the U.S. after the Supreme Court's decision in Google Inc. v. Oracle America, Inc., which overturned a lower court ruling that reimplementing an interface infringed copyright.

The Supreme Court took Google's appeal on two issues. One was whether program interfaces are protectable by copyright law. The Supreme Court declined to decide that issue, even though many amicus curiae (friend of the court) briefs filed by software developers, organizations such as the Electronic Frontier Foundation, the Center for Democracy & Technology, and the Computer & Communications Industry Association, as well as numerous intellectual property scholars, supported Google's argument that program interfaces are uncopyrightable.

A second issue was whether Google's reimplementation of 11,500 declarations from 37 Java Application Program Interface (API) packages in its Android smartphone platform was fair use or infringement. Although a jury rendered a verdict in favor of Google's fair use defense after a two-week trial, the Court of Appeals for the Federal Circuit (CAFC) overturned this verdict. The CAFC concluded that no reasonable jury could have found Google's appropriation of that many lines of computer code was fair use. This is the ruling that Supreme Court's decision reversed.

The penultimate sentence in Justice Breyer's opinion for the 6-2 majority succinctly states the Court's conclusion: "where Google reimplemented a user interface, taking only what was needed to allow users [that is, programmers] to put their accrued talents to work in a new and transformative program, Google's copying of the Sun Java API was a fair use of that material as a matter of law."

After explaining Oracle's claims against Google, this column reviews the Court's reasons for rejecting Oracle's arguments on the fair use issue.  ... ' 

The Ethics of Dark Patterns

A look at 'Dark Patterns',  had not heard the term.  Linking to ACM look at ethics by people that  build such interfaces.

Dark Patterns: Past, Present, and Future in ACMQueue
The evolution of tricky user interfaces
Arvind Narayanan, Arunesh Mathur, Marshini Chetty, and Mihir Kshirsagar

Dark patterns are user interfaces that benefit an online service by leading users into making decisions they might not otherwise make. Some dark patterns deceive users while others covertly manipulate or coerce them into choices that are not in their best interests. A few egregious examples have led to public backlash recently: TurboTax hid its U.S. government-mandated free tax-file program for low-income users on its website to get them to use its paid program;9 Facebook asked users to enter phone numbers for two-factor authentication but then used those numbers to serve targeted ads;31 Match.com knowingly let scammers generate fake messages of interest in its online dating app to get users to sign up for its paid service.13 Many dark patterns have been adopted on a large scale across the web. Figure 1 shows a deceptive countdown timer dark pattern on JustFab. The advertised offer remains valid even after the timer expires. This pattern is a common tactic—a recent study found such deceptive countdown timers on 140 shopping websites. ... "

Sprayable User Interfaces

We saw this demonstrated, I believe at the MIT Media Lab,when we visited as a corporate sponsor,  a long time ago.   Apparent the work is still ongoing.

Sprayable User Interfaces
MIT News
Rachel Gordon

Researchers at the Massachusetts Institute of Technology's Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a technique for spraying on user interfaces. The SprayableTech system automatically generates cardboard stencils of artwork users create in a three-dimensional editor, which can be used to apply the art to surfaces by airbrushing on functional inks. The user can add sensors and displays that control various appliances, and a microcontroller is attached to link the interface to the board that runs the code. The researchers tested SprayableTech on multiple items, including a musical interface on a concrete pillar, an interactive sofa connected to a TV, and a wall display for controlling lighting levels. CSAIL's Michael Wessely said, "We view this as a tool that will allow humans to interact with and use their environment in newfound ways.”  ... '

Replacing Screen by Realities?

Will every screen be replaced by a virtual reality experience?

In Smithsonian:
Alex Kipman on how it's "inevitable" that devices like the HoloLens will replace every screen. Reading a Clive Thompson interview with the HoloLens creator in his office at Microsoft's HQ, I get a distinct sense of deja vu, as he outlines much the same vision as when I met him in that very same room back in 2017. While a cynic might say that Kipman is a brilliant marketer who excels at staying on message, I can attest to the fact that when you meet him in person it is difficult to doubt the sincerity of his belief in that vision, and in the compelling picture he paints for how the technology will transform our lives. And I can also relate to Thompson in being impressed by how fun and empowering the experience of borrowing knowledge from a holographic guide to perform a task for which you're in no way qualified can feel .... " 

Combining Touchless and Tactile Interaction

Quite unusual combination, can it greatly improve interaction in new ways?

Intuitive in the Virtual Reality
Helmholtz-Zentrum Dresden-Rossendorf
By Simon Schmitt, Press Release

Researchers at Helmholtz-Zentrum Dresden-Rossendorf in Germany and Johannes Kepler University Linz (JKU) in Austria have developed an electronic sensor that can simultaneously process both touchless and tactile stimuli. When applied to human skin, the sensor could provide a seamless interactive platform for virtual and augmented reality applications. The researchers said the sensor was conceived as an electronic counterpart to human skin, and could “massively amplify” the interaction of people and machines. Said JKU's Martin Kaltenbrunner, "A better interface between humans and machines can be built on this foundation."  .... '

Pets and Machine Learning Interactions

In Pete Warden's Blog, interesting views.   Have seen some of that in my own menagerie of chatbots and responsive assistants.    But I think ultimately we will want assistant than amusement.  Machine learning is collaborative in the sense that it solves narrow problems.  So does a 'Push Button' model of tech.  So will a 'pet model' be attentive and responsive?  A neighbor has a guide dog, which has been trained to be more attentive and responsive, rather than pet.   Seems more the model we will see.

Why ML interfaces will be more like pets than machines

When I talk to people about what’s happening in deep learning, I often find it hard to get across why I’m so excited. If you look at a lot of the examples in isolation, they just seem like incremental progress over existing features, like better search for photos or smarter email auto-replies. Those are great of course, but what strikes me when I look ahead is how the new capabilities build on each other as they’re combined together. I believe that they will totally change the way we interact with technology, moving from the push-button model we’ve had since the industrial revolution to something that’s more like a collaboration with our tools. It’s not a perfect analogy, but the most useful parallel I can think of is how our relationship with pets differs from our interactions with machines.

To make what I’m saying more concrete, imagine a completely made-up device for helping around the house (I have no idea if anyone’s building something like this, so don’t take it as any kind of prediction, but I’d love one if anybody does get round to it!). It’s a small indoors drone that assists with the housework, with cleaning attachments and a grabbing arm. I’ve used some advanced rendering technology to visualize a mockup below:

Brain Machine Interface

The idea has been around for years, now approaching reality.  In Wired: 

 " .... 2017 has been a coming-out year for the Brain-Machine Interface (BMI), a technology that attempts to channel the mysterious contents of the two-and-a-half-pound glop inside our skulls to the machines that are increasingly central to our existence. The idea has been popped out of science fiction and into venture capital circles faster than the speed of a signal moving through a neuron. Facebook, Elon Musk, and other richly funded contenders, such as former Braintree founder Bryan Johnson, have talked seriously about silicon implants that would not only merge us with our computers, but also supercharge our intelligence. But CTRL-Labs, which comes with both tech bona fides and an all-star neuroscience advisory board, bypasses the incredibly complicated tangle of connections inside the cranium and dispenses with the necessity of breaking the skin or the skull to insert a chip—the Big Ask of BMI. Instead, the company is concentrating on the rich set of signals controlling movement that travel through the spinal column, which is the nervous system’s low-hanging fruit. .... " 

Essence of Voice Interfaces

Very good piece here, what is the difference and essence of having voice interfaces?  Or even interfaces that are partially voice?  This is not necessarily AI but it is closer to cognitive than we had before.  Remember too, that voice more strongly implies a conversation is occurring, which embeds, context, memory and modeling who/what you are talking to.  None of this has been perfected.   Clear from my interaction with Echos and Google Home during the past few years.

“Alexa, Understand Me”  In Technology Review  by George Anders
Voice-based AI devices aren’t just jukeboxes with attitude. They could become the primary way we interact with our machines.

On August 31, 2012, four Amazon engineers filed the fundamental patent for what ultimately became Alexa, an artificial intelligence system designed to engage with one of the world’s biggest and most tangled data sets: human speech. The engineers needed just 11 words and a simple diagram to describe how it would work. A male user in a quiet room says: “Please play ‘Let It Be,’ by the Beatles.” A small tabletop machine replies: “No problem, John,” and begins playing the requested song.

From that modest start, voice-based AI for the home has become a big business for Amazon and, increasingly, a strategic battleground with its technology rivals. Google, Apple, Samsung, and Microsoft are each putting thousands of researchers and business specialists to work trying to create irresistible versions of easy-to-use devices that we can talk with. “Until now, all of us have bent to accommodate tech, in terms of typing, tapping, or swiping. Now the new user interfaces are bending to us,” observes Ahmed Bouzid, the chief executive officer of Witlingo, which builds voice-driven apps of all sorts for banks, universities, law firms, and others.  .... " 

VR Transforming the Workplace?

Niche or everywhere?  Still think this will take some time before it is common in the workplace. A typical machine interface?   AR more likely.

Why Virtual Reality Will Transform a Workplace Near You   By Logan Kugler 

Communications of the ACM, Vol. 60 No. 8, Pages 15-17

" ... VR and AR companies see the technology as the natural evolution of popular collaboration apps; instead of desktop interfaces or pinging phone apps, however, they envision a world in which we collaborate with coworkers and peers in highly realistic virtual or virtually augmented environments—a more immersive, versatile, and natural way to get business done.

This is no pipe dream like the VR of decades past. Thanks to the commercial viability of popular systems and apps, ambitious VR/AR products are already changing how professionals work, train, and cooperate. Company reps use the technologies to better train for customer-service interactions and to troubleshoot issues faster in real time using digital models. Manufacturers rely on the technology to better collaborate on the design and maintenance of components, and hospital systems employ VR and AR to remotely train doctors faster, less expensively, and more effectively.

This is just the beginning. Your next meeting might take place in a VR environment that makes it easier to identify who's speaking and how others feel about your ideas (if Peter Diamandis' company High Fidelity, which creates "open source software for creating, hosting, and exploring shared VR experiences," has its way). Entire collaboration exercises or corporate retreats might be hosted in virtual environments (courtesy of technology like that offered by AltspaceVR, a virtual reality software company building a new communication platform already used by people in more than 150 countries)   .... " 

Bots as the Next User Interface

Good thoughts on thinking of coneversation as iterface.   Is a challenge to do it well.   In Adage:

SXSW: 4 Best Practices to Make Bots the Next Big User Interface
By Nat Ives

People spend more and more time on their smartphones, but with such a limited set of apps that it's becoming harder and harder for newcomers to reach them.

Chatbots are the solution, two proponents told a crowded ballroom at SXSW on Saturday, day two of the conference. That's because they play out on one of the entrenched sets of apps that people already chronically use, they argued: messaging apps.

"We believe that conversations will become the new user interface," said Laura Newton, a product manager at the youth-oriented messaging app Kik.   .... "

Surface Dial Examined

I have been a long time experimenter with alternative interfaces.  From mice, to feet driven mice, to touch pens, speech and neural interaction and more.  So I was interested when I saw the 'Surface Dial', being used for another channel of interaction on the Microsoft Surface.  There are other similar devices out there as well.  Most important, what software and other hardware works with the Dial? How can this be used efficiently with other interactions, like voice and mice and touch?  What does it mean for an integrated assistant like Cortana?  It could lead to the ultimate need for 'hands free' interaction, since it engages another hand.   Verge talks software and the Dial system.

Neural Brain to Text Progress

Remember this topic being brought up for a long time, and often claims that is was close to happening.  Have now been involved in the spoken interface for some time, a reliable thought interface sounds very appealing.   Considerable detail in the full article about the challenge.

Front. Neurosci., 27 September 2016  
Automatic Speech Recognition from Neural Signals: A Focused Review

Christian Herff* and Tanja Schultz
Cognitive Systems Lab, Department for Mathematics and Computer Science, University of Bremen, Bremen, Germany

Speech interfaces have become widely accepted and are nowadays integrated in various real-life applications and devices. They have become a part of our daily life. However, speech interfaces presume the ability to produce intelligible speech, which might be impossible due to either loud environments, bothering bystanders or incapabilities to produce speech (i.e., patients suffering from locked-in syndrome). For these reasons it would be highly desirable to not speak but to simply envision oneself to say words or sentences. Interfaces based on imagined speech would enable fast and natural communication without the need for audible speech and would give a voice to otherwise mute people.

 This focused review analyzes the potential of different brain imaging techniques to recognize speech from neural signals by applying Automatic Speech Recognition technology. We argue that modalities based on metabolic processes, such as functional Near Infrared Spectroscopy and functional Magnetic Resonance Imaging, are less suited for Automatic Speech Recognition from neural signals due to low temporal resolution but are very useful for the investigation of the underlying neural mechanisms involved in speech processes. In contrast, electrophysiologic activity is fast enough to capture speech processes and is therefor better suited for ASR. Our experimental results indicate the potential of these signals for speech recognition from neural data with a focus on invasively measured brain activity (electrocorticography). As a first example of Automatic Speech Recognition techniques used from neural signals, we discuss the Brain-to-text system. ... " 

No UI is the New UI

A novel look at conversational, messenger style interfaces.   Implies that the US can well understand context and even intent.  What are the implications of error?

No UI is the New UI
On the rise of UI-less apps and why you should care about them as a designer.

If you don’t know about these apps, what make them special is that they don’t use a traditional UI as a mean of interaction. Instead, the entire app revolves around a single messaging screen. These are called ‘Invisible’ and ‘Conversational’ apps, and since my initial post, a slew of similar apps came to market. Even as of writing this, Facebook is releasing M, a personal assistant that’s integrated with Messenger to help you do about anything. ... " 

Augmented Reality Projector Interfaces

In Computing Now   An example of new approaches to collaborative interface.  Haptic (touch) included.

Spatial User Interfaces for Large-Scale Projector-Based Augmented Reality
Michael R. Marner, University of South Australia
Ross T. Smith, University of South Australia
James A. Walsh, University of South Australia
Bruce H. Thomas, University of South Australia

Spatial augmented reality environments deal with nonplanar projection surfaces, physical objects that provide natural passive haptic feedback, and systems that naturally support collaboration between users. ....   Full article pdf

Research: Hand Written Notes Improve Recall

Interesting take in the Chronical for Higher Education.   " ... Distractions posed by laptops in the classroom have been a common concern, but new research suggests that even if laptops are used strictly to take notes, typing notes hinders students’ academic performance compared with writing notes on paper with a pen or pencil. ... "   Though I doubt that we are heading back to that.

Moving from Simulation to the Holodeck

Looking at beginnings of the Holodeck.  How does this related to virtual reality in general?  Again I think this should always be connected to virtual reality.  In general simulation is underused in computing technology.  Consider different levels of reality:  Simulation: Augmented Reality: Virtual Reality.  To Holodeck.   How should these be intertwined to provide the greatest computing leverage?

It was pointed out to me that some may not know of the concept of the Holodeck.

Touchscreen Design

New ingenious touchscreen user interfaces.  Suitable for navigation and other tasks that support and augment  driving?