Driving Value with the Metaverse in the Enterprise

Irving talks some of the work that IBM did with the Virtual World  'Second Life' in the mid 2000's at IBM which we also participated in, a nice look with lots of interesting links.  This also fills me in on lots of later work, such as metaverse located meetings and task interactions.  Could not get our folks interested  Much of that was documented in this blog.  Is the addition can AI 'Enhancement' enough to make the proposition much more interesting?  Will search, examine these for later comment.

A collection of observations, news and resources on the changing nature of innovation, technology, leadership, and other subjects      By Irving Wladawsky-Berger 

Driving Value with the Metaverse in the Enterprise     (much more at the link)

A killer app is an IT application whose value to individuals and companies is relatively simple to explain, its use is fairly intuitive, and the application turns out to be so useful that it can singlehandedly propel the success of a new product or service. Word processing and spreadsheets, for example, were major factors in the widespread adoption of personal computers in the 1980s.

I got interested in 3D immersive environments in the mid-2000s when IBM launched an exploratory initiative on the topic. As part of these efforts, we conducted a number of experiments on Second Life, a platform that allowed its users to develop a variety of virtual world applications. We built a number of workplace applications, including virtual meetings, and online events, and even built a virtual replica of Beijing’s Forbidden City in partnership with the Palace Museum that was opened to the public. But, despite our high expectations, we couldn’t find any compelling enterprise killer apps, and after about three years, the project was dissolved. In the ensuing years, 3D immersive technologies and applications continued to advance, but mostly focused on video-games and related industries.

The past few years have seen a surge of interest in the metaverse and extended reality. “After desktop computing, the consumer internet and the smartphone boom, the consumer-computing industry is past due its Next Big Thing,” wrote The Economist in  “A reality check for the metaverse is coming,” an article in its 2022 year-end issue. “[A]n internet which is still largely flat — based on two-dimensional text, images and video — is ripe for replacement with one that is three-dimensional and immersive.” ... ' 

Mobile Augmented Reality Talent Sources

Agree, this is not common, perhaps in the game space?   Appreciate the lack of connective knowledge.   We spent some time at this in two very different domains.  (Defense and Packaged Goods) 

The talent Needed to adopt mobile AR in industry    in ChiefLeaningOfficer

Chief Learning Officer Network by Elizabeth Loutfi / December 12, 2022

Over the past two decades, mobile and augmented reality technologies have been introduced into the design, assembly, maintenance, quality and logistics sectors to assist job performance.

Mobile AR technology is used to assist training and support job performance during a task. For example, mobile AR was used in training technicians’ maintenance and assembly skills, such as a worker using AR on a tablet to perform tasks with a piece of a machine.

Meanwhile, mobile AR can also support mobile workers in completing job tasks by accessing and processing information during their work and with the minimum help from others. For instance, according to a study on AR in warehouse operations, warehouse workers reduced errors in operations such as receiving, storing, picking and shipping with AR on wearable devices.

Additionally, many pilot studies and field tests revealed the positive results that mobile AR brought to companies and organizations, such as improved job performance, reduced time on tasks and errors, and increased confidence. However, the talent required for learning and development professionals to successfully adopt mobile AR in the workplace remains unknown. 

This article presents four emerging findings from a recent study about the talent learning and development professionals needed to adopt mobile AR in the industry. The participants came from North American companies and organizations, representing industries such as supply chain, biomedical, utility, electronics engineering, manufacturing and AR consulting services.

Instructional design knowledge and skills

Instructional design knowledge and skills are usually required when developing training programs in different modalities, such as face-to-face, online or blended. When adopting mobile AR technology to drive job performance, both industry and consulting service participants recognized the importance of “good instructional designs and learning development capability.”  

Furthermore, instructional design knowledge and skills are interpreted in two types: The first type refers to the ability of developing the content to design effective learning or job performance with AR. A CEO from an AR consulting service suggested that it was necessary “to identify a clear performance objective and to break down” based on the criterion to avoid creating a “lot of content, a lot of media and stuff, and then it’s not well received.” 

The second type refers to the ability of understanding needs and business problems. A learning director from the supply chain industry commented that “most of the consultants are coming to us with technology without understanding the problem.” He viewed that “technology should be the enabler. It should not be the driver of the solution.”

Mobile AR technical skills

Mobile AR technical skills refers to the ability to use a tool or a system to develop AR content and deploy it to end users. It is similar to a content development tool or learning management system in the e-learning industry.

The views about mobile AR technical skills were very mixed. A project manager from an electronics engineering and manufacturing industry thought the current AR tool was very powerful but complex. She suggested spending time learning the AR tool. However, several participants from the industry and AR consulting services viewed the current AR tools as easy to use, just like developing PowerPoint.

Overall, participants agreed that the better the AR authoring tool, the less L&D professionals should worry about AR technical skills. More attention should be focused on design, such as whether to develop content for learning or supporting job performance.   ... 

Augmented Reality with Heads-up Displays

 Some of our earliest AR work was with heads up displays (HUDs) which replicated real maintenence environments.  Here they are dealing with automotive examples, where fast reaction and realist attention would be key.  Good detail here on important concepts.

The Road Ahead for Augmented Reality  By Keith Kirkpatrick

Communications of the ACM, December 2021, Vol. 64 No. 12, Pages 20-22  10.1145/3490317

Automotive head-up displays (HUDs), systems that transparently project critical vehicle information into the driver's field of vision, were developed originally for military aviation use, with the origin of the name stemming from a pilot being able to view information with his or her head positioned "up" and looking forward, rather than positioned "down" to look at the cockpit gauges and instruments. The HUD projects and superimposes data in the pilot's natural field of view (FOV), providing the added benefit of eliminating the pilot's need to refocus when switching between the outside view and the instruments, which can impact reaction time, efficiency, and safety, particularly in combat situations.

In cars, the main concern is distracted driving, or the act of taking the driver's attention away from the road. According to the National Highway Transportation Safety Administration, distracted driving claimed 3,142 lives in 2019, the most recent year for which statistics have been published. Looking away from the road for even five seconds at a speed of 55 mph is the equivalent of driving the length of a football field with one's eyes closed.

As such, the desire to ensure that drivers keep their eyes focused on the road, instead of looking down at the gauges on the dashboard, was the impetus for the development of HUDs suitable for use in production automobiles. The first automotive HUD that was included as original equipment was found on the 1988 Oldsmobile Cutlass Supreme and Pontiac Grand Prix; both were monochromatic, and displayed only a digital readout of the speedometer.

Thanks to the increasing inclusion of a variety of automotive sensors and cameras, advanced driver assistance system (ADAS) features and functions (such as automatic braking, forward collision avoidance, lane-keeping assist, and blind-spot monitoring, among others), and more powerful on-vehicle processors, automakers have been installing HUD units in commercial vehicles that provide more essential driving data, such as speed, engine RPMs, compass heading, directional signal indicators, fuel economy, and other basic information, allowing the driver to concentrate on the road instead of looking down to check the dash or an auxiliary screen.

The technology enabling most types of HUD is based on the use of a processor to generate a digital image of data coming from sensors. These images then are digitally projected from a unit located in the dash of the car onto a mirror or mirrors, which then reflect that image onto either a separate screen located behind the steering wheel, or onto the vehicle's windshield, directly in the driver's forward view. Common projection and display technologies used include liquid crystal display (LCD), liquid crystal on silicon (LCoS), digital micromirror devices (DMDs), and organic light-emitting diodes (OLEDs), which have replaced the cathode ray tube (CRT) systems used in the earliest HUDs, as they suffered from brightness degradation over time.

The HUDs that project the information onto a separate transparent screen are called combiner HUDs; these were popular because the physical space required to install the system was modest, and because the system was fully integrated, OEMs did not need to design a system that accounted for each vehicle's unique windshield angle or position. However, this type of HUD was limited by several factors; namely, the optical viewing path of a combiner HUD is shorter than looking through a windshield, and the driver's eyes must refocus slightly to the shorter visual distance when switching between looking out the windshield and checking the display. Furthermore, there is a practical limit to the size and field of vision (FOV) offered by combiner units; adding mirrors and a larger combiner screen would apper obtrusive and less elegant in a modern vehicle than simply using the windshield as a display surface.   .... '

Army Night-Vision AR

 An AR ability with enhanced low light vision.    Integration of maps.   When we tested plant applications we added blueprint and technical data, but also were sometimes also in low-no light locations.   So could see the application there too.  

The Army's Latest Night-Vision Tech Looks Like Something Out of a Video Game

The Washington Post, Dalvin Brown, May 24, 2021

The U.S. Army is training on a new night-vision system that generates a videogame-like view of a scene. The helmet-mounted Enhanced Night Vision Goggle-Binocular device features thermal imaging and augmented reality (AR) capabilities, incorporating smartphone and gaming-systems technology into traditional night-vision hardware. The goggles have a mode that outlines objects with glowing white light, and an AR overlay that can display maps and navigation data. The system, which also features an intensity tool that allows it to be used in daylight, is a result of years of work to modernize the tools used by the military.  ... " 

LiDAR Test Starts up Again

 Am now finally getting into my LIDAR test,  on a Apple 12 Pro phone for now.   Using the package Polycam, which looks easy to use and is free for the initial testing, then has a monthly fee.  Examples of use here:  https://sketchfab.com/polycam/collections/polycam    And allows you to share scanned models with other packages.    Anyone else doing this?  Lets share a test!

More about how to use LiDAR here.    Below is more background, repeating ... what I placed here earlier to give background.

Did you buy a new iPad Pro? How about a new iPhone 12 Pro? These are the LiDAR apps to try right now, regardless of your background or experience.   By  Brian M Wolfe

The second-generation 11-inch iPad Pro, fourth-generation 12.9-inch iPad Pro, and now the iPhone 12 Pro/Pro Max offer a LiDAR (Light Detection and Ranging) Scanner that will play a huge role in Apple's augmented reality initiatives in the years to come.

Here's more about LiDAR and apps you can use right now on your new iPad Pro to test the technology.

What is LiDAR?

LiDAR uses light to measure distance using invisible pulsating green spectrum lasers. These pulses (which occur thousands of times per minute) measure how long it takes for the light to get back to the sensor. By doing so, it paints a "picture" of the environment in front of the scanner. .... "

Facebook Wants Watch like Controllers for AR Glasses

 Is Facebook aiming to take us long and deep with augmented reality controlled from our watches?    I still think there is something missing to make the idea universal.

Facebook reveals new watch-like controllers for its future AR glasses

The wearable devices will read nerve signals from the brain and deliver tactile feedback through haptic buzzers around the wrist.... 

By Mark Sullivan   in FastCompany

While other tech companies tend to keep their R&D under a blanket of secrecy, Facebook is opening its labs and showing the world how it’s developing its future augmented reality (AR) glasses. The company held the second in a series of its “Inside the Lab” media roundtables on Tuesday and introduced new technology that would enable users to control their AR glasses using their fingers.

AR glasses project light onto the eyes and seem to overlay the real world with digital imagery. As you’re looking through the glasses at a statue, for instance, the glasses might display a label with helpful information right next to it. Many people in tech circles believe AR glasses, in some form, will eventually replace the smartphone as our go-to personal computing device.

In this week’s roundtable, company executives, starting with chief technology officer Mike Schroepfer, showed off wrist wearables that detect the movement of the glasses wearer’s fingers. The wearable uses a technology called electromyography (EMG), which intercepts the electrical signals the brain uses to direct finger movements. It then translates the electrical pulses into digital commands that can control functions of the device functions. ... "

Pin Screens to the Walls

Is it the future of display to have information pinned on the wall.  Or floating in space?

Qualcomm’s new AR ‘Smart Viewer’ lets you pin virtual screens to your walls

Lenovo adopted the design for its ThinkReality glasses

By Adi Robertson@thedextriarchy  in TheVerge

Chip maker Qualcomm has introduced a new reference design for augmented reality glasses: an AR “smart viewer” you can tether to a phone or PC via USB-C. Called the XR1 Smart Viewer, the system is meant to be lightweight and look (sort of) like sunglasses, while also enabling features like hand tracking and spatial awareness. The first glasses based on its design are set for release in mid-2021. .. "

LiDAR Space Scanning, Modeling and Enabling

Some time ago attended a brainstorming session on the use of 'space scanning' to support advertisements and other future AR functionality.   Well the time is now closer, but how close are we to actual functionality?   I am testing for now an IPhone 12 Pro with Apps, and will present more here.   For now here is an article on current Apps that support LiDAR scanning.  IKEA is known to be testing.   Will be testing the idea.   Any thoughts out there of current tests that could be supported? 

Best apps for testing and experiencing LiDAR on your new iPhone 12 Pro  and more here on uses .

Did you buy a new iPad Pro? How about a new iPhone 12 Pro? These are the LiDAR apps to try right now, regardless of your background or experience.   By  Brian M Wolfe

The second-generation 11-inch iPad Pro, fourth-generation 12.9-inch iPad Pro, and now the iPhone 12 Pro/Pro Max offer a LiDAR (Light Detection and Ranging) Scanner that will play a huge role in Apple's augmented reality initiatives in the years to come.

Here's more about LiDAR and apps you can use right now on your new iPad Pro to test the technology.

What is LiDAR?

LiDAR uses light to measure distance using invisible pulsating green spectrum lasers. These pulses (which occur thousands of times per minute) measure how long it takes for the light to get back to the sensor. By doing so, it paints a "picture" of the environment in front of the scanner. .... "

Digital Product Development from Visualization

And forward to product design and development, from visualization. 

Visualization Adds Transparency to Digital Product Development

Fraunhofer-Gesellschaft (Germany),  January 4, 2021

Visualization software developed by researchers at Germany's Fraunhofer Institute for Computer Graphics Research (IGD) and Austria's Linz Center of Mechatronics could embed transparency within digital product development. The Fraunhofer IGD team applied this tool to the development of electric motors, while its Linz partner produced technical data and mathematically modeled all product parameters. The visualization covers all criteria relevant for motor development, and portrays the interactions of individual parameters, relaying what occurs when any value is changed and the impact on other criteria, in real time. Fraunhofer IGD's Lena Cibulski said, "The software is an ideal choice whenever dealing with many design options and a number of incompatible quality criteria that require a compromise to be made."

Inferring Design Plans from Short Clips

 Currently examining this kind of approach.  Phones are getting these kinds of abilities. 

Researchers design AI that can infer whole floor plans from short video clips   By Kyle Wiggers   @Kyle_L_Wiggers

Floor plans are useful for visualizing spaces, planning routes, and communicating architectural designs. A robot entering a new building, for instance, can use a floor plan to quickly sense the overall layout. Creating floor plans typically requires a full walkthrough so 3D sensors and cameras can capture the entirety of a space. But researchers at Facebook, the University of Texas at Austin, and Carnegie Mellon University are exploring an AI technique that leverages visuals and audio to reconstruct a floor plan from a short video clip.  ...  '

Shoe Fitting Augmented Reality

Likely we will commonly see AR type methods for trying on clothes.

Does the Shoe Fit? Try It On with Augmented Reality

The New York Times   By Kate Conger

Retailers are increasingly reliant on augmented reality (AR) technology to help consumers try on products virtually, hoping to recreate the real-world shopping experience during the current pandemic. Many retailers are using AR offerings from Snapchat parent company Snap, which started adding shopping filters in January, and currently offers AR try-ons for luxury brands. Some companies have developed AR applications exclusively for trying on their own products, like sneaker firm Wanna Kicks; others have teamed with retailers to help create AR experiences focused on their items. Snap in June released a technical library of tools to help developers recognize and classify objects, in order to generate AR filters for Snapchat. Retailers using AR hope virtual try-ons can make product exchanges and returns less likely.  ... " 

Improving Amateur CGI with iPhone AR

Quite an interesting thought ... positioning you in  your real world.

iPhone AR tech can improve amateur CGI
CamTrackAR uses iOS' AR know-how to make adding effects easier.

Daniel Cooper, @danielwcooper
Match Moving is the process of anchoring a CGI object inside a real-world space, so that the camera treats it as if it was really there. If you’ve seen anything where a camera flies past a movie’s title like it was a road sign, then you’re familiar with how it looks. The process is commonly used in film and TV, but even as it’s gotten a lot cheaper, it’s still difficult to achieve without rotoscoping. HitFilm developer FXHome thinks that it’s found a way of harnessing the iPhone camera to make Match Moving accessible even for rank amateurs.

CamTrackAR is a new app from the British developer, which uses the iPhone’s superior augmented reality skills to simplify the process.  When you start the app, you need to look around a space until it identifies the floor, at which point you can start placing nulls. Nulls are essentially the anchor points within a space that will help VFX artists later add in the CGI elements to the live action.   ... "

Army Testing AR Goggles for Commanding Dogs

Had not seen the idea before.    For remotely sending commands to dogs.

The Army is Testing AR Goggles for Dogs

 A military dog wearing augmented reality goggles. 

Seattle-based company Command Sight has developed augmented reality devices that can be worn by dogs, through which human handlers can provide visual clues to direct the animal to a specific spot. An often-heard prediction is that augmented reality (AR) could one day become a central tool in our everyday work and play – but it turns out that the technology might not be only suited for humans. 

The U.S. military has unveiled a new project in partnership with Seattle-based company Command Sight, to fit working dogs with AR goggles that would enable soldiers to give orders to the animal at a distance.

Military dogs intervene in tactical operations, patrol, detection and specialized searches. Oftentimes, they can find themselves scouting dangerous areas, looking for explosive devices or materials, for example.

From ZDNet

Better VR Hearing

Adding additional sensors in AR cold lead to specific new application areas.  Spatial Audio could add to maintenance capabilities when linked to machine learning.  Also that Facebook continues to be interested in 'smart glasses' direction.  Perhaps not superhearing though. 

Facebook wants its AR glasses to give wearers superhearing

It could improve VR headset audio, too

By Adi Robertson@thedextriarchy  in TheVerge

 A new Facebook experiment shows how sound could play a major role in augmented reality. Chief scientist Michael Abrash and his team at FRL Research (formerly Facebook Reality Labs) released details today https://about.fb.com/news/2020/09/facebook-reality-labs-research-future-of-audio/  about what the team calls “perceptual superpowers” — AR systems that figure out what you’re trying to hear, then amplify it and dampen background noise. Combined with spatial audio features, the device creates the aural equivalent of a hologram overlay in a pair of glasses.  ... " 

Making the Real World Digital: AR Copy-Paste

Recall having the broad idea being brought up to transfer pieces of a real-world manufacturing plant into a process  model.  First for demonstration, and later even for pasting up a digital twin analytical model.   Why not merge the real and AR world?    With the click of a phone?   Here a startup in beta for something like that.  Will be following.  I see it as even more outside the consumer domain.  Could be a real killer app when put in the right hands.

AR Copy Paste uses AI to transfer real objects into productivity apps
Jeremy Horwitz in VentureBeat

As consumers continue to await affordable augmented reality glasses, the key platforms for AR applications have been smartphones — portable devices that use their screens and cameras to create little windows into a merged real and digital world. Now a developer is using AI-backed phones to pull real-world objects into the digital realm, enabling users to instantly transfer physical items into professional apps, such as Adobe’s Photoshop, Microsoft’s Powerpoint, and Apple’s Keynote.

Currently in an invite-only beta, Cyril Diagne’s AR Copy Paste app can be held up to plants, clothes, books, magazines, and other physical objects, using a seemingly simple interface to snap a photo of each item before transferring it to an identified space in an open document.  ... "

Mirror Arrays Make Augmented Reality More Realistic

Augmented reality, adding realism.  Most AR systems today provide a 'gamelike' view of artificial objects.   A solution.

Mirror Arrays Make Augmented Reality More Realistic
IEEE Spectrum
By Michelle Hampson

Researchers at Stanford University have developed a compact augmented reality (AR) system that uses an array of miniature mirrors to create virtual images that appear more "solid" in front of real-world objects. The miniature mirrors in Stanford's system change positions tens of thousands of times per second, enhancing the system's occlusion. The system combines virtual projection and light-blocking abilities into one element, relying the dense array of miniature mirrors to switch between a see-through state and a reflective state. It computes the optimal arrangement for the mirrors and adjusts accordingly. Stanford’s Brooke Krajancich said the system uses a lot of computing power, and may require more power than other AR systems.  .. ' 

Innovative AR in Automobiles

Some quite innovative AR systems being used to compete among upscale automobiles.   But ultimately distracting? 

With In-Car AR, Drivers Get a New View of the Road Ahead  By  Alex Davies in Wired
Navigation screens with augmented reality are the latest in an arms race among upscale auto makers like Mercedes and Cadillac.

"... The addition of augmented reality to navigation is the latest in the unending arms race among upscale auto makers hoping to lure wealthy customers with ever fancier tech. Cadillac just announced that the 2021 Escalade will offer a similar guidance system. Hyundai’s luxury brand, Genesis, is debuting the technology in Korea on the GV80 SUV. And like many such novelties—massage seats, perfume systems, gesture controls—it’s a very cool idea with a lot of room for improvement.  ... " 

On Realism Considered

Not sure what to fully make of this, but recently have listened to some podcast conversations about how we were all in a Matrix-like simulation of sorts, and this seems to be in a similar vein.    Posting it not because I think this is true, but because I think the whole concept can open your eyes to how complex  things can be simulated,  and thus enhanced and delivered. 

I recall talking this to (non engineering) management about this at times, and they seemed perplexed.  But then I pointed out there was a long tradition in engineering and their enterprise in doing exactly that.  Even today's Machine Learning can be seen as a means of simulating the world to understand it.   So here in the spirit of all that:

From the Edge

Realism Is False
A Conversation with Donald D. Hoffman [1.27.20]

. . . I want to propose that realism is false, and what we're seeing is more like a user interface or a virtual reality headset. Think about a virtual reality game of tennis. You're playing VR tennis with a friend, you both have your headset and body suits on, you see your friend's avatar on a tennis court and you start playing. Your friend hits the tennis ball to you, and you hit the same tennis ball back to your friend, but is your friend seeing exactly the same tennis ball that you're seeing? Well, of course not. There's no public tennis ball. You have some photons being sprayed to your eye by your headset, and those photons are causing your visual system to create your own perception of what you would call a green tennis ball. Your friend has a headset on, which is spraying photons to his eye, and his visual system is creating his own green tennis ball perception.

It turns out that both of those perceptions are coordinated by something else, namely a supercomputer that's sending the photons to both headsets, causing both headsets to work in coordination. . . .

All the things that we would do to say that objects really exist even when they're not perceived hold here in virtual reality. . . . That doesn't mean that the tennis ball exists and has any physical properties when it's not perceived; it just means that there is some objective reality.   ....   " 

DONALD D. HOFFMAN is a full professor of cognitive science at the University of California, Irvine. He is the author, most recently, of The Case Against Reality: Why Evolution Hid the Truth from Our Eyes. Donald D. Hoffman's Edge Bio Page

AR Continues Struggle in Retail

An area we spent lots of early time in.  Good overview article of current work.

Despite advancements, AR struggles to take off in retail   By Anna Hensel in Modern Retail
This is the third part in a series by Modern Retail about the technologies that were going to change retail — and where they are now. See our previous stories, about the future of robotics, and the truth about RFID.

Three years ago, retailers began spending money on AR.

In 2017, Williams-Sonoma acquired Outward, an AR and 3D imaging startup for $112 million. The following year, Ulta Beauty also acquired an AR startup for an undisclosed amount, as did L’Oreal. Retailers like Walmart and Nike acquired other startups that specialized in virtual reality and computer vision, but also with the goal of using them to create experiences that incorporated some type of virtual reality component.

That ushered in a stampede of retailers eager to experiment with AR. Michael Kors and Warby Parker were some of the first companies to experiment with running AR in Facebook ads. Macy’s launched AR experiences for both furniture and beauty shoppers.

AR right now is big among two categories: beauty or furniture. Sephora, Ulta Beauty, Wayfair, Williams-Sonoma and Ikea have all launched virtual try-on experiences that use AR. Retailers in apparel have also experimented with adding these types of tools — the most recent being Asos — many of those efforts remain firmly in the experimentation phase. Because of the nature of the technology, it’s really those three industries where it can make a real difference — especially in fashion.

But even for the retailers who have invested heavily in AR by building their own virtual try on experiences, they haven’t been able to point definitively to how much their AR tools have resulted in a sales increase.

“There’s still some hyping [around AR], no doubt,” said Andrew Lispman, e-commerce analyst at eMarketer. “But I think the reality has set in when you see the initial use cases…are mostly edge cases. They may work well for their category, but it’s not a use case that extends between all or even most categories.”   ...." 

Will Augmented Reality Change Everything we See?

From the Penn Alumni Mag, a broad, descriptive and mostly academic oriented view of the world of augmented reality.  Below the intro and at the link more...

Augmenting Reality
Will augmented reality change everything we see? A growing number of Penn alumni, staff, and faculty think so. And even as they bump up against its challenges and limitations, they’re still committed to pulling AR further into our lives.

By Molly Petrilla | Illustration by Roman Klonek

Sidebar: Need a HoloLens? Try the Library

I’m killing scorpions inside Stephen Lane’s office.

They skitter out of openings in the wall, crawling up and down, sometimes even jumping right at me. I try zapping them away but sometimes I miss, blasting holes around Lane’s door and exposing white pipes and other slivers of the building’s bones.

Lane doesn’t seem to mind. In fact, he’s laughing. The computer and information science professor who was just sitting behind his desk, earnestly answering questions about the future of technology, has transformed into a jazzed-up gamer who’s excited to share his toy with a novice.

“Are you getting them?” he asks as scorpions dance around us. “Just keep firing!”

Even as giant predatory arachnids are scurrying in front of me, and even as my fingers trigger laser beams to destroy them, I’m still inside Lane’s real office. If I turn around, I can see his dark wood desk with its stacks of papers and cup of coffee. I can see his crowded bookshelves and his whiteboard. I can see Lane himself—plaid shirt, bristly mustache, belt with colorful little sailboats. But thanks to the Microsoft HoloLens I’m wearing—a headset that wraps around my forehead and hangs down over my eyes, resting on the bridge of my nose—the game is with us too, turning a plain white wall into a scorpion lair.

And that’s augmented reality.

RoboRaid is a simple introduction for the uninitiated. “It’s a game, but it really shows off some of the capabilities,” Lane says just before passing me the HoloLens.

AR isn’t about fully losing yourself in an imaginary world the way virtual reality is. It’s about enhancing your actual surroundings with computer-generated images and objects. At its most ambitious, AR can require programming expertise and a $3,500 headset like the HoloLens. But it can also be as simple as pulling out your smartphone and turning yourself into a bunny with an Instagram filter.

On that morning inside Lane’s office, I’m not the only one at Penn testing out augmented reality’s abilities and discovering its limits. Across campus and well beyond, a number of faculty, alumni, staff, and students are all focused on applying AR to their fields. You’ll find them inside operating rooms and classrooms, heading up teams at Google and the New York Times, and working for a leading AR headset company.

They’re all convinced that augmented reality can change our world—that it will change our world—in every area from medicine and education to research, retail, and entertainment. But they’re also starting to wonder: What will it take to get past the tipping point, and what might life be like after we do? ... "