Towards 'Reality' OS

Retagging our Reality:

In Engadget: 

By Mariella Moon|@mariella_moon|December 2, 2022 8:05 AM

Apple has internally changed the name of its upcoming mixed reality headset's accompanying software from "realityOS" to "xrOS," according to Bloomberg's Mark Gurman. As the reporter notes, the new name better represents the software's capabilities: "XR," after all, stands for extended reality, and the headset is expected to have both augmented and virtual reality features. 

In addition to the internal name change, Gurman says a shell corporation named Deep Dive LLC has also filed a trademark for the brand "xrOS" in the US and in other countries, including ones in the European Union and in Asia, the UK, Australia, Mexico, Ukraine, Japan and Canada. In its application, Deep Dive wrote that it's applying for a trademark for "head-mounted displays" and devices that provide "virtual reality and augmented reality experiences." Apple hasn't confirmed whether it's behind this filing. 

Earlier this year, though, Vox Media product manager Parker Ortolani found a patent application for "realityOS" filed by a shell company called Realityo Systems LLC. Bloomberg also reported back in August that yet another shell company with a different name filed applications for "Reality One," "Reality Pro" and "Reality Processor."  ... '   '

Hints at Apple Mixed Reality Efforts

Apple has done innovation quite well,  Look forward to their Mixed Reality efforts. 

Apple Might Use ‘Reality’ Naming Convention for Its Mixed Reality Devices

By Josh Norem on August 30, 2022 at 12:15 pm  in ExtremeTech

It’s been previously reported that Apple has been working on a mixed reality device for over seven years now. Incredibly, we still have no firm launch plans or any definitive details about the mysterious device. However, recent trademark filings by law firms allegedly associated with Apple give us a clue about possible naming conventions.

Previously it had been reported Apple was working on an operating system for the clandestine headset named Reality OS (rOS). As it turns out, the rest of the device’s parts might have similar branding. Trademark applications have been filed in various countries for the names Reality One, Reality Pro, and Reality Processor, Bloomberg reports. This seems to indicate two models for Apple’s upcoming headset; the base model One and a Pro version. Whatever custom silicon it puts inside will inevitably be labeled a Reality processor. Because TSMC hasn’t spun up its 3nm fab just yet, it’ll likely be a 5nm part that’s based on the M2 design. It will reportedly include 16GB of memory and a second chip to handle graphics. .... ' 

Mixed Reality in China

Considerable piece in the CACM, below the intro.

The Present and Future of Mixed Reality in China   in CACM

By Guofeng Zhang, Xiaowei Zhou, Feng Tian, Hongbin Zha, Yongtian Wang, Hujun Bao

Communications of the ACM, November 2021, Vol. 64 No. 11, Pages 64-69    10.1145/3481619

Virtual reality (VR) technology can be used to generate a lifelike virtual world, allowing users to roam freely in this virtual world and interact with it. Augmented reality (AR) is a technology that superimposes and fuses virtual scenery or information with the real physical environment, and interactively presents it in front of users, so that the virtuality and reality share the same space. Mixed reality (MR) is a further development of VR and AR. Its concept was proposed by Paul Milgram and Fumio Kishino in 1994, who described the entire continuum from reality to virtuality.14

VR/AR research in China began in the late 1990s. In 2002, the 973 project (National Basic Research Program) "Fundamental Theories, Algorithms of Virtual Reality and Its Implementation" was established, with Hujun Bao from Zhejiang University as the chief scientist. In 2009, the project received continuous support, and the focus was shifted to AR and MR. The research of these two projects lasted for 10 years. Zhejiang University, the Institute of Software and the Institute of Automation of the Chinese Academy of Sciences, Beijing Institute of Technology (BIT), Beihang University, Tsinghua University, and other institutions participated in these two projects. The projects made many world-class research advances in VR/AR related areas such as automatic camera tracking, efficient modeling, real-time rendering, multichannel human-computer interaction, VR/AR headset display, 3D registration, as well as VR/AR engines and software platforms. The projects also trained a group of outstanding VR/AR researchers, which significantly boosted VR/AR research in China.  .... ' 

Microsoft Does Mesh Presence and Shared Experiences

Microsoft seem to finally be bringing out AR/Telepresence/Mixed Reality for broader applications.  Much more at the link.  With offers to developers and some existing applications at the link.    Worth a look.  Have now seen several interesting past demonstrations, which lacked real usefulness.  

Introducing Microsoft Mesh

Microsoft Mesh enables presence and shared experiences from anywhere – on any device – through mixed reality applications. ... " 

Apple Doing Mixed Reality

 A bit of a surprise, not much in the space as yet, but certainly, a place where their AI will be quite useful. Business oriented or consumer? 

Apple mixed-reality headset will reportedly launch in ‘the next several months’  By Alex Blake  in DigitalTrends

Rumors that Apple is working on a top-secret mixed-reality headset have exploded in recent months. Excitingly, they just got another boost from reputable reporter Mark Gurman, who claims Apple will unveil the device at an in-person event within “the next several months.”

In a newsletter discussing Apple’s future Worldwide Developers Conference (WWDC) events, Gurman wrote: “Sometime in the next several months, the company is poised to announce a mixed-reality headset, its first major new device since 2015. If possible, Apple won’t want to make such a critical announcement at an online event. It wants employees, the media, its partners, and developers in the room.”

Tantalizingly, WWDC itself falls within that date range, as it is due to begin on June 7 this year. However, before we all get too excited, this year’s WWDC event is set to be digital-only, which seemingly rules it out for the mixed-reality headset reveal.  ... " 

Korean Mixed Reality Glasses

Another attempt at more generalized smart glasses, here aimed at mixed reality in particular.  Casual everyday wearable, or context specific wearable for the job. linked closely to mobile.

Nreal Light mixed reality glasses launch in Korea with the Galaxy Note 20
LG Uplus is bundling the Nreal Light with a Samsung Galaxy Note 20 or an LG Velvet.

By Richard Lai, @richardlai in Engadget

After successfully fighting off Magic Leap’s claim that it stole trade secrets, Nreal is finally able to launch its Light mixed reality glasses into the consumer market. Starting today, folks in Korea can pre-order the Light — locally rebranded as “U+ Real Glass” — as part of a mobile phone plan on the LG Uplus network, so long as you pick the Samsung Galaxy Note 20 or the LG Velvet as your handset. That way you can buy the Light at a subsidized price of 349,500 won (about $295). You can also purchase the glasses separately for 699,000 won (about $590) at an LG Uplus store from August 21st, if you’d rather use them with other phones. ... "

HoloLens2 Available and Shipping Now

Considerable overview piece from Microsoft.  They are now hooking it to a number of capabilities such as Dynamics and Azure driven Edge Computing.    Impressive. If significant mixed reality applications are out there, they could well be the leaders.

The making of the HoloLens 2: How advanced AI built Microsoft’s vision for ubiquitous computing  By John Roach

Video demonstrations at the link.
Business apps available now

Pilot and deploy quickly with a full suite of Dynamics 365 mixed reality applications  
Create spatially aware holograms.  Use the world as your canvas
Build cross-platform, spatially aware mixed reality experiences.
HoloLens 2   Learn more about the ultimate Intelligent edge device.
Azure   Explore services and devices for mixed reality development.
Dynamics 365 Get started on day one with business-ready applications.

Redmond, Washington – The first time people don the new HoloLens 2 on their heads, the device automatically gets to know them: It measures everything from the precise shape of their hands to the exact distance between their eyes.

The artificial intelligence research and development that enabled those capabilities “was astonishingly complicated” but essential to making the experience of using the device “instinctual,” said Jamie Shotton, a partner scientist who leads the HoloLens science team in Cambridge, United Kingdom.

“We want you to know how to use HoloLens without having to be taught how to use it,” he said. “We know how to interact with things in the real, physical world: We pick things up, we press buttons, we point to things. We aim, as far as possible, to translate that directly into mixed reality.”

Microsoft today announced the HoloLens 2 is now shipping to customers. The sensor-packed holographic computing headset uses AI to displace space and time, creating a mixed reality of people, places and things in order to facilitate one of our most basic human impulses: exchanging knowledge.

Microsoft Technical Fellow Alex Kipman said the headset defines the highest watermark of intelligent edge devices – AI-capable technologies that can collect and process data even without a reliable internet connection – and that can then share some or all of that data with the intelligent cloud when connected.

On a recent day, Kipman sketched a diagram of this ubiquitous computing fabric on a digital whiteboard in his office.

“HoloLens,” he said, “is the first native device to be invented from the ground up with this worldview in mind.”  .... "

Airbus Tests Hololens

An example of the use of Hololens and VR/Mixed Reality for advanced manufacturing systems.  Most obvious space for this, which we tested .... integrating large quantities of background information such as blueprints and online analyses and metadata for manufacturing compliance specifications. 

Airbus works with Microsoft to trial HoloLens in aircraft manufacturing
Aircraft company Airbus believes mixed reality systems, powered using headsets such as HoloLens, will drive its digital manufacturing strategy .... "

A Look at Future HoloLens Mixed Reality

Good half hour Podcast interview about HoloLens-2 current and future on Twit-TV

Gives a good update of HoloLens at Microsoft and how they are now focusing its uses

Today's special guest engineering GM for mixed reality apps, Lorraine Bardeen of Microsoft.  Lets talk more about what MS is doing with HoloLens, iOS, Android in mixed reality. ... 

Links to Dynamics 365 Mixed Reality and Perception Apps

Use for First line workers.  Field service inspection.

Initially will be partially delivered on IOS, Android phones. Then to applications with a HoloLens.

Now looking directly at manufacturing and supporting live demonstration, inspection, maintenance.

Apparently are not driving at a consumer orientation at this time.

This is exactly what we saw in the 80s for plant maintenance, and in some much rarer areas for previewing marketing in place.

HoloLens 2 Open for Preorder

Early reviews seem to say it is a considerable improvement.    Targeted towards business rather than games.  Still quite expensive for the consumer,  $3500 per device.   How much directly usable off the shelf applications are available?   Lots of demo videos below.  Interest in preorder, with no date of shipment mentioned?

HoloLens 2
Mixed reality is ready for business
A new vision for computing, built on a history of innovation

Mixed reality on HoloLens 2 combines an untethered device with apps and solutions that help people across your business learn, communicate, and collaborate more effectively. It's the culmination of breakthroughs in hardware design, artificial intelligence (AI), and mixed reality development from Microsoft, designed to help you lead your industry into the future—starting today.  .... "

Good podcast at Twit: Windows Weekly overview:   https://twit.tv/shows/windows-weekly/episodes/610?autostart=false 

Hololens 2 to have Eye Movement Detection

Good addition of detecting and I assume controlling with eye movement.   The control by gesture has never been good enough for many applications.  Need more general applications to get people used to the interaction

Microsoft HoloLens 2 augmented reality headset unveiled  By Leo Kelion in BBC Tech.

Microsoft has a new version of its augmented reality headset, which now detects where its users are looking and tracks the movements of their hands.

It said that HoloLens 2 wearers would find it easier to touch and otherwise interact with graphics superimposed over their real-world views.

Other improvements include filling more of a user's view and automatically recognising who they are.

The firm is pitching the kit as being ready for use in business environments.

Many experts believe mixing together graphics and real-world views has greater potential than virtual reality, which removes the user from their immediate environment. .... "

Holoforge for Historical Experience

Saw a number of examples early on of using VR to effectively show historical data,  good example for use, for both teaching and for interacting with complex data.

The Mont Saint-Michel: Digital perspectives on the model
Celebrating French art, history, and culture through mixed reality and artificial intelligence.
Share

The Musée des Plans-Reliefs is bringing to life the historic Mont-Saint-Michel relief map—an example of the 17th century’s most advanced mapping technology—via a mixed reality experience that uses current mapping innovations to immerse viewers in a vital piece of French history and culture.

For centuries, technology has been influencing the way people engage with the world and shape the course of history. In 17th and 18th-century France, large-scale 3D maps—painstakingly built by hand down to the most intricate details—were the most advanced mapping technology of their time. They were considered such valuable strategic tools that leaders like Napoleon and King Louis XIV considered them military secrets and hid them from public view.

The Musee des Plans-Reliefs in Paris is home to more than 100 of these historic relief maps that  have withstood the test of time. But the crown jewel of the collection is the model of Mont-Saint-Michel—a rocky headland off the Normandy coast with a Benedictine abbey that’s an architectural marvel in its own right—presented by a monk to Louis XIV in 1701. A new HoloLens experience is bringing these technological feats, both the relief map and Mont-Saint-Michel itself, to life for a new generation.

The museum is partnering with Microsoft, HoloForge, and Iconem to create “The Mont Saint-Michel: Digital perspectives on the model,” a HoloLens experience celebrating French culture and innovation. The goal of the exhibit is to use mixed reality technology in a way that empowers the Musée des Plans-Reliefs to unlock a more vital kind of storytelling.  ....  " 

Windows Insider on Mixed Reality

Been following Microsoft's interest in 'mixed reality' aka augmented reality for a client.  Just discovered Microsoft insider podcasts, which looks at near future directions in Windows operating systems,  applications and their potential use in the enterprise.  Most recently they have just featured a considerable podcast on mixed reality that is worth a look.  Quote:

Alex Klipman:  (from video):  Now we're standing together at the threshold of the next revolution of computing.  Now, the thing that excites me about this revolution is that computers will empower us to renegotiate our very contract with reality, giving us the capability to transcend time, space, and devices

In this revolution we will immerse ourselves in virtual worlds of our choosing, and we'll be able to accomplish impossible things.  And we'll be able to do all of this while creating lasting memories with the people that we love

Our very sense of reality is set to be transformed as we enter this new era of computing, the era of mixed reality .... " 

Update on Hololens

Micorosft's update on  AR work.  Always looking for practical business examples.   A bit thin on new applications covered.  Here covered:

Driving collaboration in car design
Ford design and engineering teams are able to iterate more quickly when they use HoloLens to visualize full-scale models in 3D. Processes that used to take weeks now take days, because they no longer need to physically build every prototype. ... 

Modernizing crime scene investigation
HoloLens enables police officers to capture evidence in real time with tuServ, Black Marble’s award-winning app for police operations. Gathering digital evidence decreases the risk of disturbing physical evidence. .... 

Bring life to Mixed Reality
Create holograms of people and performances for your mixed reality, virtual reality, and augmented reality experiences. Mixed Reality Capture Studios are now open San Francisco, Seattle, and London. ... "

Ford Using MS HoloLens for Car Design

Complex and cooperative design continues to be a place where augmented reality is likely to be valuable.

Ford Using Microsoft HoloLens For Car Design By Rebecca Hills-Duty in VR Focus.

HoloLens allows designers and engineers to prototype new cars faster and more cheaply.
Several businesses have been exploring the possibilities offered by Microsoft’s mixed reality (MR) headset. Several car manufacturers such as Volvo and Audi have been experimenting with augmented reality (AR) and MR, and now Ford is joining them.

Ford were already using the HoloLens in a limited capacity, but now the company is expanding its use after a pilot phase. Usually when designing a car, designers and artists will often start with simple pen and paper, then move on to crafting a solid 3D version using modelling clay. Designers and artists can now use HoloLens to adjust 3D models, taking into account real-world limitations and adjustments without needing to rebuild the model from scratch every time.

Designers and engineers can put an overlay over a currently existing car and prototype adjustments for the new model without needing to completing built it anew, or physically build every new prototype. Designers and engineers save a great deal of time and money be using this approach with the HoloLens. It also means that radical new ideas and concepts can be considered, and many more ideas tried out to find the perfect fit. .... " 

The Nature of Mixed Reality

Have noted Microsoft's use of the mixed reality term.   To me its just confusing, and unless Microsoft completely takes over the market, seems a waste of time and attention to think about.  Are newly available capabilities and hardware changing this?  The Verge looks at this: 

I’ve heard a lot about Microsoft’s Windows Mixed Reality, but up until this week I’ve been a little confused. Microsoft insists on calling its new headsets “Mixed Reality” ones, when they’re really just virtual reality. At IFA in Berlin this week, Microsoft is showcasing all of its Mixed Reality headsets from PC makers like Acer, Asus, Dell, HP, and Lenovo. Most of them are arriving next month, priced around $349 to compete with the Oculus Rift and HTC Vive. So what is Windows Mixed Reality really like? I got a chance to try it and find out. ... " 

Mixed Reality Partner Program

Received:

Introducing the Mixed Reality Partner Program

If your company is a digital agency, systems integrator, or solution provider, the Mixed Reality Partner Program can equip your company with the skills to successfully drive commercially-focused mixed reality solutions for your clients. .... " 

Microsoft Discusses Their Mixed Reality Focus

Interesting to watch,  because Microsoft crafts or influences much of the worlds personal computer user experience.  So will user experience become VR or mixed reality?   These technologies give great demonstrations, but how much value do they give to the experience, unless you are gaming or working with a hands-free application of some sort?  Competition with phone based systems?   Links to the conference presentations.

Microsoft Discuss Their Mixed Reality Future

Watch their Taipei conference session in full.
By Kevin Eva   

Earlier this week we brought you news from Taipei where Microsoft were taking part in a ‘WinHEC’ – a Windows Hardware Development Conference, where a number of representatives went through the current crop of mixed reality (MR) related Microsoft products, demonstrated problems and solutions that the technology can solve and explained what the way the technology works. All with an overview of Microsoft’s current plans with regard to MR. ... "

What if Augmented Reality is Everywhere?

During innovation sessions we came up with a number of AR scenarios.  Location aware AR allows you to place content in any space.      So imagine if you walked into a restaurant wearing VR glasses, observing through a camera, or even soon with eye implants.   This could overlay on your perception, for example, ads about another restaurant, hung on the walls.   Like today in sports bars,  where you can see ads for other sports bars on TVs.  This would take it much further.  

Pokemon Go was another recent example, where players could 'see' things placed in a space,  for the purposes of a game,  sometimes with chaotic or dangerous results.    Then who owns public spaces that can be manipulated constantly and dynamically by augmented reality?   Every wall, every space and surface, open for augmentation, graffiti,  ads?

" ... what the world will be like when Augmented Reality is ubiquitous. Of particular interest is the potential for legal battles over Augmented Reality space. ... "

This has even led to a US lawsuit   Augmented reality lawsuit provides augmented view of 1st Amendment   “They’re passing two dimensional laws in a three dimensional world.” ... 

See for example Augnite:

" ... Full service Augmented Reality app development projects, managed teams and optimisation ... Augnite works with blue chip clients to deliver high performance AR apps for Microsoft Hololens and Daqri. With offices in Dundee and London, Augnite enable clients across multiple industry sectors, to develop and optimise world class AR apps that generate new revenue streams and streamline costs. ... " 

They write about the implications:
       https://www.augnite.com/happens-augmented-reality-everywhere/

They suggest this might look like the below:

Microsoft Reveals Hololens Progress

VRFocus on Upcoming applications to be shown at the Microsoft Build Event.  Good indication of what is being worked on

Microsoft Build 2017 Sessions Revealed
The conference takes place later this week.  May 10.
 By Peter Graham  Last updated May 8, 2017 

HoloLensMicrosoftMicrosoft Build 2017
For virtual reality (VR), augmented reality (AR) and mixed reality (MR) tech enthusiasts and developers this week is quite a big one. Starting today and finishing Wednesday is the NVIDIA GPU Technology Conference (2017) featuring the latest immersive news and insights from the company. Wednesday also sees the start of Microsoft Build 2017, set to include even more of the very latest tech, and today the sessions have now been unveiled.

As VRFocus readers may expect, Microsoft Build 2017 will be far more focused on MR and AR than VR, with sessions revolving around HoloLens and the Universal Windows Platform (UWP). .... " 

( At link, session titles of work underway)  .....