Matterport and Facebook AI Collaborate on 3D Spaces

Impressive efforts, including mention of staging digital twin simulations with scanned 3D Data.

Matter and Facebook AI Collaborate on Worlds Largest Dataset of 3D Spaces

SUNNYVALE, Calif. — Matterport, the spatial data company leading the digital transformation of the built world, which has entered into a definitive agreement to enter into a business combination with Gores Holdings VI (NASDAQ: GHVI, GHVIU, and GHVIW, today announced a collaboration with Facebook AI Research (FAIR) through which it will make the largest-ever dataset of 3D indoor spaces available exclusively for academic, non-commercial uses. The Habitat-Matterport 3D Research Dataset (HM3D) is an unprecedented collection of 1,000 high-resolution Matterport digital twins made up of residential, commercial, and civic spaces generated precisely from real-world environments. HM3D will play a significant role in advancing embodied AI research which seeks to teach robots and virtual AI assistants to understand and interact with the complexities of the physical world.

“Until now, this rich spatial data has been glaringly absent in the field, so HM3D has the potential to change the landscape of embodied AI and 3D computer vision,” said Dhruv Batra, Research Scientist at Facebook AI Research. “Our hope is that the 3D dataset brings researchers closer to building intelligent machines, to do for embodied AI what pioneers before us did for 2D computer vision and other areas of AI.”

HM3D is free and available now for academic, non-commercial research. Researchers can use it with FAIR’s Habitat simulator to train embodied agents, such as home robots and AI assistants, at scale. HM3D is a foundational step towards helping these agents navigate through real-world environments and better understand the variations of spaces such as bedrooms, bathrooms, kitchen and hallways, as well as the different configurations of those rooms within every structure. It can also assist robots in recognizing how objects within rooms are typically arranged so that instructions are correctly understood. This research could one day be used in production applications like robots that can retrieve medicine from a bedroom nightstand or AR glasses that can help people remember where they left their keys. 

“We are excited to collaborate with Facebook as we provide the academic and research communities access to this unique spatial dataset that is sure to impact how we work and live,” said Conway Chen, Vice President of Business Development and Alliances at Matterport. “With more than five million spaces captured with the Matterport platform, we are the only company that can offer a diverse library of high-resolution, data-rich digital twins of various styles, sizes, and complexities from across the world. HM3D can also be used more broadly by academia, and we can’t wait to see what innovations emerge.” 

“We’ve been challenged with a lack of spatial data to advance innovation in real-estate, construction, robotics, augmented reality, and more. But with the HM3D dataset from the collaboration between Matterport and Facebook AI, we're excited about the significant progress we'll make in advancing research in indoor scene reconstruction, generation, and analysis at a house-scale for the first time," said Yasutaka Furukawa, Associate Professor of Computing Science at Simon Fraser University.

Matterport has defined the spatial data category for the built world with its market-leading platform that transforms any physical space into an accurate and immersive digital twin. Today, many industries leverage Matterport’s digital twins and spatial data insights to fundamentally change the way business is done. Real estate agents gain a deeper understanding of every property that gets digitized and gain the ability to see what is most important to thousands of prospective buyers. Retailers can digitize multiple stores at any point in time to track fixed assets and inventory. Developers and property owners can use Matterport to document every stage of construction and create a robust digital system of record over the entire lifecycle of the building. Matterport’s rapidly expanding developer platform makes it easy for partners and developers to create powerful new tools and draw new property insights, fueling industry-wide innovation that makes every space more valuable, accessible and useful. ... ' 

Facebook Wants Watch like Controllers for AR Glasses

 Is Facebook aiming to take us long and deep with augmented reality controlled from our watches?    I still think there is something missing to make the idea universal.

Facebook reveals new watch-like controllers for its future AR glasses

The wearable devices will read nerve signals from the brain and deliver tactile feedback through haptic buzzers around the wrist.... 

By Mark Sullivan   in FastCompany

While other tech companies tend to keep their R&D under a blanket of secrecy, Facebook is opening its labs and showing the world how it’s developing its future augmented reality (AR) glasses. The company held the second in a series of its “Inside the Lab” media roundtables on Tuesday and introduced new technology that would enable users to control their AR glasses using their fingers.

AR glasses project light onto the eyes and seem to overlay the real world with digital imagery. As you’re looking through the glasses at a statue, for instance, the glasses might display a label with helpful information right next to it. Many people in tech circles believe AR glasses, in some form, will eventually replace the smartphone as our go-to personal computing device.

In this week’s roundtable, company executives, starting with chief technology officer Mike Schroepfer, showed off wrist wearables that detect the movement of the glasses wearer’s fingers. The wearable uses a technology called electromyography (EMG), which intercepts the electrical signals the brain uses to direct finger movements. It then translates the electrical pulses into digital commands that can control functions of the device functions. ... "

Facebook AI Releases Searchable Code Snippets

Seems a useful thing, what Facebook has put on the web to support AI efforts is impressive.  See more at the Facebook.ai site.  Also note the claimed validation against malware. 

Facebook Releases AI Code Search Datasets   By Anthony Alford in InfoQ

Facebook AI released a dataset containing coding questions paired with code-snippet answers, intended for evaluating AI-based natural-language code search systems.  The release also includes benchmark results for several of Facebook's own code-search models and a training corpus of over 4 million Java methods parsed from over 24,000 GitHub repositories.

In a paper published on arXiv, researchers described their technique for collecting the data. The training data corpus was collected from the most popular GitHub repositories of Android code, ranked by number of stars. Every Java file in the repositories was parsed, identifying the individual methods. Facebook used the resulting corpus in their research on training code-search systems. To create the evaluation dataset, they started with a question-and-answer data dump from Stack Overflow, selecting only questions that had both "Java" and "Android" tags. Of these, they kept only questions that had an upvoted answer that also matched one of the methods identified in the training data corpus. The resulting 518 questions were manually filtered to a final set of 287. According to the researchers:

Our data set is not only the largest currently available for Java, it’s also the only one validated against ground truth answers from Stack Overflow  .... " 

Facebook Open Source AI

What Facebook is Doing for Open-Source AI.   Technical details and detailed pointers to resources.  Announces what they call Tensor Comprehensions.

Announcing Tensor Comprehensions

By: Nicolas Vasilache, Oleksandr Zinenko - Inria & DI ENS, Theodoros Theodoridis - ETH Zürich, Priya Goyal, Zachary DeVito, William S. Moses - MIT CSAIL, Sven Verdoolaege, Andrew Adams, Albert Cohen - Inria & DI ENS & FAIR

Today, Facebook AI Research (FAIR) is announcing the release of Tensor Comprehensions, a C++ library and mathematical language that helps bridge the gap between researchers, who communicate in terms of mathematical operations, and engineers focusing on the practical needs of running large-scale models on various hardware backends. The main differentiating feature of Tensor Comprehensions is that it represents a unique take on Just-In-Time compilation to produce the high-performance codes that the machine learning community needs, automatically and on-demand.  ....

What to expect next

This release will allow researchers and programmers to write layers in a notation that is similar to the maths they use in their papers and communicate concisely the intent of their program. They will also be able to take that notation and translate it easily into a fast implementation in a matter of minutes rather than days. As the toolchain grows, we expect usability and performance to increase and benefit the whole community.

We will release PyTorch integration for Tensor Comprehensions at a later date.

We are grateful for frequent exchanges with and feedback from the frameworks teams and are looking forward to bringing this exciting new technology to your favorite ML framework.

FAIR is committed to open science and working with the machine learning community to push AI research further. Tensor Comprehensions is already a collaboration between Facebook, Inria, ETH Zurich and MIT. Our work is in the early stages and we’re excited to share it early and look forward to improving it with feedback from the community.

Get started:
Tensor Comprehensions is available under the Apache 2.0 license.
Documentation
On ArXiv
On Slack
Email: tensorcomp@fb.com