Backing off the Metaverse

I would like to see another take at the Metaverse,  but its not ready.  Will it ever be?     The alternative direction is guessable.   But they has been working AI for some time.  Recall Jarvis?

Mark Zuckerberg Abandons Metaverse for a new Shiny Toy   in Futurism

Poor, poor, Horizon Worlds. According to Facebook-turned-Meta CTO Andrew Bosworth, the company's metaverse of dead-eyed avatars has been all but abandoned by Meta CEO Mark Zuckerberg — who, in an added blow, is instead said to be spending the bulk of his time chasing the investor-appeasing Silicon Valley squirrel that is generative AI.

"We've been investing in artificial intelligence for over a decade, and have one of the leading research institutes in the world," Bosworth told Nikkei Asia in an interview on Wednesday. "We certainly have a large research organization, hundreds of people."

"We just created a new team, the generative AI team, a couple of months ago; they are very busy," he added. "It's probably the area that I'm spending the most time [in], as well as Mark Zuckerberg and [Chief Product Officer] Chris Cox."

That sound you just heard in the distance? A single, pixelated tear, hitting the deserted Horizon Worlds' floor.

To be fair, as Bosworth noted, Facebook has been working on AI R&D for years, and their large language model (LLM), LLaMa, is rumored to be quite powerful. The CTO also threw the metaverse a generative AI-laced bone elsewhere in the Nikkei Asia interview, explaining that Meta hopes that AI will ultimately make its beleaguered metaverse better.

"Previously, if I wanted to create a 3D world, I needed to learn a lot of computer graphics and programming," Bosworth told the outlet. "In the future, you might be able to just describe the world you want to create and have the large language model generate that world for you."

In the short term, though? Meta, per Bosworth's comments, is unsurprisingly gearing up to deploy AI in its advertising machine, the channel where the company makes the vast majority of its money — as opposed to the metaverse, which at this point is a multibillion-dollar money pit. (Get ready for those hyper-targeted, AI-generated ads, y'all. Cheers to living in hell!)  ... ' 

What Data Does Facebook Have?

From Schneier, with considerable comment ....    From a Court Deposition: 

" .... Facebook’s stonewalling has been revealing on its own, providing variations on the same theme: It has amassed so much data on so many billions of people and organized it so confusingly that full transparency is impossible on a technical level. In the March 2022 hearing, Zarashaw and Steven Elia, a software engineering manager, described Facebook as a data-processing apparatus so complex that it defies understanding from within. The hearing amounted to two high-ranking engineers at one of the most powerful and resource-flush engineering outfits in history describing their product as an unknowable machine.  ...."

Metaverse Wearable Device

A Meta-Facebook device on every face?

Metaverse wearable devices ‘could be as big as phones,’ Qualcomm CEO says  in Yahoo Finance

by Julie Hyman·Anchor

Wed, November 17, 2021, 2:57 PM·2 min read

The company whose chips power Facebook’s Oculus virtual-reality headset has already tapped into the metaverse opportunity. Now Qualcomm’s CEO says glasses could eventually be as widespread as smartphones. 

Qualcomm CEO Cristiano Amon knows smartphones well. The company’s main business is providing semiconductors for those devices. As Apple turns toward manufacturing its own chips for its phones, Qualcomm has been working on diversifying its revenue into products and services for the automotive industry, the Internet of Things — and the metaverse. They’re not new entrants into the market. 

“For all of the devices that are being commercially deployed, you have one thing in all of them — which is Snapdragon XR,” Amon told Yahoo Finance Live, referencing Qualcomm’s extended reality version of its main Snapdragon platform. “So we’re very excited about that opportunity, and it could be as big as phones, if you have companion glasses that you carry together with your smartphone.” 

Amon pointed to Facebook’s name change as one marker of success for that company’s XR device: "Even looking at how the company is now called Meta is a reflection of how successful Oculus has been,” he said.

Metaverse products were one aspect of the diversification strategy Amon laid out during Qualcomm’s investor day on Tuesday. He outlined how Qualcomm is working on reducing its reliance on Apple, and is targeting $46 billion in revenue by 2024, including $9 billion from the Internet of Things. ... ' 

Facebook Goes Tactile

 Been surprised  by directions Facebook is taking,  Giving AI a bad name? In Spectrum IEEE

Why Facebook (Or Meta) Is Making Tactile Sensors for Robots Durable and affordable fingers and skin could help virtual agents understand their world BY EVAN ACKERMAN

Facebook, or Meta as it's now calling itself for some reason that I don't entirely understand, is today announcing some new tactile sensing hardware for robots. Or, new-ish, at least—there's a ruggedized and ultra low-cost GelSight-style fingertip sensor, plus a nifty new kind of tactile sensing skin based on suspended magnetic particles and machine learning. It's cool stuff, but why?

Obviously, Facebook Meta cares about AI, because it uses AI to try and do a whole bunch of the things that it's unwilling or unable to devote the time of actual humans to. And to be fair, there are some things that AI may be better at (or at least more efficient at) than humans. AI is of course much worse than humans at many, many, many things as well, but that debate goes well beyond Facebook Meta and certainly well beyond the scope of this article, which is about tactile sensing for robots. So why does Facebook Meta care even a little bit about making robots better at touching stuff? Yann LeCun, the Chief AI Scientist at Facebook Meta, takes a crack at explaining it:

Before I joined Facebook, I was chatting with Mark Zuckerberg and I asked him, "is there any area related to AI that you think we shouldn't be working on?" And he said, "I can't find any good reason for us to work on robotics." And so, that was kind of the start of Facebook AI Research—we were not going to work on robotics.

After a few years, it became clear that a lot of interesting progress in AI was happening in the context of robotics, because robotics is the nexus of where people in AI research are trying to get the full loop of perception, reasoning, planning, and action, and getting feedback from the environment. Doing it in the real world is where the problems are concentrated, and you can't play games if you want robots to learn quickly.

It was clear that four or five years ago, there was no business reason to work on robotics, but the business reasons have kind of popped up. Robotics could be used for telepresence, for maintaining data centers more automatically, but the more important aspect of it is making progress towards intelligent agents, the kinds of things that could be used in the metaverse, in augmented reality, and in virtual reality. That's really one of the raison d'être of a research lab, to foresee the domains that will be important in the future. So that's the motivation.

Well, okay, but none of that seems like a good justification for research into tactile sensing specifically. But according to LeCun, it's all about putting together the pieces required for some level of fundamental world understanding, a problem that robotic systems are still bad at and that machine learning has so far not been able to solve:

How to get machines to learn that model of the world that allows them to predict in advance and plan what's going to happen as a consequence of their actions is really the crux of the problem here. And this is something you have to confront if you work on robotics. But it's also something you have to confront if you want to have an intelligent agent acting in a virtual environment that can interact with humans in a natural way. And one of the long-term visions of augmented reality, for example, is virtual agents that basically are with you all the time, living in your automatic reality glasses or your smartphone or your laptop or whatever, helping you in your daily life as a human assistant would do, but also can answer any question you have. And that system will have to have some degree of understanding of how the world works—some degree of common sense, and be smart enough to not be frustrating to talk to. And that is where all of this research leads in the long run, whether the environment is real or virtual.

AI systems (robots included) are very very dumb in very specific ways, quite often the ways in which humans are least understanding and forgiving of. This is such a well established thing that there's a name for it: Moravec's paradox. Humans are great at subconscious levels of world understanding that we've built up over years and years of experience being, you know, alive. AI systems have none of this, and there isn't necessarily a clear path to getting them there, but one potential approach is to start with the fundamentals in the same way that a shiny new human does and build from there, a process that must necessarily include touch.  .... ' 

Regulators Want more Details of Payment Systems

To be expected, how can the details of this be used to improve security?

Regulators demand more info about tech payment systems  in TheVerge

Query orders sent out to Amazon, Apple, Google, Facebook, Paypal, and Square

By Russell Brandom

The Consumer Financial Protection Bureau has issued a series of information-gathering orders on the business practices of major digital payments systems. Orders were issued to Amazon, Apple, Facebook, Google, PayPal, and Square, the bureau said in an announcement on Thursday morning. Chinese pay systems operated by WeChat and AliPay will also be studied but are not subject to any CFBP orders.

Issued as part of the CFPB’s investigative authority, the orders are not an indication of specific wrongdoing, but they suggest the CFBP will be taking a more active role in regulating tech companies’ consumer products in the future. In particular, they are meant to root out any data collection or anti-competitive conduct that may have not yet come to light.

The companies did not immediately respond to requests for comment. ... ' 

Facebook's Tools can Target a Single User

Well I would think the title above is clearly true, but the question is how effective is it in a useful context? And is that ethically more powerful than a typical ad?  

Researchers Show Facebook's Ad Tools Can Target Single Users

TechCrunch, Natasha Lomas, October 15, 2021

Spanish and Austrian researchers have showed that Facebook can target ads to a single individual, given sufficient knowledge of that person’s assigned interests. The researchers described a data-driven model that characterizes a metric indicating the likelihood a Facebook user can be identified based on interests attached to them by the social media giant's ad platform. The model found that "the four rarest interests or 22 random interests from the interests set FB assigns to a user make them unique on FB with a 90% probability," the researchers wrote. These findings raise issues about potentially harmful uses of Facebook's ad-targeting tools, and about the legality of the platform's personal data processing system. ...

Italian Data Protection Authority Queries Smart Glasses

 I thought there might be some regulatory push back on Facebook smart glasses that gather and store data and images, here perhaps the first case.    

Italy Data Authority Asks Facebook for Clarifications on Smart Glasses

Reuters, Elvira Pollina, September 10, 2021

Italian data protection authority Garante has asked Facebook to provide it with clarifications related to its newly launched smart glasses, to determine the product's compliance with that nation’s privacy laws. Developed with Ray-Ban manufacturer EssilorLuxottica, the Ray-Ban Stories glasses allow users to hear music, take calls, or shoot photos and videos and share them across Facebook's services via a companion application. Garante made its request via the Irish Data Protection Commissioner (DPC), which oversees Facebook because the social-media giant's European headquarters are based in Ireland. The regulator said it wanted clarification on measures Facebook has deployed to shield people occasionally filmed, especially children, and on systems adopted to anonymize collected data, and features of the smartglasses' voice assistant.  ... ' 

Facebook Roy-Ban Smart Glasses

 Happened on this, more on a Facebook Smart Glasses.  Facebook Voice Assistant?  Hmmm ..  Too much in your face all the time digital world?  

Facebook and Ray-Ban Debut ‘Stories’ Smart Glasses With Facebook Voice Assistant

ERIC HAL SCHWARTZ on September 9, 2021 at 1:00 pm

Facebook and Ray-Ban have released a new line of smart glasses called Ray-Ban Stories controlled by the Facebook Assistant voice assistant. The two companies have been rumored to be collaborating on voice-controlled smart glasses, although the resulting focus on taking photos and videos is not what was anticipated.  ... '

Facebook Announces Droidlet

Had not heard of this before,  but interesting take by Facebook.  Still have my doubts about Facebook, they seem to be too intent to get into our brains,  but some of their research work is worth understanding. I like thinking about elements of the process as being composed  of agents that have intent.  And the ability to readily test alternatives.    Gets to intelligence that way.   Tell me more Facebook. 

Introducing droidlet, a one-stop shop for modularly building intelligent agents

The annals of science fiction are brimming with robots that perform tasks independently in the world, communicate fluently with people using natural language, and even improve themselves through these interactions. These machines do much more than follow preprogrammed instructions; they understand and engage with the real world much as people do.

Robots today can be programmed to vacuum the floor or perform a preset dance, but the gulf is vast between these machines and ones like Wall-E or R2-D2. This is largely because today’s robots don’t understand the world around them at a deep level. They can be programmed to back up when bumping into a chair, but they can’t recognize what a chair is or know that bumping into a spilled soda can will only make a bigger mess.

To help researchers and even hobbyists to build more intelligent real-world robots, we’ve created and have open-sourced the droidlet platform

Droidlet is a modular, heterogeneous embodied agent architecture, and a platform for building embodied agents, that sits at the intersection of natural language processing, computer vision, and robotics. It simplifies integrating a wide range of state-of-the-art machine learning (ML) algorithms in embodied systems and robotics to facilitate rapid prototyping.

People using droidlet can quickly test out different computer vision algorithms with their robot, for example, or replace one natural language understanding model with another. Droidlet enables researchers to easily build agents that can accomplish complex tasks either in the real world or in simulated environments like Minecraft or Habitat.

There is much more work to do — both in AI and in hardware engineering — before we will have robots that are even close to what we imagine in books, movies, and TV shows. But with droidlet, robotics researchers can now take advantage of the significant recent progress across the field of AI and build machines that can effectively respond to complex spoken commands like “pick up the blue tube next to the fuzzy chair that Bob is sitting in.” We look forward to seeing how the research community uses droidlet to advance this important field.   ... ' 

Facebook Projects Your Eyes onto VR

Interesting things out of Facebook labs.

Facebook Can Project Your Eyes onto a VR Headset

The Verge, Adi Robertson, August 4, 2021  from ACM

The reverse passthrough virtual reality (VR) technique devised by engineers at Facebook Reality Labs (FRL) can project a user's face onto the front of a VR headset. The passthrough VR feature displays a live video feed from the headset's cameras, so users can see the real world while wearing the device. Researchers designed a prototype headset that augments a standard display with a stack of lenses and cameras. Stereo cameras capture an image of the wearer's face and eyes within the headset, and movement is mapped onto a digital model projected onto an outward-facing light field display. The FRL team's research highlights the potential of light field displays and opportunities for enhanced in-person social interactions  ... 

Facebook can project your eyes onto a VR headset, and it’s exactly as uncanny as it sounds

It’s got a TV eye on you,  By Adi Robertson@thedextriarchy .... 

Using the Rust Language for Code Safety

In a recent dev oriented conversation someone mentioned Rust as a programing Language to improve code safety.   I heard about it again today re Facebook's usage.   Also the Rust Foundation seems to have support from major players.   Here is an overview and history of its use.  

A brief history of Rust at Facebook

Facebook is embracing Rust, one of the most loved and fastest-growing programming languages available today. In addition to bringing new talent to its Rust team, Facebook has announced that it is officially joining the nonprofit Rust Foundation. Alongside fellow members including Mozilla (the creators of Rust), AWS, Microsoft, and Google, Facebook will be working to sustain and grow the language’s open source ecosystem.

For developers, Rust offers the performance of older languages like C++ with a heavier focus on code safety. Today, there are hundreds of developers at Facebook writing millions of lines of Rust code. And while it’s clear that Facebook is increasingly invested in the future of the language, it’s important to understand how we grew to this point.  ... '

Facebook Research Allocates Ad Funding (without AI)

Struck me because we did something similar and also quite different for advertising fund allocation very early on.    We the results of integer optimization models with montecarlo simulation, constrained to advertising agreements.   I like the fact that AI is never mentioned!  But in much later spins on this,  what could be called AI was used to check for accuracy and regulatory compliance to contractual agreements.  Nice.

Auto-placement of ad campaigns using multi-armed bandits  in Facebook Research

By: Vashist Avadhanula, Riccardo Colini Baldeschi, Stefano Leonardi, Karthik Abinav Sankararaman, Okke Schrijvers

What the research is:

We look at the problem of allocating the budget of an advertiser across multiple surfaces optimally when both the demand and the value are unknown. Consider an advertiser who uses the Facebook platform to advertise a product. They have a daily budget that they would like to spend on our platform. Advertisers want to reach users where they spend time, so they spread their budget over multiple platforms, like Facebook, Instagram, and others. They want an algorithm to help bid on their behalf on the different platforms and are increasingly relying on automation products to help them achieve it.

In this research, we model the problem of placement optimization as a stochastic bandit problem. In this problem, the algorithm is participating in k different auctions, one for each platform, and needs to decide the correct bid for each of the auctions. The algorithm is given a total budget B (e.g., the daily budget) and a time horizon T over which this budget should be spent. At each time-step, the algorithm should decide the bid it will associate with each of the k platform, which will be input into the auctions for the next set of requests on each of the platforms. At the end of a round (i.e., a sequence of requests), the algorithm sees the total reward it obtained (e.g., number of clicks) and the total budget that was consumed in the process, on each of the different platforms. Based on just this history, the algorithm should decide the next set of bid multipliers it needs to place. The goal of the algorithm is to maximize the total advertiser value with the given budget across the k platforms.   ... "    

Full paper:  https://arxiv.org/abs/2103.10246 

Facebook Wants Watch like Controllers for AR Glasses

 Is Facebook aiming to take us long and deep with augmented reality controlled from our watches?    I still think there is something missing to make the idea universal.

Facebook reveals new watch-like controllers for its future AR glasses

The wearable devices will read nerve signals from the brain and deliver tactile feedback through haptic buzzers around the wrist.... 

By Mark Sullivan   in FastCompany

While other tech companies tend to keep their R&D under a blanket of secrecy, Facebook is opening its labs and showing the world how it’s developing its future augmented reality (AR) glasses. The company held the second in a series of its “Inside the Lab” media roundtables on Tuesday and introduced new technology that would enable users to control their AR glasses using their fingers.

AR glasses project light onto the eyes and seem to overlay the real world with digital imagery. As you’re looking through the glasses at a statue, for instance, the glasses might display a label with helpful information right next to it. Many people in tech circles believe AR glasses, in some form, will eventually replace the smartphone as our go-to personal computing device.

In this week’s roundtable, company executives, starting with chief technology officer Mike Schroepfer, showed off wrist wearables that detect the movement of the glasses wearer’s fingers. The wearable uses a technology called electromyography (EMG), which intercepts the electrical signals the brain uses to direct finger movements. It then translates the electrical pulses into digital commands that can control functions of the device functions. ... "

Facebook's Seer Recognizes Images

Thinking the implications of this in business practice.

Facebook’s new AI model SEER can teach itself to recognize images  By Mike Wheatley in SiliconAngle

Researchers at Facebook Inc. have created an artificial intelligence-based image recognition model called SEER  (Technical) that’s able to describe what it’s seeing without being trained first on a labeled dataset.

Facebook said that SEER, which is an acronym for “Self-SupeERvised,” is a breakthrough that could lead to a “revolution” in computer vision.

The SEER model, outlined in a paper released March 2, was fed 1 billion publicly available images without annotations or labels from Instagram. It then worked through the dataset, learning as it progressed, and was eventually able to achieve extremely high accuracy in tasks such as object detection.

Self-supervised AI learning is already established in the AI field. It refers to AI systems that can learn directly from whatever information they are given without being trained on carefully labeled datasets that can teach them how to perform a given task, such as recognizing an object in a photo or translating a piece of text.  ... ' 

Better VR Hearing

Adding additional sensors in AR cold lead to specific new application areas.  Spatial Audio could add to maintenance capabilities when linked to machine learning.  Also that Facebook continues to be interested in 'smart glasses' direction.  Perhaps not superhearing though. 

Facebook wants its AR glasses to give wearers superhearing

It could improve VR headset audio, too

By Adi Robertson@thedextriarchy  in TheVerge

 A new Facebook experiment shows how sound could play a major role in augmented reality. Chief scientist Michael Abrash and his team at FRL Research (formerly Facebook Reality Labs) released details today https://about.fb.com/news/2020/09/facebook-reality-labs-research-future-of-audio/  about what the team calls “perceptual superpowers” — AR systems that figure out what you’re trying to hear, then amplify it and dampen background noise. Combined with spatial audio features, the device creates the aural equivalent of a hologram overlay in a pair of glasses.  ... " 

Facebook Releases Deepfakes

The more data the better.  Negative examples are also useful to proof the model, examine its edges,  detect movement from the model later.   I don't believe in restricting how we can use models, only in honestly testing their use.

Facebook released a database of 100,000 deepfakes to teach AI how to spot them
The videos are designed to help improve AI’s performance—as even the best methods are still not accurate enough.

by Will Douglas Heaven  

Deepfakes⁠ have struck a nerve with the public and researchers alike. There is something uniquely disturbing about these AI-generated images of people appearing to say or do something they didn’t.

With tools for making deepfakes now widely available and relatively easy to use, many also worry that they will be used to spread dangerous misinformation. Politicians can have other people’s words put into their mouths or made to participate in situations they did not take part in, for example.

That’s the fear, at least. To a human eye, the truth is that deepfakes are still relatively easy to spot. And according to a report from cybersecurity firm DeepTrace Labs in October 2019, still the most comprehensive to date, they have not been used in any disinformation campaign. Yet the same report also found that the number of deepfakes posted online was growing quickly, with around 15,000 appearing in the previous seven months. That number will be far larger now.   ... "

A Consortium for Confidential Computing

Intriguing new effort:

The Confidential Computing Consortium (CCC) brings together hardware vendors, cloud providers, and software developers to accelerate the adoption of Trusted Execution Environment (TEE) technologies and standards.

Join the Consortium as a Member

A common, cross-industry way of describing the security benefits, risks, and features of confidential computing will help users make better choices for how to protect their workloads in the cloud. Of the three data states, “in use” has been less addressed because it is arguably the most complicated and difficult. This is a major change to how computation is done at the hardware level and how we structure programs, operating systems and virtual machines. This cross-industry collaboration will accelerate this transformation in security in the enterprise. To get involved, please email us info@confidentialcomputing.io. ..." 

SiliconAngle gives some perspective:

Accenture, AMD, Facebook and Nvidia sign up to advance ‘Confidential Computing’  By Mike Wheatley in SiliconAngle

An industry group called the Confidential Computing Consortium, which aims to standardize the way that data is encrypted while in use, said today it’s adding 10 more members to its community.  ...
The consortium is working to standardize on a method for ensuring that data can be encrypted while it’s being processed in-memory, without exposing it to other parts of a computer system. Although methods for encrypting data at rest and data in transit have already been widely adopted in the technology industry, there’s still no reliable way to secure information as it’s actually being used.

That may soon change, though, as the consortium is advancing the development of an open-source framework called the Open Enclave Software Development Kit. It’s used to build Trusted Execution Environments for applications that can run on multiple kinds of computing architectures.   .... "

Facebook Acquires Mapillary, Competes with Street View

First I had seen this.  Generalized tool for looking at location.  Commercial users for now.

Facebook acquires Google Street View competitor Mapillary
The street imaging platform will now be accessible to commercial users.
By Rachel England, @rachel_england in Engadget

Facebook has done a fair bit with maps in recent times — last year it opened up its AI tool to OpenStreetMap users and announced its plans to recreate an AR planet based on a 3D map of the world. More recently, it’s deployed an interactive map to keep users up-to-date with the spread of COVID-19. Now, it’s acquired Google Street View alternative Mapillary. ... "

Facebook's Blenderbot

 Good indication of what Facebook has been doing in this space..  From O'Reilly: 

Facebook open-sources BlenderBot
Facebook AI has open-sourced BlenderBot, an open domain chatbot. It’s said to “feel more human,” because it blends a diverse set of conversational skills—including empathy, knowledge, and personality—together in one system. The model has 9.4 billion parameters, which is 3.6 times more than the largest existing system. Here’s the complete model, code, and evaluation setup.  

Facebook Blender Chatbot Does Open Source

Anything that can make a chatbot more conversational, and have the ability to use and update context
continually is good. 

Facebook releases its 'Blender' chatbot as an open-source project
It could help tomorrow's AI converse more naturally with people

Andrew Tarantola, @terrortola in Engadget

The virtual assistants that inhabit our smartphones are helpful, sure, but they’re not going to pass the Turing test any time soon. They’re designed for understanding specific commands and actions like checking on restaurant reservations or getting updates on the weather, rather than, say, carrying on an in-depth conversation with a human. But chatbots could soon become far more loquacious thanks to Facebook, which this morning released a startlingly lifelike chatbot that it’s been developing, dubbed Blender, as an open-source resource for AI research.

Facebook has been pouring money and resources into its Natural Language Processing technologies for a few years now and those efforts appear to have paid off. The company claims that Blender is the single largest open-source chatbot created to date. It’s been trained on a whopping 9.4 billion parameters -- nearly 4x as many as Google’s Meena and more than 10x as many as the previous largest OS chatbot available on the internet.

“One of the recent findings in the area of NLP, and AI in general, has been that as you scale, as these neural network models larger and larger, they tend to perform better,” Stephen Roller, a research engineer at Facebook’s AI lab (FAIR), told Engadget. “We had a number of issues when we were trying to train this thing. When you start to get that large, these things no longer are able to fit on a single GPU anymore.”  .... "

See more on their Blended Talk Ability.