Lamboozling Attackers: A New Generation of Deception

Previous piece I just got to, worthwhile.

Lamboozling Attackers: A New Generation of Deception

By Kelly Shortridge, Ryan Petrich

Communications of the ACM, June 2022, Vol. 65 No. 6, Pages 44-53  10.1145/3498578

Deception is a powerful resilience tactic that provides observability into attack operations, deflects impact from production systems, and advises resilient system design. A lucid understanding of the goals, constraints, and design trade-offs of deception systems could give leaders and engineers in software development, architecture, and operations a new tactic for building more resilient systems—and for bamboozling attackers.

Unfortunately, innovation in deception has languished for nearly a decade because of its exclusive ownership by information security specialists. Mimicry of individual system components remains the status-quo deception mechanism despite growing stale and unconvincing to attackers, who thrive on interconnections between components and expect to encounter systems. Consequently, attackers remain unchallenged and undeterred.

This wasted potential motivated our design of a new generation of deception systems, called deception environments. These are isolated replica environments containing complete, active systems that exist to attract, mislead, and observe attackers. By harnessing modern infrastructure and systems design expertise, software engineering teams can use deception tactics that are largely inaccessible to security specialists. To help software engineers and architects evaluate deception systems through the lens of systems design, we developed a set of design principles summarized as a pragmatic framework. This framework, called the FIC trilemma, captures the most important dimensions of designing deception systems: fidelity, isolation, and cost.

The goal of this article is to educate software leaders, engineers, and architects on the potential of deception for systems resilience and the practical considerations for building deception environments. By examining the inadequacy and stagnancy of historical deception efforts by the information security community, the article also demonstrates why engineering teams are now poised—with support from advancements in computing—to become significantly more successful owners of deception systems.

Deception: Exploiting Attacker Brains

In the presence of humans (attackers) whose objectives are met by accessing, destabilizing, stealing, or otherwise leveraging other humans' computers without consent, software engineers must understand and anticipate this type of negative shock to the systems they develop and operate. Doing so involves building the capability to collect relevant information about attackers and to implement anticipatory mechanisms that impede the success of their operations. Deception offers software engineering teams a strategic path to achieve both outcomes on a sustained basis.  .... '

Detecting Deception with Machine Leaning

Quite interesting, but my guess, if something like this sees much use, will be heavily regulated.  Below intro, more at the link. 

Detecting Deception

By Sandrine Ceurstemont,   Commissioned by CACM Staff  in CACM

 People are not good at detecting when someone is lying. Studies have shown that our ability to perceive deception is barely greater than chance.  Wasiq Khan, a senior lecturer in artificial intelligence and data sciences at Liverpool John Moores University in the U.K., thinks that is partly because it requires the ability to identify complex clues in speech, facial movements, and gestures, attributes that he says "cannot be observed by humans easily."

Automated systems that use machine learning may be able to do better. Khan and his colleagues developed such a system while working on a project for the EU, where the aim was to explore new technologies that could be put in place to improve border control. They examined whether deception could be detected automatically from eye and facial cues, such as blink rate and gaze aversion. "I wanted to investigate whether face movements or eye movements are important," says Khan.

The team recorded videos of 100 participants to use as their dataset. The volunteers were filmed while role-playing a scenario that might occur at a nation's port of entry, in which they are asked about what they had packed in their suitcase by an avatar controlled by the researchers in another room. Half of the participants were asked to lie, and the other half were told to be truthful.   

The videos were then analyzed using an automated system called Silent Talker. It examined each video frame and used an algorithm to extract information from the interviewees about 36 face and eye movements. Results were noted in binary format where 1 could be assigned when the person's eyes were closed, for example, and 0 if they were open.  The team them tried to determine which facial and eye features were correlated with deception by using various clustering algorithms. "The video analysis is complex," says Khan.

The algorithms identified features that seemed to be most important for detecting deception, which all involved tiny eye movements. The team then trained three machine learning algorithms using both the more significant features and the total set of attributes. Eighty percent of the dataset was used for training, including 40 truthful and 40 deceitful interviews, while the remaining 20% was held for testing.

Khan and his colleagues found the machine learning methods were all able to predict deception quite well from the identified features. Overall accuracy ranged from 72% to 78% depending on the method, where the greatest accuracy was obtained by focusing solely on eye movements. "We identified that eye features are important and contain significant clues for deception," says Khan.  ... ' 

Deceiving AI

 Made me think, usually models we create are to determine some state, current or future, to be more accurate.   But now we can make models that have more precisely deceptive results.     Yes, can see why DARPA is interested.  Includes  a visual overview.

Deceiving AI   By Don Monroe

Communications of the ACM, June 2021, Vol. 64 No. 6, Pages 15-16 10.1145/3460218

Over the last decade, deep learning systems have shown an astonishing ability to classify images, translate languages, and perform other tasks that once seemed uniquely human. However, these systems work opaquely and sometimes make elementary mistakes, and this fragility could be intentionally exploited to threaten security or safety.

In 2018, for example, a group of undergraduates at the Massachusetts Institute of Technology (MIT) three-dimensionally (3D) printed a toy turtle that Google's Cloud Vision system consistently classified as a rifle, even when viewed from various directions. Other researchers have tweaked an ordinary-sounding speech segment to direct a smart speaker to a malicious website. These misclassifications sound amusing, but they could also represent a serious vulnerability as machine learning is widely deployed in medical, legal, and financial systems.

The potential vulnerabilities extend to military systems, said Hava Siegelman of the University of Massachusetts, Amherst. Siegelman initiated a program called Guaranteed AI Robustness against Deception (GARD) while she was on assignment to the U.S. Defense Advanced Research Projects Agency (DARPA). To illustrate the issue to colleagues there, she said, "I showed them an example that I did, and they all started screaming that the room was not secure enough." The examples she shares publicly are worrisome enough, though, such as a tank adorned with tiny pictures of cows that cause an artificial intelligence (AI)-based vision system to perceive it to be as a herd of cows because, she said, AI "works on the surfaces."

The current program manager for GARD at DARPA, Bruce Draper of Colorado State University, is more sanguine. "We have not yet gotten to that point where there's something out there that has happened that has given me nightmares," he said, adding, "We're trying to head that off."

Researchers, some with funding from DARPA, are actively exploring ways to make machine learning more robust against adversarial attacks, and to understand the principles and limitations of these approaches. In the real world, these techniques are likely to be one piece of an ongoing, multilayered security strategy that will slow attackers but not stop them entirely. "It's an AI problem, but it's also a security problem," Draper said. ... '

Towards Deception Detection? In Humans, in Machines? In Crowds?

Following this in the Language Log Blog for some time.  We even looked at large databases of human comments on products. But could such comments really be marked as 'deception'?    And when our brands started to converse with customers, relating their experiences, concerns and needs,  could such conversations be closer to speaking 'truth'?  Or not?  How about if we linked it to other behavioral  cues?   Or is that a violation of privacy?    All that detail, though discussed, was never implemented.  But can it be done better now?   Is there truth in a crowd response?   See the tags under 'deception'  here, which I am about to review.

New directions in deception detection?

March 28, 2021 @ 11:23 am · Filed by Mark Liberman under Nonverbal communication, Psychology of language

Jessica Seigel, "The truth about lying", Knowable Magazine 3/25/2021

You can’t spot a liar just by looking — but psychologists are zeroing in on methods that might actually work

The featured research is a review by Aldert Vrij, Maria Hartwig, and Pär Anders Granhag, "Reading Lies: Nonverbal Communication and Deception", Annual Review of Psychology 2019   :... '

Deception and Concealment Technology

Worked with MITRE way back.  Now it is also about active defense.  Not much revealed here, but its interesting. 

MITRE Shield Matrix Highlights Deception & Concealment Technology

The role that these technologies play in the MITRE Shield matrix is a clear indicator that they are an essential part of today's security landscape.

It's an age-old question: How do you know if you need more security? MITRE has been diligently working to document tactics and techniques to assess security readiness and answer this very challenging question. In late August, MITRE, a nonprofit organization, released a new knowledge matrix, called MITRE Shield, to complement the ATT&CK matrix.

The organization called it "an active defense knowledge base MITRE is developing to capture and organize what we are learning about active defense and adversary engagement." With its focus on active defense measures, MITRE designed Shield to help defenders understand their cybersecurity options and take proactive steps to defend their assets. Among the most common active defense techniques are cyber-deception and concealment technologies, which are featured heavily in the new Shield matrix.  ... "

When Bots Negotiate, Will Humans Deceive?

Interesting piece that approaches the meaning of 'cooperation'.   

When bots do the negotiating, humans more likely to engage in deceptive techniques  by University of Southern California

 Recently computer scientists at USC Institute of Technologies (ICT) set out to assess under what conditions humans would employ deceptive negotiating tactics. Through a series of studies, they found that whether humans would embrace a range of deceptive and sneaky techniques was dependent both on the humans' prior negotiating experience in negotiating as well as whether virtual agents where employed to negotiate on their behalf. The findings stand in contrast to prior studies and show that when humans use intermediaries in the form of virtual agents, they feel more comfortable employing more deceptive techniques than they would normally use when negotiating for themselves

Lead author of the paper on these studies, Johnathan Mell, says, "We want to understand the conditions under which people act deceptively, in some cases purely by giving them an artificial intelligence agent that can do their dirty work for them."

Nowadays, virtual agents are employed nearly everywhere, from automated bidders on sites like eBay to virtual assistants on smart phones. One day, these agents could work on our behalf to negotiate the sale of a car, argue for a raise, or even resolve a legal dispute.

Mell, who conducted the research during his doctoral studies in computer science at USC, says, "Knowing how to design experiences and artificial agents which can act like some of the most devious among us is useful in learning how to combat those techniques in real life."

The researchers are eager to understand how these virtual agents or bots might do our bidding and to understand how humans behave when deploying these agents on their behalf.  ... '

Paper is here:  https://www.jair.org/index.php/jair/article/view/11924

When Your AI Learns to Lie

Some very important questions as these methods evolve.

AI Deception: When Your Artificial Intelligence Learns to Lie

We need to understand the kinds of deception an AI agent may learn on its own before we can start proposing technological defenses

This piece was written as part of the Artificial Intelligence and International Stability Project at the Center for a New American Security, an independent, nonprofit organization based in Washington, D.C. Funded by Carnegie Corporation of New York, the project promotes thinking and analysis on AI and international stability. Given the likely importance that advances in artificial intelligence could play in shaping our future, it is critical to begin a discussion about ways to take advantage of the benefits of AI and autonomous systems, while mitigating the risks. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

In artificial intelligence circles, we hear a lot about adversarial attacks, especially ones that attempt to “deceive” an AI into believing, or to be more accurate, classifying, something incorrectly. Self-driving cars being fooled into “thinking” stop signs are speed limit signs, pandas being identified as gibbons, or even having your favorite voice assistant be fooled by inaudible acoustic commands—these are examples that populate the narrative around AI deception. One can also point to using AI to manipulate the perceptions and beliefs of a person through “deepfakes” in video, audio, and images. Major AI conferences are more frequently addressing the subject of AI deception too. And yet, much of the literature and work around this topic is about how to fool AI and how we can defend against it through detection mechanisms.

I’d like to draw our attention to a different and more unique problem: Understanding the breadth of what “AI deception” looks like, and what happens when it is not a human’s intent behind a deceptive AI, but instead the AI agent’s own learned behavior. These may seem somewhat far-off concerns, as AI is still relatively narrow in scope and can be rather stupid in some ways. To have some analogue of an “intent” to deceive would be a large step for today’s systems. However, if we are to get ahead of the curve regarding AI deception, we need to have a robust understanding of all the ways AI could deceive. We require some conceptual framework or spectrum of the kinds of deception an AI agent may learn on its own before we can start proposing technological defenses.  ... " 
------
" ... I'm not worried about the machine that passes the Turing Test, I'm worried about the machine smart enough to pretend to fail it. ..."

Further Deception Detection

See other models, for example the U of MD, which also been working on this.  Of course lie detection using biometics has been around for a long time, but is still not generally accepted in US courts of law.  Will this be treated similarly?  But then if this detection is integrated with other facial recognition data and rolled into an algorithm?  Implications unclear.

Using Data Science to Tell Which of These People Is Lying    By University of Rochester

University of Rochester researchers are applying data science and an online crowdsourcing framework to read facial and verbal cues for signs of deception.

The Automated Dyadic Data Recorder framework was used to generate the largest publicly available deception dataset currently in existence. Participants sign up on Amazon Mechanical Turk to be assigned the roles of describer or interrogator. The former is displayed as an image they must memorize thoroughly, and the computer instructs them to either lie or truthfully relate the image details. The interrogator then asks the describer a set of irrelevant baseline queries to record individual behavioral differences that are fed to a "personalized model."

The researchers have culled 1.3 million frames of facial expressions from 151 pairs of individuals conducting this experiment, analyzing the information with data science. Among their findings is the detection of five types of smile-related expressions people make in response to questions, including one most frequently associated with lying.    .... "

From University of Rochester

Detecting Deception

Skeptical, like to see more.

A New AI That Detects 'Deception' May Bring an End to Lying as We Know It    Futurism   By Dom Galeon in Futurism in ACM News

Researchers at the University of Maryland (UMD) have developed the Deception Analysis and Reasoning Engine (DARE), which uses artificial intelligence (AI) to autonomously detect deception in courtroom trial videos. The team trained DARE to seek out and classify human micro-expressions, such as "lips protruded" or "eyebrows frown," and analyze audio frequency for vocal patterns that signal whether a person is lying or not. DARE then was tested with a training set of videos in which actors were told to either lie or be honest. UMD's Bharat Singh says DARE outperformed the average person in detecting lies, and notes "a remarkable observation was that the visual AI system was significantly better than common people at predicting deception." Singh estimates DARE scored an area under the curve (AUC) of 0.877, which rose to 0.922 when combined with human annotations of micro-expressions, while ordinary people score an AUC of 0.58. .... "