Computer-Related Risks and Remediation Challenges

Deep,  even beyond AI, getting deeper? 

Computer-Related Risks and Remediation Challenges

By Peter G. Neumann

Communications of the ACM, June 2023, Vol. 66 No. 6, Pages 28-30  10.1145/3593005

Credit: Zamurovic Brothers

This Inside Risks column focuses on some of our computer technologies that are directly or tangentially involved in undesirable misuses, and what effective remedies might be desirable. The following enumeration is illustrative: the items are by no means comprehensive, and some cases fall into multiple categories.

Uses of technology to address problems that are otherwise inherently not purely technological: for example, cryptocurrencies being used to address dissatisfaction with financial systems perceived as rigged in favor of Wall Street and banks, including money-laundering and other illegal activities used to avoid law enforcement.a Increases in electronic gambling (including offshore) are yet another step toward depriving addicts and others of their well-being.

Uses of technology to facilitate crimes that were hitherto not technology-oriented: for example, online spear-phishing scams with ransomware and demands for cryptocurrency payments, with no real assurance of ultimate recovery in the absence of demonstrable defenses. Law enforcement seems to not have much leverage here, and offshore attacks make the problems even more difficult.

Uses of technology with well-defined legitimate purposes but poorly established or administered foundations; for example, elections that have inadequate oversight and no worthy audit trails, irrespective of whether administered in-person or using the Internet. Even the compositional trustworthiness of integrated electronic systems (voting machines, ballot scanners, vote counters) needs much more assurance from a total-system perspective, given the interfaces among the components may themselves be compromised.b Furthermore, abuses of social media with character assassination and disinformation—incuding Chatbots—are creating enormous problems. Misuses of digital commerce also falls into this category.

Uses of technology that is not sufficiently trustworthy for its intended application needs. This is particularly relevant to high-end national security and life-critical applications, with respect to security, privacy, system and network integrity, human safety, high-probability system survivability, and more. Artificial intelligence is seen by some as a panacea, even when embedded in untrustworthy systems and networks whose compromise might in turn decrease its integrity, total-system safety, and predictability. Of course, these problems can also arise in many less-critical applications where exploits are unfortunately surprisingly easy to perpetrate—as in the ubiquity of the Internet of Almost Everything where very little assurance exists today.

Uses of technology that has been compromised in favor of questionable business models: for example, targeted advertising in social media and gaming that results in widespread privacy violations, devious operation, greed, and obliviousness to the risks. Zeynep Tufekci's op-ed in The New York Times, "The Shameful Secret of Southwest's Failure," (Jan. 5, 2023), examines Southwest Airlines' repeated failures to upgrade their archaic computer software, which resulted in the recent total system-wide meltdown. The drive to get self-driving cars on the road quickly appears to be quite controversial, and full of dangerous behavior. Short-term optimization and failure to consider long-term risks seem to be much more important to many organizations than long-term stability. The book by Earl Boebert and James Blossom, Deepwater Horizon: A Systems Analysis of the Macondo Disaster, Harvard Press illustrates hasty iatrogenic technological remediations motivated by a compromised business model.

This list highlights just a few potential types of misuses of technology that remain widespread from year to year, with relatively few repercussions—a long-time topic in the ACM Risks Forum.c One theme that runs through the items listed here (and in previous Communications Inside Risks columns) is that today's technology is not trustworthy enough for many critical applications, even if it were used carefully. Some of these concerns could be addressed in the future by increasing research and development in system assurance, as suggested in the June 2022 Communications Inside Risks column "Total-System Trustworthiness." This would require at least better hardware and better software engineering practices, and greater oversight. Other issues reach way outside of technology, but are seriously exacerbated by the Internet, the Dark Web, rampant disinformation, a general lack of risks awareness, and other factors.  ... '