U.S. and E.U. Complete Long-Awaited Deal on Sharing Data

Good Direction.? 

U.S. and E.U. Complete Long-Awaited Deal on Sharing Data

By The New York Times, July 11, 2023

Credit: Ksenia Kuleshova/The New York Times

A deal to ensure that data from Meta, Google and scores of other companies can continue flowing between the United States and the European Union was completed on Monday, after the digital transfer of personal information between the two jurisdictions had been thrown into doubt because of privacy concerns.

The decision adopted by the European Commission is the final step in a yearslong process and resolves — at least for now — a dispute about American intelligence agencies' ability to gain access to data about European Union residents. The debate pitted U.S. national security concerns against European privacy rights.

The accord, known as the E.U.-U.S. Data Privacy Framework, gives Europeans the ability to object when they believe their personal information has been collected improperly by American intelligence agencies. An independent review body made up of American judges, called the Data Protection Review Court, will be created to hear such appeals.

Didier Reynders, the European commissioner who helped negotiate the agreement with the U.S. attorney general, Merrick B. Garland, and Commerce Secretary Gina Raimondo, called it a "robust solution." The deal sets out more clearly when intelligence agencies are able to retrieve personal information about people in the European Union and outlines how Europeans can appeal such collection, he said.

From The New York Times

View Full Article  

NASA Engineers Help Create Virtual World of Data

 Virtual vs Actual

NASA Engineers Help Create Virtual World of Data

NASA

Andrew Wagner, June 23, 2023

Engineers at the U.S. National Aeronautics and Space Administration Jet Propulsion Laboratory (JPL) and the California Institute of Technology (Caltech) partnered to develop software for virtually exploring relationships between points of data using artificial intelligence and three-dimensional visualization. Explained JPL's Scott Davidoff, "We made a data world where an analyst could look at any science or engineering problem and see patterns and correlations more clearly than they can in a flat version." This process, called intelligent exploration, can extract immediate insights beyond the capabilities of two-dimensional graphs "because it's literally drag and drop," according to Caltech's Ciro Donalek.  ... ' 

Sound Vibrations Can Encode, Process Data Like Quantum Computers

Remarkable ...

Sound Vibrations Can Encode, Process Data Like Quantum Computers

By New Scientist, May 25, 2023

University of Arizona researchers demonstrated that trapping sound in a simple mechanical device can imitate certain properties of quantum computers.

The researchers built an object that could act like a qubit by gluing together three aluminium rods over half a meter long each, then generated vibrations at one end and detected them at the other.

They observed that information could be input in the "phi-bits" (localized "chunks" of sound produced in the rods) by tuning the sound, and that the phi-bits could be forced into a superposition (a mixture of their individual states).

The researchers used the system to perform simple computations, as well as producing quantum-like states.

From New Scientist

View Full Article -    May Require Paid Subscription

Vector Databases

Databases and Generative.

How vector databases can revolutionize our relationship with generative AI

Rick Hao, Speedinvest  @hao_rick

April 30, 2023 8:20 AM

Person's face with swirling dots and lines. Vector database and AI concept.

Generative AI has received a lot of attention already this year in the tech world and beyond. Whether it’s ChatGPT’s prose or Stable Diffusion’s art, 2022 provided an insight into the potential for AI to disrupt creative industries.

Want must read news straight to your inbox?

But behind the headlines, 2022 brought an even more important development in AI: the rise of the vector database.

While their impacts are less immediately obvious, the adoption of vector databases could completely upend the way we interact with our devices, along with dramatically improving our productivity in a vast range of administrative and clerical tasks.

Ultimately, vector databases will be essential infrastructure in bringing about the societal and economic changes promised by AI.

EVENT

Transform 2023

Join us in San Francisco on July 11-12, where top executives will share how they have integrated and optimized AI investments for success and avoided common pitfalls.

But what is a vector database? To understand that, we have to make sense of the underlying problem it addresses: unstructured data.

The database dilemma

Databases are one of the software industry’s longest-lasting and most resilient verticals. The total spend on databases and database management solutions doubled from $38.6B in 2017 to $80B in 2021. And since 2020, databases have only further entrenched their position as one of the most rapidly growing software categories, owing to further digitization following mass shifts to remote working.

However, the modern database is still constrained by a problem that has persisted for decades: the problem of unstructured data. This is the up to 80% of data stored globally that has not been formatted, tagged or structured in a way that allows it to be rapidly searched or recalled. 

For a simple analogy of structured vs. unstructured data, think of a spreadsheet with multiple columns per row. In this case, a row of “structured data” has all the relevant columns filled in, whereas a row of “unstructured data” does not. In the case of the unstructured entry, it may be that the data has been automatically imported into the first column of the row; someone now needs to break up that cell and populate data into relevant columns.

Why is unstructured data a problem? In short, it makes it harder to sort, search, review and use information in a database. However, our understanding of unstructured data is relative to how data is usually structured. ... ' 

GPT is Back in Italy

This block was reported some time ago and now has been resolved.  How this will long time connects with Euro GDPR is still unclear. 

ChatGPT accessible again in Italy, By Shiona McCallum,Technology reporter   in the BBC

Access to the ChatGPT chatbot has been restored in Italy.

It was banned by the Italian data-protection authority at the start of April over privacy concerns.

It maker, OpenAI, which is backed by Microsoft, said it had successfully "addressed or clarified" the issues raised.  It said its privacy policy was now accessible to people before they registered for ChatGPT and there was a new tool to verify the age of users.

The Italian data-protection authority, also known as Garante, had temporarily restricted the chatbot and launched a probe over the artificial intelligence application's suspected breach.   As Garante had accused OpenAI of failing to check the age of ChatGPT's users who are supposed to be aged 13 or above, OpenAI said it would offer a tool to verify users' ages in Italy upon sign-up.

OpenAI explained that it would also provide a new form for European Union users to exercise their right to object to its use of personal data to train its models.

The Italian regulator told the BBC it "welcomed the measures OpenAI implemented" but called for even more compliance.  In particular, the spokesperson said, around "implementing an age verification system and planning and conducting an information campaign to inform Italians of what happened as well as of their right to opt-out from the processing of their personal data for training algorithms."

Garante said it would carry on its "fact-finding activities regarding OpenAI also under the umbrella of the ad-hoc task force that was set up by the European Data Protection Board."   An OpenAI spokesperson said it appreciated the Garante for being collaborative, and that it would continue ongoing constructive discussions.

Millions of people have used ChatGPT since it launched in November 2022.   It can answer questions using natural, human like language and it can also mimic other writing styles. Microsoft has spent billions of dollars on it and it was added to Bing last month.   It has also said that it will embed a version of the technology in its Office apps, including Word, Excel, Powerpoint and Outlook.

Bard, Google's; rival artificial-intelligence chatbot, is now available, but only to specific users over the age of 18.

Exploratory Data Analysis with Pandas Python 2023

Nicely done general,  beginners piece on data analysis

by Rob Mulla

 https://www.youtube.com/watch?v=xi0vhXFPegw

122,509 views  Premiered Dec 31, 2021  Medallion Python Data Science Coding Videos

In this video about exploratory data analysis with pandas and python, Kaggle grandmaster Rob Mulla will teach you the basics of how to explore data using python and pandas. Exploratory Data Analysis it a necessary tool for any data scientist. Pandas is a MUST for anyone getting into data science with python. Python is the #1 coding language for data science and has been growing over the years as an essential tool, with Pandas being the main data wrangling module. Kaggle Grandmaster Rob goes over it all in this video. In this video we discuss the basics of how to use explore data including...

Timestamps:

00:00 Introduction

01:00 Imports and reading data

03:35 Data Understanding

06:40 Data Preparation

20:57 Feature Understanding

27:35 Feature Relationships

35:30 Asking a Question about the Data

40:00 Final Thoughts

Follow me on twitch for live coding streams: https://www.twitch.tv/medallionstallion

Commercial Sat Constellations

Been following for some time,   Current and future plans.   Have watched many launches. Ultimate data leverage goals

In  McKinsey:  Expectations versus reality: Commercial-satellite constellations

April 14, 2023 | Commentary

By  Ryan Brukardt,  Jesse Klempner, and Brooke Stokes

Many commercial-satellite operators are still creating overly ambitious plans, but more companies are now making steady progress toward their launch goals.

Interest in building and launching commercial-satellite constellations continues to be high because of recent technological advances and cost reductions, combined with better access to capital. While some industry analysts feared that private investment in the space sector would plummet from the record highs achieved in 2021, the decrease in 2022 was less severe than expected and capital inflows were the second-highest on record. Excitement about commercial-satellite constellations spans multiple sectors, with companies exploring wide-ranging use cases, ranging from national security to climate monitoring to improve life on Earth.

McKinsey Global Private Markets Review 2023: Private markets turn down the volume

In 2021, we conducted an analysis to determine if constellation operators were able to translate their ambitious plans for satellite count and launch into reality. The results showed that operators had not yet placed an asset in orbit in more than half of the announced constellations. We conducted an updated analysis in early 2023 to determine whether interest in satellite constellations remains strong and whether operators are now more successful in achieving their goals. Three findings stand out (exhibit):   ... '  (much more) 

EU Digital Strategy

 So many directions to regulate.    Note considering the influence US Law

The EU digital strategy: The impact of data privacy on global business

McKinseu:  March 1, 2023 | Commentary

By  Daniel Mikkelsen, Sebastian Scheurle,   Henning Soller , and Malin Strandell-Jansson

New data regulations from the European Union require organizational attention, and three key steps can help navigate the data privacy landscape.

The data regulations in the European Union (EU) have recently received significant attention specifically due to the advent of the General Data Protection Regulation and the rulings around Schrems II—whereby the Court of Justice of the European Union found that the protection of personal data had limitations due to domestic law in the United States—as well as the access and use by US public authorities of personal data transferred from the EU, and recent developments such as e-privacy.

While these developments have led to major changes in data privacy, one of the other goals of the regulation—to establish a market for data and facilitate data exchange between companies—has not been reached to date.

This lack of action has led to the potential for further regulatory activity to define an agenda for how to uplift the data capabilities of European companies, create a market for data, and regulate activities around AI. These activities are typically summarized as the EU digital strategy. While regulation adds further requirements and obligations to any data-enabled business, it also creates an opportunity for competitive advantages for those that best derisk their data transformations.

The EU digital strategy offers organizations both challenges and opportunities, but these regulations will likely continue to evolve, so organizations should remain aligned with the regulatory process. ... ' 

Surfing Research Data Waves

 Key Data and Metadata gathered and collected for future use.

Surfing the Research Data Wave

University of Stuttgart (Germany), February 10, 2023

A new data exchange format developed by a team led by researchers at Germany's University of Stuttgart aims to provide a means for accessing and reusing large volumes of complex research data. With EnzymeML, researchers can record the comprehensive results of an enzymatic experiment and store that data in a structured and standardized manner, which ensures the machine-readable documents are interoperable and can be reused by other research groups. EnzymeML also allows for seamless communications between experimental platforms, electronic lab notebooks, enzyme kinetics modeling tools, publication platforms, and enzymatic reaction databases. Said University of Stuttgart's Simone Lauterbach, "We demonstrate the feasibility and usefulness of the EnzymeML toolbox using six scenarios where data and metadata from various enzymatic reactions is collected, analyzed, and uploaded to public databases for future use." ... .'

Putting Limits to Vehicle Tech

Limits of autonomous vehicle tech?

Carmakers Are Pushing Autonomous Tech. This Engineer Wants Limits.   By The New York Times.February 16, 2023

Systems like General Motors Super Cruise can steer, brake and accelerate vehicles on their own.

Companies sometimes market these systems as if they made cars autonomous. But their legal fine print requires drivers to stay alert and be ready to take control of the vehicle at any time.

Last fall, Missy Cummings sent a document to her colleagues at the National Highway Traffic Safety Administration that revealed a surprising trend: When people using advanced driver-assistance systems die or are injured in a car crash, they are more likely to have been speeding than people driving cars on their own.

The two-page analysis of nearly 400 crashes involving systems like Tesla's Autopilot and General Motors' Super Cruise is far from conclusive. But it raises fresh questions about the technologies that have been installed in hundreds of thousands of cars on U.S. roads. Dr. Cummings said the data indicated that drivers were becoming too confident in the systems' abilities and that automakers and regulators should restrict when and how the technology was used.

People "are over-trusting the technology," she said. "They are letting the cars speed. And they are getting into accidents that are seriously injuring them or killing them."

From The New York Times

View Full Article    

Training Data from Audiobooks?

 Ultimately its all about training data.  And the quality of that data. Of course a Google gets lots of language data, but its it the right quality and type?    Authors giving up data that could replace them.  300 audiobooks probably not near enough.  

Audiobook Narrators Fear Apple Used Their Voices to Train AI   By Wired, February 16, 2023

An email to SAG-AFTRA union members seen by WIRED said the two companies had agreed to stop all “use of files for machine learning purposes” for union members affected

Gary Furlong, a Texas-based audiobook narrator, had worried for a while that synthetic voices created by algorithms could steal work from artists like himself. Early this month, he felt his worst fears had been realized.

Furlong was among the narrators and authors who became outraged after learning of a clause in contracts between authors and leading audiobook distributor Findaway Voices, which gave Apple the right to "use audiobooks files for machine learning training and models." Findaway was acquired by Spotify last June.

Some authors and narrators say they were not clearly informed about the clause and feared it may have allowed their work or voices to contribute to Apple's development of synthetic voices for audiobooks. Apple launched its first books narrated by algorithms last month. "It was very disheartening," says Furlong, who has narrated over 300 audiobooks and is one of more than a dozen narrators and authors who told WIRED of their concerns with Findaway's agreement. "It feels like a violation to have our voices being used to train something for which the purpose is to take our place," says Andy Garcia-Ruse, a narrator from Kansas City.

From Wired

View Full Article   

Sharing Health Data without consumer Consent

Sharing Health Data without consumer Consent

ACM NEWS

FTC Accuses GoodRx of Sharing User Data Without Consent

By CNet, February 2, 2023

GoodRX users may have given up more than they thought

GoodRx will pay $1.5 million and be barred from sharing user data with outside companies for advertising purposes under a deal that would settle allegations that it shared some of its users' most intimate health-related information with companies like Facebook and Google.

The Federal Trade Commission characterized the action, which is pending approval by a federal court, as the first of its kind under its Health Breach Notification Rule, adding that the agency won't hesitate to use its full legal authority to take action against companies that willingly misuse or exploit consumer data.

"Digital health companies and mobile apps should not cash in on consumers' extremely sensitive and personally identifiable health information," Samuel Levine, director of the FTC's Bureau of Consumer Protection, said in a statement.

According to the FTC's complaint, GoodRx, which helps users find deals on prescription drugs and offers telehealth and other health-related services, shared its users' data with outside companies for advertising purposes, despite promising it wouldn't.

From CNet   

A Photo Bot Feeds Data from JWST Space Telescope

 Instructive,   A means to to feed data from multiple sources?  50 GB a day!  What can we learn form this?

Telescope broke the universe

Scientists were in awe of the flood of data that arrived when the new space observatory booted up.

A clutch of massive stars takes center stage in this mosaic image of the Tarantula Nebula, captured with JWST's Near Infrared Camera. They are surrounded by, and will help sculpt, clouds of gas and dust—the raw material for yet more stars. ... '

NASA, ESA, CSA, STSCI, WEBB ERO PRODUCTION TEAM   by Jonathan O'Callaghan archive page,  January 21, 2023

https://twitter.com/jwstphotobot

JWST Photo Bot   @JWSTPhotoBot   Automated by @adamlastowka

I post public data from JWST's MIRI and NIRCAM instruments!

INFO: https://github.com/Rachmanin0xFF/jwst-twitter-bot

Earth-Sun L2 Halo Orbit Joined July 2022

2,853 Followers  ... ' 

Smart Appliances Could Stop Working After Two Years Due to Lack of Software Updates

Well known, but worth a caution.  Even for large companies.   I worked with a device that allowed me to incidentally get useful testing data from a company.   They stopped making it and the data stream stopped.

Smart Appliances Could Stop Working After Two Years Due to Lack of Software Updates  By BBC News, January 17, 2023

U.K. informed consumer brand Which? warns smart appliances could stop operating after only two years due to a cessation of software updates from manufacturers.

Brands are "abandoning" products like TVs, smartphones, and washing machines, while the risk of hacking also escalates as the appliances age.

Which? said, "Hardly any brands even came close to matching their expected lifespan" with their smart update policies, while others did not clarify the duration of support.

Terminating software support for these products means their functionality cannot improve, and security patches are among the updates manufacturers provide.

"Manufacturers must up their game and provide vital tech updates for their smart products for longer, otherwise the government should consider further intervention in this area, including specifying a minimum number of years that these products have to be supported for," said Which?'s Rocio Concha.

In BBC News 

Particles of Light and Fluid Flow

Not quite grokking this,  or its value, but considering it.

Particles of Light May Create Fluid Flow, Data-Theory Comparison Suggests

By Brookhaven National Laboratory

December 20, 2022

This graphic shows the energy density at different times during the hydrodynamic evolution of the matter created in a collision of a lead nucleus (moving to the left) with a photon emitted from the other lead nucleus (moving to the right).

The “elliptic flow” pattern was one of the earliest hints that particle collisions at the Relativistic Heavy Ion Collider could create a quark-gluon plasma.

A computational analysis by scientists at the U.S. Department of Energy's Brookhaven National Laboratory and Wayne State University supports the idea that photons colliding with heavy ions can create a fluid of “strongly interacting” particles.

The researchers found calculations defining such a scheme correlate with data collected by the ATLAS detector at Europe's Large Hadron Collider (LHC).

The calculations are based on the hydrodynamic particle flow observed in head-on collisions of various types of ions at the LHC and the Brookhaven Lab's Relativistic Heavy Ion Collider.

Said Brookhaven Lab's Bjoern Schenke, "For these low energy photon-lead collisions, it is important to run a full 3D hydrodynamic model (which is more computationally demanding) because the particle distribution changes more rapidly as you go out in the longitudinal direction."

From Brookhaven National Laboratory  View Full Article

Universities, Rich in Data, Struggle to Capture Its Value

Needs more automatic matching to actual uses, needs, outcomes. 

Universities, Rich in Data, Struggle to Capture Its Value     By UCLA

December 29, 2022

Merging together data from various sources is a major challenge, respondents said.

Universities are struggling to make sense of their vast administrative datasets, according to researchers Christine L. Borgman at UCLA and Amy Brand at MIT.

The two surveyed senior officials focused on university data governance and management to identify a shortage of coordination and expertise. Respondents cited issues about commercial control of internal systems, local capacity for data-informed planning, a lack of databases of record, and few coordinated information management strategies and administrators with data-science backgrounds.

"We unexpectedly found a pervasive void of infrastructure thinking and a relatively limited set of data-informed planning successes," said Brand.

The researchers's study, published in Science, suggests universities could expand infrastructure investments to enhance access, integration, and intelligence, as well as fortify their data management capacity.

From UCLA   View Full Article   

Helicopter on Mars

 Evan Ackerman talks the remarkable success of the Mars helicopter Perseverance. Nicely done NASA.  The idea of a detachable drone shows its power of application,  A means of managing opportunistic data gathering in very unordered spaces.

Mars Helicopter Just Keeps on Going 

Maybe NASA’s little Martian flyer should be the one called Perseverance   By Evan ACKERMAN

The original mission of the Mars Helicopter (named Ingenuity) was to successfully complete a single 30-second long flight on Mars. That happened back in April. After several more successful flights, Ingenuity’s 30-day mission came to an end, but the helicopter was doing so well that NASA decided to keep it flying. Several months later, JPL promised that Ingenuity would “complete flight operations no later than the end of August,” but as of late November, the little helicopter has completed 17 flights with no sign of slowing down. ... ' 

Dealing with Tsunami of Data

Continues.

RisingWave Emerges to Tackle Tsunami of Real-Time Data

Alex Woodie in Datanami

Only the most advanced companies have overcome the technical complexity involved with processing streaming data in real time. One of the vendors aiming to reduce this complexity and make stream data processing available to the masses is RisingWave Labs, which today announced $36 million in financing.

The early days of stream data processing brought us stand-alone systems that were capable of acting upon vast streams of data, and doing so with low latency and reliability. Stream processing frameworks like Apache Storm made headway in addressing these challenges and led the way to more sophisticated frameworks like Apache Flink and others.

Things got significantly more complex when companies realized they needed to know something about the past to take the best action on the newest data, which necessitated the integration of stream processing frameworks with databases or data lakes, where the historical record lived as persisted data. Architectural blueprints, such as the Lambda and Kappa architectures, were proposed to address this unique challenge, but the technical complexity in keeping these dual-path systems running are immense.

Today we’re seeing the emergence of a new category of product–the streaming database–aimed at solving this problem. Instead of running data through a dedicated stream processing framework like Storm or Flink, the backers of streaming databases think that all the data processing–including the business end of a streaming big data pipeline like Kafka, Kinesis, or Pulsar–can be handled by the SQL query engine contained in a relational database.

RisingWave is a Postgres-compatible database developed to process data streams in the cloud (Image courtesy RisingWave Labs)

That’s the approach taken with RisingWave, a new open source streaming database that emerged just over a year ago. Yingjun Wu, a former AWS and IBM engineer, created RisingWave as a cloud-native database with the goal of providing the bernefits of stream processing without the technical complexity inherent with stream processing frameworks.

“Existing open-source systems are very costly to deploy, maintain, and use in the modern cloud environment,” Wu, who is the CEO of RisingWave Labs, says in a press release today. “Our goal is not to build yet another streaming system that is 10X faster than existing systems, but to deliver a simple and cost-effective system that allows everyone to benefit from stream processing.”

Developed in Rust, RisingWave is a Postgres-compatible database can do many of the things that stream processing frameworks do, but within the context and control of a familiar relational database running in the cloud and the SQL language, according to Wu, who has a PhD from National University of Singapore and was also a visiting PhD at Carnegie Mellon University.

“[RisingWave] consumes streaming data, performs continuous queries, and maintains results dynamically in the form of a materialized view,” Wu says in a blog post earlier this year. “Processing data streams inside a database is quite different from that inside a stream computation engine: streaming data are instantly ingested into data tables; queries over streaming and historical data are simply modeled as table joins; query results are directly maintained and updated inside the database, without pushing into a downstream system.”

The open source project, which available on GitHub via an Apache 2.0 license, is being adopted by organizations for a range of uses, including real-time analytics and alerting; IoT device tracking; monitoring user activity; and online application data serving. The company, which changed its name from Singularity Data three weeks ago, recently unveiled the beta of a hosted commercial version of RisingWave; it’s slated to become generally available next year.  ... ' 

Lost Something? Search 91.7 Million Files from the '80s, '90s, 2000s

Seeing it all?

Lost Something? Search 91.7 Million Files from the '80s, '90s, 2000s By Ars Technica, October 21, 2022

The files on Discmaster come from the Internet Archive, uploaded by thousands of people over the years.

A new website allows users to sift through 91.7 million computer files from CD-ROM releases and floppy discs dating back to the 1980s.  Hosted by tech archivist Jason Scott, the Discmaster site is the work of a group of anonymous programmers and features images, text documents, music, games, shareware, videos, and more from the Internet Archive.

Discmaster allows users to search by file type, format, source, file size, file date, and other criteria.

Said Scott, "The value proposition is the value proposition of any freely accessible research database." Much of the file format conversion is performed on the back end, to make the vintage files more accessible.

From Ars Technica

View Full Article    

Sensor System for IoT Devices Integrates Processing, Computing to Save Energy, Protect Data

Smart Sensors that store and processes data.

Sensor System for IoT Devices Integrates Processing, Computing to Save Energy, Protect Data

University of Virginia Engineering

Karen Walker, October 11, 2022

The University of Virginia's Kyusang Lee and colleagues are exploring a field Lee calls the artificial intelligence of things with the development of a smart sensor that stores and processes data. The smart sensor will sit at the edge of a device, which itself sits at the outer reaches of a wireless network. It can detect and process diverse signals that emulate human biology, including imagery for vision; ultrasound for hearing; pressure and strain linked to motion and touch; and virus detection via chemical sensing. Said the University of Virginia’s Kyusang Lee, “I believe our sensor will be especially useful in robotics that rely on combined sensory inputs and real-time integrated processing.”  .... '