Automating and Optimizing Experiment Data Collection

 Collecting data from processes, and using some automatic method of choosing, pre-analysis, cleansing, visualizing and tagging, associating with metadata .... can be very useful.   Here a more complex example.

SMART Algorithm Makes Beamline Data Collection Smarter
By Lawrence Berkeley National Laboratory

The "data deluge" in scientific research stems in large part from the growing sophistication of experimental instrumentation and optimizing tools — often using machine- and deep-learning methods — to analyze increasingly large data sets. But what is equally important for improving scientific productivity is the optimization of data collection — aka "data taking" — methods.

Toward this end, Marcus Noack, a postdoctoral scholar at Lawrence Berkeley National Laboratory in the Center for Advanced Mathematics for Energy Research Applications (CAMERA), and James Sethian, director of CAMERA and Professor of Mathematics at UC Berkeley, have been working with beamline scientists at Brookhaven National Laboratory to develop and test SMART (Surrogate Model Autonomous Experiment), a mathematical method that enables autonomous experimental decision making without human interaction. A paper describing SMART and its application in experiments at Brookhaven's National Synchrotron Light Source II (NSLS-II) are described in "A Kriging-Based Approach to Autonomous Experimentation with Applications to X-Ray Scattering," published in Scientific Reports.

"Modern scientific instruments are acquiring data at ever-increasing rates, leading to an exponential increase in the size of data sets," says Noack, lead author on the paper. "Taking full advantage of these acquisition rates requires corresponding advancements in the speed and efficiency not just of data analytics but also experimental control."

The goal of many experiments is to gain knowledge about the material that is studied, and scientists have a well-tested way to do this: they take a sample of the material and measure how it reacts to changes in its environment. User facilities such as Brookhaven's NSLS-II and the Center for Functional Nanomaterials offer access to high-end materials characterization tools. The associated experiments are often lengthy, and complicated procedures and measurement time is precious. A research team might only have a few days to measure their materials, so they need to make the most of each step in each measurement.  .... "