Deep Learning for Material Design

 Notice the advanced methods being used, like genetic algorithms to improve training.  Use for materials.  Material design continues to be a strong area of use.   See my 'materials' tag.  Technical.

Deep Learning Framework to Enable Material Design in Unseen Domain

KAIST (South Korea), September 29, 2021

Researchers at South Korea's KAIST and the University of California, Berkeley have developed a framework that uses a deep neural network to facilitate more efficient material or structure design beyond the domain of the initial training set. The method compensates for the weak predictive power of neural networks.

This involves three steps:

• using genetic algorithms to search for candidates with improved properties close to the training set, and mixing superior designs in the training set;

• determining whether the candidates actually have improved properties, and using data augmentation to duplicate validated designs and expand the training set; and

• using transfer learning to update the neural network with newly generated superior designs to broaden the reliable prediction domain.

Researchers are using the optimization framework to design metamaterial structures, segmented thermoelectric generators, and optimal sensor distributions.  ... " 

Automating AI By Evolution

Akin to genetic algorithms, which we experimented with, but found that the combinatorics (very large number of solutions)  of real world problems were too large or unstable for us to practically use.  I can see the similarities to what is presented here, so worth following.    Though would seem to have the same issues.  A move towards the automation of  AI approaches. Technical.

AI Is Evolving All by Itself
in Science
By Edd Gent

Google's Quoc Le and colleagues have designed a program that borrows concepts from Darwinian evolution, including survival of the fittest, to assemble artificial intelligence (AI) that generationally improves with effectively no human input. The AutoML-Zero program generates 100 candidate algorithms by randomly combining mathematical operations, then tests them on a simple task, like an image-recognition problem. AutoML-Zero compares the algorithms' performance to that of hand-designed algorithms, with copies of top-performing algorithms mutated by randomly replacing, editing, or deleting some of their code to create variations; these ‘offspring’ are added to the population while older algorithms are removed. AutoML-Zero was able to reproduce decades of AI research in days. Le said, "Our ultimate goal is to actually develop novel machine learning concepts that even researchers could not find."  ... "

Zappos Uses Genetic Algorithms

Surprising application in place and an indication of multiple algorithms in parallel.

At Zappos, Algorithms Teach Themselves 
The Wall Street Journal  (with paywall)
Jared Council

Online shoe and clothing retailer Zappos sees promise in a self-learning algorithm's ability to address the problem of its search engine producing irrelevant results. Zappos' chief data scientist Ameen Kazerouni said several years ago his team began testing a genetic algorithm, which has since become critical to boosting the search engine's relevancy. Genetic algorithms generate various solutions to a problem, using natural-selection principles like reproduction and mutation to return the optimal or "fittest" solution. The algorithms were designed to parse out the intent of a search phrase, with those that perform best on an internal "relevance test," which models how users engage with search results, having the greatest odds of having their traits inherited by the next generation. Zappos uses three genetic algorithm engines in parallel to generate better search results.   .... " 

Genetic Algorithms Revisited

We used Genetic Algorithms a few times,  but turned out it was rarely useful for the kind of marketing and supply chain problems we encountered.    Often because the problems contained much metadata complexity that was hard to incorporate.   GA's are a way to simulate problems to get less than optimal solutions, when you can readily state the problem parameters.    Note the mention below that GA's have become a way to look for optimal 'hyperperameters' used in machine learning methods.   So they are back in vogue.

Tags: Algorithms, Evolutionary Algorithm, Explained, Genetic Algorithm, Key Terms, Optimization

Genetic Algorithms.  Terms and Motivation   By Matthew Mayo, KDnuggets.

Genetic algorithms, inspired by natural selection, are a commonly used approach to approximating solutions to optimization and search problems. Their necessity lies in the fact that there exist problems which are too computationally complex to solve in any acceptable (or determinant) amount of time.

Take the well-known travelling salesman problem, for example. As the number of cities involved in the problem grow, the time required for determining a solution quickly becomes unmanageable. Solving the problem for 5 cities, for example, is a trivial task; solving it for 50, on the other hand, would take an amount of time so unreasonable as to never complete.

It turns out that approximating such optimization problems with genetic algorithms is a sensible approach, resulting in reasonable approximations. Genetic algorithms have had a place in the machine learning repertoire for decades, but their recent revival as tools for optimizing machine learning hyperparameters (and traversing neural network architecture search spaces) has brought them to the attention of a new generation of machine learning researchers and practitioners.

This article presents simple definitions for 12 genetic algorithm key terms, in order to help better introduce the concepts to newcomers. .... "

Choosing the Optimal AI Algorithm

Automating aspects of deep learning is an obvious next step.   Note also the method described here is an evolutionary method (Aka Genetic Algorithm).    Consider also that the selection and testing of data needs to be addressed.  That's usually harder because there are more choices involved.   See the blog post and paper referred to below ...

IBM’s new system automatically selects the optimal AI algorithm   By Kyle Wiggers  in VentureBeat

Not all deep learning systems — that is to say, systems consisting of layered nodes that ingest data, transform it, output it, and pass it on — are created equal. No algorithm is appropriate for every task, and finding the optimal one can be a long and frustrating exercise. Luckily, there’s hope: IBM developed a system that automates the process.

Martin Wistuba, a data scientist at IBM Research Ireland, described in a recent blog post and accompanying paper the method. He claims it’s 50,000 times faster than other approaches, with only a small increase in error rate.

“At IBM, engineers and scientists select the best architecture for a deep learning model from a large set of possible candidates. Today this is a time-consuming manual process; however, using a more powerful automated AI solution to select the neural network can save time and enable non-experts to apply deep learning faster,” he wrote. “My evolutionary algorithm is designed to reduce the search time for the right deep learning architecture to just hours, making the optimization of deep learning network architecture affordable for everyone.”    ... "

(Updated) Optimization using Genetic Methods

In our earliest days,  addressing supply chain and blending type manufacturing problems, we were an optimization shop.  Using the math structure of difficult combinatorial problems to find best solutions based on known goals and constraints.    But if you couldn't glean enough low level structure, we tested genetic methods, described here.   In this era of faster machines and more contextual information even more useful to try today.  Also for certain kinds of structure, also consider Dynamic Programming.  Happen to be examining that again today.

In KDNuggets  By Ahmed Gad, KDnuggets Contributor 

This article gives a brief introduction about evolutionary algorithms (EAs) and describes genetic algorithm (GA) which is one of the simplest random-based EAs.

Selection of the optimal parameters values for machine learning tasks is challenging. Some results may be bad not because the data is noisy or the used learning algorithm is weak, but due to the bad selection of the parameters values. This article gives a brief introduction about evolutionary algorithms (EAs) and describes genetic algorithm (GA) which is one of the simplest random-based EAs.

Introduction

Suppose that a data scientist has an image dataset divided into a number of classes and an image classifier is to be created. After the data scientist investigated the dataset, the K-nearest neighbor (KNN) seems to be a good option. To use the KNN algorithm, there is an important parameter to use which is K. Suppose that an initial value of 3 is selected. The scientist starts the learning process of the KNN algorithm with the selected K=3. The trained model generated reached a classification accuracy of 85%. Is that percent acceptable? In another way, can we get a better classification accuracy than what we currently reached? We cannot say that 85% is the best accuracy to reach until conducting different experiments. But to do another experiment, we definitely must change something in the experiment such as changing the K value used in the KNN algorithm. We cannot definitely say 3 is the best value to use in this experiment unless trying to apply different values for K and noticing how the classification accuracy varies. The question is “how to find the best value for K that maximizes the classification performance?” This is what is called optimization.

In optimization, we start with some kind of initial values for the variables used in the experiment. Because these values may not be the best ones to use, we should change them until getting the best ones. In some cases, these values are generated by complex functions that we cannot solve manually easily. But it is very important to do optimization because a classifier may produce a bad classification accuracy not because, for example, the data is noisy or the used learning algorithm is weak but due to the bad selection of the learning parameters initial values. As a result, there are different optimization techniques suggested by operation research (OR) researchers to do such work of optimization. According to [1], optimization techniques are categorized into four main categories:  .... " 

  (Update) A comment I got made me add this.  'Optimization' in business practice implies you can get the provably, best possible solution to a problem.   But in reality it almost always means you only can get the best solution within some specific context.     A context can include structure, constraints and goals.    It may also vary over time.    It may be wrong because its too hard to completely understand the problem.  But its still often useful to get a better solution, even if not provably optimal, if its better than todays practice.     Further if you can calculate this 'theoretical' best solution, it can give you better understanding of a problem, and what to strive for.    - FAD 

Intro to Genetic Algorithms

We used Genetic Algorithms (GA) in the enterprise. Have not heard much about them in the press lately, so it was nice to see this presentation online.   See previous writing here about the idea and suitable applications at the tag below.   Worth understanding:

Evolve with Genetic Algorithms

Charlie Koster introduces genetic algorithms, what they are, what they are useful for, and demonstrates code that shows how to create and use them. .... " 

Biology of the Evolution of Cooperation

Very old problem in Biology.   Why should organisms cooperate? A solution?  Could this same technique be used when solving genetic algorithms?  Or say the collaborative genetic evolution of intelligent swarms?

New Model of Evolution Finally Reveals How Cooperation Evolves

By treating evolution as a thermodynamic process, theorists have solved one the great problems in biology.    by Emerging Technology from the arXiv  June 21, 2017

One of the great unanswered question in biology is why organisms have evolved to cooperate. The long-term benefits of cooperation are clear—look at the extraordinary structures that termites build, for example, or the complex society humans have created.

But evolution is a random process based on the short-term advantages that emerge in each generation. Of course, individuals can cooperate or act selfishly, and this allows them to accrue benefits or suffer costs, depending on the circumstances. But how this behavior can spread and lead to the long-term emergence of cooperation as the dominant behavior is a conundrum that has stumped evolutionary biologists for decades. ... " 

Bots Build their Own Language

More on applications of reinforcement.    We did something similar using a genetic driver, but no mention of that here.  Would that improve learning?

It Begins: Bots are Learning to Chat in their own Language   by Cade Metz  In Wired: 

" .... As detailed in a research paper published by OpenAI this week, Mordatch and his collaborators created a world where bots are charged with completing certain tasks, like moving themselves to a particular landmark. The world is simple, just a big white square—all of two dimensions—and the bots are colored shapes: a green, red, or blue circle. But the point of this universe is more complex. The world allows the bots to create their own language as a way collaborating, helping each other complete those tasks.

All this happens through what’s called reinforcement learning, the same fundamental technique that underpinned AlphaGo, the machine from Google’s DeepMind AI lab that cracked the ancient game of Go. Basically, the bots navigate their world through extreme trial and error, carefully keeping track of what works and what doesn’t as they reach for a reward, like arriving at a landmark. If a particular action helps them achieve that reward, they know to keep doing it. In this same way, they learn to build their own language. Telling each other where to go helps them all get places more quickly. .... " 

Genes, Games, Randomness and the Power of Algorithms

In the CACM:  Sex as an algorithm.   Fascinating piece.  Not really technical, but a conceptually dense description.  Reminds me of some of our own experiments with genetic algorithms to solve complex corporate problems.  More in the tag below.  These were not really successful,  but why?

Probably because we were not successful in creating the right representation of the problem to work with.  Never obvious.    Even less so for this kind of solution method.   Also such genetically found algorithms are by their nature 'black boxes' whose operation cannot be directly explained.    Worth the scan or read for broad insight into the idea of genetically finding useful solutions.

Robots Learn from Insects

Have heard relatively little about genetic methods lately, here as a means of adapting initial descriptive solutions. Worth considering to achieve some process optimization in complex situtations.

In CACM: " ... Paris Diderot University researchers, led by researcher Jose Halloy, have developed a way to generate a robot cockroach's behavior automatically using a combination of descriptions of cockroach habits, combining models of individual movements with group activity. The researchers then used evolutionary algorithms to optimize the models.  " 

Evolutionary Computation for Creativity

Nicely done interview regarding  evolutionary and genetic methods in O'Reilly.   This was an area we explored in the enterprise, but to my knowledge never used practically.    We did use neural networks to replace some statistical analysis techniques.   It was suggested that it could provide more un-ordinary and creative solutions to common problems.   Worth a revisit, I believe there is much opportunity to include creative dimensions here.    "  .... Evolutionary computation: Stepping stones and unexpected solutions .... An interview with Risto Miikkulainen. ... " 

Robotics Evolution

In the BBC:  Pointing to research about adding evolving capabilities to robotics.  Using learning to improve.  This has not improved quite as far as it might imply.    It does add some physical and sensory  capabilities to what has been done primarily in abstract computing systems to date.  It also seeks to add new elements of creativity and innovation to robotic systems.   Evolution further implies the need for numerous alternative attempts to solve a problem, which links to swarms and genetic systems.    Also covered here.  See tag links below.

IBM Rival Sentient

An AI rival to understand: Sentient: With some emphasis on big data and genetic methods.  And the direct use of biomimicry through the use of genetic algorithms, an area we experimented in.

" ... Sentient was inspired because of its team’s work on Siri, which exposed them to “actual real applications of machine learning and artificial intelligence,” Mr. Blondeau said. They saw what they thought was a unique opportunity to combine that technology “with what the Internet gives us now in terms of the ability to reach and harness enormous amounts of compute—things that couldn’t be dreamt of 10 years ago, and could be done five years ago but nobody had done that.”

He said Sentient combines technologies in evolutionary computation, which mimics in software the way biological life evolved on Earth, and deep learning, which looks at the way nervous systems are architected and work. These technologies are used either independently or together and are scaled across millions of nodes.... " 

Genetically Tailored Beverages

More examples of personalization:

Coca-Cola CTO sees genetically-tailored beverages in the future
"It's not going to be that far out that we will talk about personalized beverages based on people's personal genes," said Guy Wollaert, Coca-Cola's chief technology officer. He pointed to current devices capable of tracking physical movement and body makeup and indicated that it won't be too long before people will be able to track whether they are dehydrated and how best to rehydrate ... "

Genetic Algorithms

I am a long time follower and user of genetic algorithms.  Here from David E Goldberg on Slideshare:  Fast, Effective Genetic Algorithms for Large, Hard Problems

New: Visual Footprint Resources App

Ken Karakotsios was the author of the remarkable SimLife genetic playground game.  He worked with us to build agent based simulation models that were used for years in the enterprise to understand product demand.  Now he has written an iPad App that can be used to understand resource usage and development.  I have had the pleasure to use it in Beta.  I particularly like its highly visual interaction and displays, which have given me ideas for visualization in my own design work.  See the Sankey diagram at right.  Likely also a good App for kids as it utilizes simulation, visualization and a timely subject. Follow along in his blog to understand its capabilities.  More examination of it will follow here.

" ... After three-plus years of research, designing, coding and testing, Footprint USA has been officially released!  

Footprint USA is an iPad app for adults and kids interested in learning more about using our planet's resources responsibly to build the future we really want. It combines tons of data with an interactive simulation to show the interrelationships among things like energy, food, land, water, and how we use our time and money.  It links the choices we make, individually and as a society, with the quality of life we get and the resulting impact on our planet. You can find it in the iTunes app store here.   You can also read more about it in my blog.

I am very grateful for all the support from my friends, family, and colleagues as you shared your ideas, time and patience to help make this a reality.  Thank you! ... " 

Recognizing Pattern with Math

Recognizing Patterns

With today’s powerful data analysis systems, users gather a ton of information—a breakdown of Wal-Mart Stores’ (WMT) sales in the U.S. or things people “like” on Facebook (FB)—in one place and then run queries. The questioner typically comes in with a preconceived idea of what he’s looking for or at least a set of preconceived biases that determine the questions he asks.

The Ayasdi software, which customers including Merck and Raytheon have been testing for several months, runs dozens of algorithms and then illuminates patterns and relations between the data points. BN ImmunoTherapeutics, for example, has turned to the software for research help on Prostvac, a prostate cancer vaccine that is undergoing clinical trials. The researchers compare genetic markers, people’s ages, medical histories, and other factors to figure out which patients will most likely benefit from the vaccine. “In the past, we would form a hypothesis and say, ‘We think these three biomarkers are important,’ ” says Amanda Enstrom, a research scientist at BN ImmunoTherapeutics. “With Ayasdi, we really allow the data to show us what the important biomarkers are.”
of information—a breakdown of Wal-Mart Stores’ (WMT) sales in the U.S. or things people “like” on Facebook (FB)—in one place and tWith today’s powerful data analysis systems, users gather a ton hen run queries. The questioner typically comes in with a preconceived idea of what he’s looking for or at least a set of preconceived biases that determine the questions he asks.

Data Company Emerges from Stanford Math

In BusinessWeek:
With today’s powerful data analysis systems, users gather a ton of information—a breakdown of Wal-Mart Stores’ (WMT) sales in the U.S. or things people “like” on Facebook (FB)—in one place and then run queries. The questioner typically comes in with a preconceived idea of what he’s looking for or at least a set of preconceived biases that determine the questions he asks.  

  The Ayasdi software, which customers including Merck and Raytheon have been testing for several months, runs dozens of algorithms and then illuminates patterns and relations between the data points. BN ImmunoTherapeutics, for example, has turned to the software for research help on Prostvac, a prostate cancer vaccine that is undergoing clinical trials. The researchers compare genetic markers, people’s ages, medical histories, and other factors to figure out which patients will most likely benefit from the vaccine. “In the past, we would form a hypothesis and say, ‘We think these three biomarkers are important,’ ” says Amanda Enstrom, a research scientist at BN ImmunoTherapeutics. “With Ayasdi, we really allow the data to show us what the important biomarkers are.”  ...

Optimizing Business Process

Rexamining a company, Charlotte Software Systems.  We had worked with their CEO Kevin Kostuik, when he and other people there worked with former component BiosGroup and then NuTech.  The company was bought by Netezza, which was then folded into IBM.    Charlotte is now an independent company.  We used some of their methods to address genetic and optimization solutions to such problems as project portfolio analysis, supply chain design and product development investment.   Worth a look.  Will  report more here as I can.  Pass me any experiences you have had.

   " ... Charlotte Software Systems specializes in cutting-edge optimization software to support enhanced tactical and strategic planning. Our systems can help optimize a number of complex business processes from large scale resource scheduling to strategic asset allocation. ...The foundation for our solutions is a powerful modeling and optimization platform upon which we capture the intricate details of your specific business process. Our solutions are delivered with attractive user interfaces which allow business specialists an intuitive way to manage their process. The resulting application is precisely tuned to fit your business while leveraging a proven platform of advanced optimization technologies. ... "