(Updated) Optimization using Genetic Methods

In our earliest days,  addressing supply chain and blending type manufacturing problems, we were an optimization shop.  Using the math structure of difficult combinatorial problems to find best solutions based on known goals and constraints.    But if you couldn't glean enough low level structure, we tested genetic methods, described here.   In this era of faster machines and more contextual information even more useful to try today.  Also for certain kinds of structure, also consider Dynamic Programming.  Happen to be examining that again today.

In KDNuggets  By Ahmed Gad, KDnuggets Contributor 

This article gives a brief introduction about evolutionary algorithms (EAs) and describes genetic algorithm (GA) which is one of the simplest random-based EAs.

Selection of the optimal parameters values for machine learning tasks is challenging. Some results may be bad not because the data is noisy or the used learning algorithm is weak, but due to the bad selection of the parameters values. This article gives a brief introduction about evolutionary algorithms (EAs) and describes genetic algorithm (GA) which is one of the simplest random-based EAs.

Introduction

Suppose that a data scientist has an image dataset divided into a number of classes and an image classifier is to be created. After the data scientist investigated the dataset, the K-nearest neighbor (KNN) seems to be a good option. To use the KNN algorithm, there is an important parameter to use which is K. Suppose that an initial value of 3 is selected. The scientist starts the learning process of the KNN algorithm with the selected K=3. The trained model generated reached a classification accuracy of 85%. Is that percent acceptable? In another way, can we get a better classification accuracy than what we currently reached? We cannot say that 85% is the best accuracy to reach until conducting different experiments. But to do another experiment, we definitely must change something in the experiment such as changing the K value used in the KNN algorithm. We cannot definitely say 3 is the best value to use in this experiment unless trying to apply different values for K and noticing how the classification accuracy varies. The question is “how to find the best value for K that maximizes the classification performance?” This is what is called optimization.

In optimization, we start with some kind of initial values for the variables used in the experiment. Because these values may not be the best ones to use, we should change them until getting the best ones. In some cases, these values are generated by complex functions that we cannot solve manually easily. But it is very important to do optimization because a classifier may produce a bad classification accuracy not because, for example, the data is noisy or the used learning algorithm is weak but due to the bad selection of the learning parameters initial values. As a result, there are different optimization techniques suggested by operation research (OR) researchers to do such work of optimization. According to [1], optimization techniques are categorized into four main categories:  .... " 

  (Update) A comment I got made me add this.  'Optimization' in business practice implies you can get the provably, best possible solution to a problem.   But in reality it almost always means you only can get the best solution within some specific context.     A context can include structure, constraints and goals.    It may also vary over time.    It may be wrong because its too hard to completely understand the problem.  But its still often useful to get a better solution, even if not provably optimal, if its better than todays practice.     Further if you can calculate this 'theoretical' best solution, it can give you better understanding of a problem, and what to strive for.    - FAD