Monte Carlo Simulation

 Good intro piece to a method we used for many purposes in the enterprise.   Even creating usable models for key processes that were used for decades.  Consider its similarities to Digital Twins.  

 Monte Carlo Simulation

Darío Weitz   in Towards Data Science, Engineer, Ms. Sc., Former Associate Professor at Ing. en Sistemas de Información, Fac. Reg. Rosario, Univ. Tecnológica Nacional, Argentina. Data Viz Consultant.

Part 1: The News Vendor Problem

In the first article of this series, we defined simulation as a numerical technique consisting of building a mathematical and/or logical model of the system under study and then experimenting with it, collecting data that allows us to obtain an estimator to help solve a complex decision problem.

In the same article, we defined a model as a simplified but valid representation of a real process or system, intending to gain some understanding of its behavior.

We also made a classification of models, distinguishing in particular between continuous models, those in which their behavior (state variables) changes continuously over time, and discrete models, those in which the state variables only change at separate points in time. Another important classification involves static models, those that are a representation of the system at a particular time, and dynamic models, those that evolve over time.

Related to the above classification there are three different types of simulations: continuous event simulation, discrete event simulation, and Monte Carlo simulation.

Principles and concepts about Discrete Event Simulation (DES) were provided in the previously indicated series. We coded several examples with SimPy, an object-oriented, process-based, discrete-event simulation framework based on pure Python. In future articles, we will develop concepts and principles related to continuous event simulation.

In this article (and probably in a couple of others) we are dealing with Monte Carlo Simulations.

Monte Carlo Methods

Monte Carlo Methods (MCM) is a collection of numerical methods for the solution of mathematical problems, where the use of random samples differentiates them from equivalent methods.

The term was coined by the Greek-American physicist Nicholas Metropolis when he was working at Los Alamos National Laboratory with John von Neumann and Stanislaw Ulam in the development of the first atomic bomb. The term gets its origin from the famous casino located in the Principality of Monaco.

The conceptual idea of the MCM consists in the estimation of certain quantities through repeated sampling from models represented in a computer. Two classes of mathematical problems are usually solved with these techniques: integration and optimization.

Concerning the contents described in this series of articles, when we refer to Monte Carlo Simulation models we are talking about static, discrete, stochastic models trying to solve an optimization problem.

From a methodological point of view, a Monte Carlo simulation is a sampling experiment whose aim is to estimate the distribution of a quantity of interest that depends on one or more stochastic input variables. We are particularly interested in calculating point estimates and confidence intervals for that quantities. Inevitably, our estimator will have a sampling error and our first task will be to determine the number of replications to improve the degree of certainty in the value of the estimator.  .... ' 

Data Observability

Was un aware of the broad topic, but can see the value.  Was also unaware of the company: Monte Carlo.  Data Quality is something we always considered.  Much more in the piece.

Monte Carlo Raises $135 Million to Grow Data Observability Biz

Monte Carlo Manta Data

in Datanami by Alex Woodie / May 26, 2022

Data observability has emerged as one of the hottest sectors in the big data market, thanks to its focus on fixing broken data pipelines. One of the hottest players in the field is Monte Carlo, which this week announced a Series D round of funding worth $135 million, at a $1.6 billion valuation.

As companies look to data for competitive advantages, they’re finding that the costs of data quality problems continues to grow too. Gartner estimated that the average customer loses nearly $13 million per year due to data downtime and data quality problems. This is the area that Monte Carlo is addressing with its data observability offering.

Barr Moses and Lior Gavish co-founded Monte Carlo three years ago with the goal of developing tools to help companies detect problems in their ETL data pipelines and even take steps to automatically fix some of them. While ETL and ELT is viewed by some as a legacy approach to moving data, they continue to be the workhorse mechanisms for moving large amounts of data from on prem systems to the cloud, and everywhere in between.

The San Francisco company borrowed from concepts popular in the SRE and DevOps space to help address the problem of bad data flowing through data pipelines. By using connectors to take read-only copies of data directly from pipelines and machine learning techniques to spot anomalies in patterns, Monte Carlo is able to continuously monitor for common data problems, and send alerts to engineers when they are detected.  .... ' 

Simulation with Python

By far my biggest effort in the enterprise was doing simulations.   Often integrated with optimizations, and later AI applications, both knowledge based and neural approaches.    Nice to see how it can be integrated with Python.   We used IBM's forms,  GPSS  and Simscript; Sometimes specified by clients.  Below an intro, more at the link

Monte Carlo Simulation and Variants with Python

Your Guide to Monte Carlo Simulation and Must Know Statistical Sampling Techniques With Python Implementation

By Tatev Karen

Monte Carlo Simulation is based on repeated random sampling. The underlying concept of Monte Carlo is to use randomness to solve problems that might be deterministic in principle. Monte Carlo simulation is one of the most popular techniques to draw inferences about a population without knowing the true underlying population distribution. This sampling technique becomes handy especially when one doesn’t have the luxury to repeatedly sample from the original population. Applications of Monte Carlo Simulation range from solving problems in theoretical physics to predicting trends in financial investments.

Monte Carlo has 3 main usages: estimate parameters or statistical measures, examine the properties of the estimates, approximate integrals.   ..... ' 

Differences Between Simulation and Digital Twins

Interesting.  Have done lots of Monte Carlo style simulation modeling, much less using digital twins.   Is this a good description?  For what contexts? .  We also did agent based modeling, which has similarities to DTs.  Anyone with similar experiences have some good thoughts?   Point me to other resource examples of value.    What to collaborate on this?  Say for choice of models or their combination? 

What is the Difference Between a Simulation and a Digital Twin Model?

 15 Apr, 2020  in Exorint

The increasing use of emerging technology to simplify complex tasks has proved rewarding across every industry in diverse ways. This includes increased operational efficiency, automating manual tasks, training and validation, as well as data analysis. It is a known fact that the integration of emerging tech has brought on the fourth industrial revolution, in which data analytics and automation are important components. The digital transformation of traditional processes is also another aspect of Industrie 4.0 and, here, simulation and the digital twin play starring roles. But what are these roles?

This article will discuss:

The definition and application of simulation technology and the digital twin

The differences between a simulation and a digital twin

The symbiotic relationship between simulations and the digital twin

What is simulation?

In computing, simulations refer to digital models that imitate the operations or processes within a system. Such simulations are used for analyzing the performances of systems and the testing and implementation of new ideas. Engineers and technicians make use of simulations across a variety of industries to test products, systems, processes, and concepts.

In many circles, simulations are run using computer-aided design software applications. But for more advanced simulations with many variables, specialized simulation software is used. Typical examples of how simulations function include their use in finite element analysis and stress analysis. In the real world, these tests involve analyzing the effect of external pressure on metals or products to enhance their design or features.

Other types of simulations include discrete event simulations, stochastic simulations, and deterministic simulations. In these types, the variables used in running the simulation are either known or random. To run simulations, some level of digitization is needed. This process may involve only mathematical concepts or the design of 2D or 3D models representing assets within a process or a product. The simulation is then run by introducing variables into the digital environment or interface.

What is a digital twin?

In its basic form, a digital twin is the digital representation of physical or non-physical processes, systems, or objects. The digital twin also integrates all data produced or associated with the process or system it mirrors. Thus, it enables the transfer of data within its digital ecosystem, mirroring the data transfer that occurs in the real world. The data used in digital twins are generally collected from Internet of Things devices, edge hardware, HMIs, sensors, and other embedded devices. Thus, the captured data represents high-level information that integrates the behavioral pattern of digitized assets in the digital twin.

The real-time digital representation a digital twin provides serves as a world of its own. Within this digital world, all types of simulation can be run. It can also be used as a planning and scheduling tool for training, facility management, and the implementation of new ideas. This highlights the fact that a digital twin is a virtual environment, thus it must consist of either 2D or 3D assets or the data they produce or are expected to produce. In the modeled virtual environment, individuals can do what they choose with few limitations including the running of simulations.

Industrial cloud

Highlighting the differences between simulations and digital twins

Although the definitions of both concepts already highlight key differences, the use of case studies makes these differences more relatable. In 2019, CKE Holdings Inc., the parent company of Hardee’s and Carl Jr’s was interested in enhancing productivity levels within these facilities. The idea was to make order picking by staff easier and reduce shop floor traffic through better layout designs.

While simulations can be used to analyze the shortest distance between workstations or the effects of more storage facilities within the restaurant, a digital twin can do much more. Using a digital twin, CKE Holdings Inc. was able to recreate digital representations of existing shop floors and run multiple simulations, design, and scheduling ideas to enhance productivity. This resulted in improvements in every aspect of the facility’s operation from staff training, scheduling, and meeting customer demands more efficiently.

This shows that while simulations may help with understanding what may happen when changes are introduced, a digital twin helps with understanding both what is currently happening and what may happen within a process. Some key differences include:

Real-time simulations – Traditional simulations are run in virtual environments that may be representations of physical environments but do not integrate real-time data. The regular transfer of information between a digital twin and its corresponding physical environment makes real-time simulation possible. This increases the accuracy of predictive analytical models and the management and monitoring policies of enterprises.

Enhancing product design – Advanced simulations have the capacity to analyze thousands of variables to provide diverse answers, but a digital twin can be used to achieve more. Boeing’s integration of digital-twin technology in aircraft design and production is an example of its capabilities. In this case, a digital twin was used to simulate parts of an aircraft to analyze how diverse materials will fare throughout the life-cycle of the aircraft. With these calculations, Boeing was able to achieve 40% improvement in the quality of certain parts it designed.

Optimize real-world products and processes – Every Tesla automobile running today has a digital twin that captures the large data sets each car produces. The captured data is used in optimizing design, predictive analytics, enhancing self-driving initiatives, and maintenance. This highlights how a digital twin immediately or directly affects the physical entity it represents unlike the theoretical results simulations provide.

Conclusion 

Regardless of the path taken, the digital transformation of assets and processes enhances industrial effort in many ways. This includes refining product design, real-time troubleshooting, and implementing new ideas. To achieve a comprehensive digital transformation of existing or planned entities, systems, and processes, accurate data capture is required. Enter smart edge technology or devices.

The accuracy of a simulation or a digital twin relies heavily on the accuracy of the data used in designing its models. In today’s shop floors, data capture is being made possible by smart edge device and human-machine interfaces and only with these types of data can a digital transformation occur. ... '

Timeline for Quantum Computing

 Monte Carlo methods are where speed is needed.  And in theory Quantum Computing could get us there.   Still some nagging issues exist. 

The Timeline for Quantum Computing is Getting Shorter  By NetworkWorld, May 19, 2021

Financial traders rely heavily on computer financial simulations for making buying and selling decisions. Specifically, "Monte Carlo" simulations are used to assess risk and simulate prices for a wide range of financial instruments. These simulations also can be used in corporate finance and for portfolio management.

But in a digital world where other industries routinely leverage real-time data, financial traders are working with the digital equivalent of the Pony Express. That's because Monte Carlo simulations involve such an insanely large number of complex calculations that they consume more time and computational resources than a 14-team, two-quarterback online fantasy football league with Superflex position.

Consequently, financial calculations using Monte Carlo methods typically are made once a day. While that might be fine in the relatively tranquil bond market, traders trying to navigate more volatile markets are at a disadvantage because they must rely on old data. If only there were a way to accelerate Monte Carlo simulations for the benefit of our lamentably ladened financial traders!

Soon there will be, according to financial services giant Goldman Sachs and QC Ware, a quantum-as-a-service provider that develops applications to run on near-term quantum-computing hardware. Researchers for the two partners reportedly have designed new quantum algorithms for running Monte Carlo simulations on near-term quantum hardware expected to be available in five to 10 years.

From NetworkWorld

Facebook Research Allocates Ad Funding (without AI)

Struck me because we did something similar and also quite different for advertising fund allocation very early on.    We the results of integer optimization models with montecarlo simulation, constrained to advertising agreements.   I like the fact that AI is never mentioned!  But in much later spins on this,  what could be called AI was used to check for accuracy and regulatory compliance to contractual agreements.  Nice.

Auto-placement of ad campaigns using multi-armed bandits  in Facebook Research

By: Vashist Avadhanula, Riccardo Colini Baldeschi, Stefano Leonardi, Karthik Abinav Sankararaman, Okke Schrijvers

What the research is:

We look at the problem of allocating the budget of an advertiser across multiple surfaces optimally when both the demand and the value are unknown. Consider an advertiser who uses the Facebook platform to advertise a product. They have a daily budget that they would like to spend on our platform. Advertisers want to reach users where they spend time, so they spread their budget over multiple platforms, like Facebook, Instagram, and others. They want an algorithm to help bid on their behalf on the different platforms and are increasingly relying on automation products to help them achieve it.

In this research, we model the problem of placement optimization as a stochastic bandit problem. In this problem, the algorithm is participating in k different auctions, one for each platform, and needs to decide the correct bid for each of the auctions. The algorithm is given a total budget B (e.g., the daily budget) and a time horizon T over which this budget should be spent. At each time-step, the algorithm should decide the bid it will associate with each of the k platform, which will be input into the auctions for the next set of requests on each of the platforms. At the end of a round (i.e., a sequence of requests), the algorithm sees the total reward it obtained (e.g., number of clicks) and the total budget that was consumed in the process, on each of the different platforms. Based on just this history, the algorithm should decide the next set of bid multipliers it needs to place. The goal of the algorithm is to maximize the total advertiser value with the given budget across the k platforms.   ... "    

Full paper:  https://arxiv.org/abs/2103.10246 

Phonons and Quantum Monte Carlo

More looks at material science.

A Successful Phonon Calculation Within the Quantum Monte Carlo Framework  By Japan Advanced Institute of Science and Technology  April 6, 2021

Phonon dispersion of diamond calculated at the variational Monte Carlo level by TurboRVB Image.

An international team of scientists has enhanced quantum Monte Carlo (QMC) computation speeds based on error reduction, and conducted proof-of-concept calculations for a periodic solid.

They found this method can lower statistical error in atomic-force evaluation by two orders of magnitude, and accelerate computation 10,000-fold.

The researchers used the technique to calculate the atomic vibrations of diamond on the Japan Advanced Institute of Science and Technology (JAIST)'s Cray-XC40 T computer, and the results were consistent with experimental values.

JAIST's Kousuke Nakano said, "The drastic reduction in computational time will greatly expand the range of QMC calculations and enable highly accurate prediction of atomic properties of materials that have been difficult to handle."

From Japan Advanced Institute of Science and Technology