Very Deep Neural Networks

Good piece in Linkedin

Very Deep Neural Networks Explained in 40 Seconds

Published on March 25, 2022,  By Vincent Granville

Machine Learning Scientist | MachineLearningRecipes.com

Very deep neural networks (VDNN) illustrated with data animation: a 40 second video, featuring supervised learning, layers, neurons, fuzzy classification, and convolution filters.

It is said that a picture is worth a thousand words. Here instead, I use a video to illustrate the concept of very deep neural networks (VDNN).

I use a supervised classification problem to explain how a VDNN works. Supervised classification is one of the main algorithms in supervised learning. The training set has four groups, each assigned a different color. The type of DNN described here is a convolutional neural network (CNN): it relies on filtering techniques. The filter is referred to, in the literature, as a convolution operator, thus the name CNN.   ... ' 

A Training Proof for Quantum AI

Technical finding regarding CNN and the ability to train.     Applicability to discovering new materials. 

Breakthrough Proof Clears Path for Quantum AI

Los Alamos National Laboratory News, October 15, 2021

Scientists at the U.S. Department of Energy's Los Alamos National Laboratory (LANL) have devised a proof that convolutional neural networks can always be trained on quantum computers, avoiding the threat of "barren plateaus" in optimization problems. LANL's Marco Cerezo said while a barren plateau eliminates any possibility of quantum speedup or advantage, "We proved the absence of barren plateaus for a special type of quantum neural network. Our work provides trainability guarantees for this architecture, meaning that one can generically train its parameters." LANL's Patrick Coles said, "With this guarantee in hand, researchers will now be able to sift through quantum-computer data about quantum systems and use that information for studying material properties or discovering new materials, among other applications." ... ' 

Better Detection of Earthquakes with Machine Learning

New work in the space using Machine Learning, we had proposed related methods when studying neural models.  Here a convolutional neural network.

Researchers Create Earthquake System Model with Better Detection Capabilities  in CACM

University of Wyoming,  October 12, 2021

The University of Wyoming's Pejman Tahmasebi and Tao Bai have invented a machine learning (ML) model that boosts the accuracy of earthquake detection significantly over current models. Tahmasebi said the model processes signal data recorded by seismometers, and can automatically distinguish seismic events from seismic noise. The model combines existing long short-term memory and fully convolutional network ML models; the former captures data signal changes over time, and the latter filters out hidden features of seismic events. Tahmasebi said the model boasts 89.1% classification accuracy, a 14.5% improvement over the state-of-the-art ConvNetQuake model.

Full article

Simulating the Market

Quite a claim, often mentioned in early AI 'tests'?  Can it work?   Is simulation sufficiently complex proxy for the market?

AI stock trading experiment beats market in simulation  by Chinese Association of Automation in TechExplore

Researchers in Italy have melded the emerging science of convolutional neural networks (CNNs) with deep learning—a discipline within artificial intelligence—to achieve a system of market forecasting with the potential for greater gains and fewer losses than previous attempts to use AI methods to manage stock portfolios. The team, led by Prof. Silvio Barra at the University of Cagliari, published their findings on IEEE/CAA Journal of Automatica Sinica.

The University of Cagliari-based team set out to create an AI-managed "buy and hold" (B&H) strategy—a system of deciding whether to take one of three possible actions—a long action (buying a stock and selling it before the market closes), a short action (selling a stock, then buying it back before the market closes), and a hold (deciding not to invest in a stock that day). At the heart of their proposed system is an automated cycle of analyzing layered images generated from current and past market data. Older B&H systems based their decisions on machine learning, a discipline that leans heavily on predictions based on past performance.....  "

More information: Silvio Barra, Salvatore Mario Carta, Andrea Corriga, Alessandro Sebastian Podda and Diego Reforgiato Recupero, "Deep Learning and Time Series-to-Image Encoding for Financial Forecasting," IEEE/CAA J. Autom. Sinica, vol. 7, no. 3, pp. 683-692, May 2020. www.ieee-jas.org/en/article/do … 109/JAS.2020.1003132

AI For Drug Design

See in  particular how these methods are designed

AI for Drug Design   By Sandrine Ceurstemont

Some recent breakthroughs in drug discovery have come about thanks to the use of artificial intelligence.

Traditional drug development is slow and expensive. It often takes more than 10 years for a new medicine to come to market, and it can cost up to $2.6 million. In the past few years, however, there has been a growing interest in using machine learning to help with the process.

"The idea is that you can screen billions of molecules on a computer and identify some which look promising, and then you just manufacture and test the small subset," says Regina Barzilay, Delta Electronics Professor of the Massachusetts Institute of Technology (MIT) Department of Electrical Engineering and Computer Science, and a member of the university's Computer Science and Artificial Intelligence Lab.

The vast size of chemical space is one of the main challenges when it comes to finding new drugs. Medicinal chemists look for new small molecules, and there could be up to 1 novemdecillion (1 followed by 60 zeros) of them, according to the American Chemical Society, more than some estimates of the number of stars in the universe. Although researchers have zeroed in on millions of these compounds through traditional methods, the number that have been synthesized and tested as drugs is thought to represent less than 0.1% of the potential drugs that exist. "The machine learning community identified it as an important area where we can contribute," says Barzilay.

There have been some recent breakthroughs in drug discovery, thanks to artificial intelligence (AI). In recent work, Barzilay and her colleagues used a deep learning system to discover a new antibiotic, which is a first. The newly discovered medicine proved effective against a wide range of bacteria in tests on mice, including tuberculosis and bacteria strains that have demonstrated resistance to current antibiotics.

Barzilay and her team decided to focus on antibiotics since a lack of new antibiotics is creating a growing health crisis. Existing antibiotics are no longer effective against many infections, as bacteria have grown resistant. Just eight new antibiotics with limited effectiveness have been approved since July 2017, according to a recent report by the World Health Organization.

To tackle the problem, the researchers developed a deep learning convolutional neural network (CNN) that can predict the antibiotic properties of new compounds. It was first trained to recognize molecules that inhibit the growth of E. coli bacteria by feeding it a collection of about 2,500 molecules whose antibacterial capabilities were known. Then, the system was presented with a library, called the Drug Repurposing Hub, containing over 6,000 molecules identified as potentially interesting to fight various human diseases.  It was asked to predict which molecules are both active against E. coli and had different structures from existing antibiotics.

One result was the new antibiotic halicin (named for the intelligent computer HAL in the movie 2001: A Space Odyssey). The medication  was being investigated as a potential treatment for diabetes.  .... " 

Algorithm Predicts Corn Yields

What seems to be some direct uses of CNN for yield prediction.

AI Algorithm Better Predicts Corn Yield
By Illinois ACES
February 24, 2020
  
An interdisciplinary research team at the University of Illinois at Urbana-Champaign has developed a convolutional neural network that generates crop yield predictions.

An interdisciplinary research team at the University of Illinois at Urbana-Champaign has developed a convolutional neural network (CNN) that generates crop yield predictions, incorporating information from topographic variables such as soil electroconductivity, nitrogen levels, and seed rate treatments.

The team worked with data captured in 2017 and 2018 from the Data Intensive Farm Management project, in which seeds and nitrogen fertilizer were applied at varying rates across 226 fields in the Midwest U.S., Brazil, Argentina, and South Africa.

In addition, on-ground measurements were combined with high-resolution satellite images from PlanetLab to predict crop yields.

Said Illinois's Nicolas Martin, while "we don’t really know what is causing differences in yield responses to inputs across a field … the CNN can pick up on hidden patterns that may be causing a response.”    ... '

Seeing in Higher Dimensions

'Seeing' is constructing useful models about spaces from sensors to understand and navigate them.   Predict current and future states. A good studey of the idea.

An Idea From Physics Helps AI See in Higher Dimensions
The laws of physics stay the same no matter one’s perspective. Now this idea is allowing computers to detect features in curved and higher-dimensional space.

The new deep learning techniques, which have shown promise in identifying lung tumors in CT scans more accurately than before, could someday lead to better medical diagnostics.

Olena Shmahalo/Quanta Magazine
John Pavlus  Contributing Writer

January 9, 2020

Computers can now drive cars, beat world champions at board games like chess and Go, and even write prose. The revolution in artificial intelligence stems in large part from the power of one particular kind of artificial neural network, whose design is inspired by the connected layers of neurons in the mammalian visual cortex. These “convolutional neural networks” (CNNs) have proved surprisingly adept at learning patterns in two-dimensional data — especially in computer vision tasks like recognizing handwritten words and objects in digital images.

But when applied to data sets without a built-in planar geometry — say, models of irregular shapes used in 3D computer animation, or the point clouds generated by self-driving cars to map their surroundings — this powerful machine learning architecture doesn’t work well. Around 2016, a new discipline called geometric deep learning emerged with the goal of lifting CNNs out of flatland.... "

Building a Computer Vision Model

A simplified, straightforward tutoral on a computer vision model.   This is the place you can get something impressive out of neural nets,  and an intro to the general AI method along them way.       Of most use too, pointers to existing databases to get started with.  We used ImageNet and WordNet tags, for example.

From KDNuggets:

How can we build a computer vision model using CNNs? What are existing datasets? And what are approaches to train the model? This article provides an answer to these essential questions when trying to understand the most important concepts of computer vision.  

By Javier Couto, Tryolabs.

Computer vision is one of the hottest subfields of machine learning, given its wide variety of applications and tremendous potential. Its goal: to replicate the powerful capacities of human vision. But how is this achieved with algorithms?

Let's have a loot at the most important datasets and approaches.

Existing datasets
Computer vision algorithms are no magic. They need data to work, and they can only be as good as the data you feed in. These are different sources to collect the right data, depending on the task:

One of the most voluminous and well known dataset is ImageNet, a readily-available dataset of 14 million images manually annotated using WordNet concepts. Within the global dataset, 1 million images contain bounding box annotations.  .... "

Priming Learning Networks

I wrote a paper on just this and related concepts that primed learning networks during the first wave of neural nets,  which we tested internally,  reviewing. 

Randomly wired neural networks and state-of-the-art accuracy? Yes it works.

How do you design the best Convolutional Neural Network (CNN)?    By George Seif

Although Deep Learning has been around for several years now, that’s still an unanswered question.

Much of the difficulty in designing a good neural net stems from the fact that they’re still black boxes. We have some high-level idea of how they work, but we don’t really know how they achieve the results that they do.   .... "

Jason Brownlee Reviews Stanford CNN Course

Thoughtful piece on a seminal course.  Useful if you are considering taking a deeper dive.

Stanford Convolutional Neural Networks for Visual Recognition Course (Review) by Jason Brownlee 

The Stanford course on deep learning for computer vision is perhaps the most widely known course on the topic.

This is not surprising given that the course has been running for four years, is presented by top academics and researchers in the field, and the course lectures and notes are made freely available.

This is an incredible resource for students and deep learning practitioners alike.

In this post, you will discover a gentle introduction to this course that you can use to get a jump-start on computer vision with deep learning methods.

After reading this post, you will know:

The breakdown of the course including who teaches it, how long it has been taught, and what it covers.

The breakdown of the lectures in the course including the three lectures to focus on if you are already familiar with deep learning.

A review of the course, including how it compares to similar courses on the same subject matter.
Let’s get started.  ...." 

Combining CNN and RNNs

Made me think .... Technical.

Combining CNNs and RNNs – Crazy or Genius?   by William Vorhies  

Summary: There are some interesting use cases where combining CNNs and RNN/LSTMs seems to make sense and a number of researchers pursuing this.  However, the latest trends in CNNs may make this obsolete.

There are things that just don’t seem to go together.  Take oil and water for instance.  Both valuable, but try putting them together?

That was my reaction when I first came across the idea of combining CNNs (convolutional neural nets) and RNNs (recurrent neural nets).  After all they’re optimized for completely different problem types.

CNNs are good with hierarchical or spatial data and extracting unlabeled features. Those could be images or written characters.  CNNs take fixed size inputs and generate fixed size outputs.
RNNs are good at temporal or otherwise sequential data. Could be letters or words in a body of text, stock market data, or speech recognition.  RNNs can input and output arbitrary lengths of data.  LSTMs are a variant of RNNs that allow for controlling how much of prior training data should be remembered, or more appropriately forgotten.  We all know to reach for the appropriate tool based on these very unique problem types.

So are there problem types that need the capability of both these tools?

As it turns out, yes.  Most of these are readily identified as images that occur in a temporal sequence, in other words video.  But there are some other clever applications not directly related to video that may spark your imagination.  We’ll describe several of those below.  .... "

Neural Network for Snippets

A favorite topic, how do we make sense and value of written knowledge?

A Neural Network to Extract Knowledgeable Snippets and Documents 

Tech Xplore   By Ingrid Fadelli

Chinese Academy of Sciences researchers have created a convolutional neural network (CNN)-based model to extract knowledgeable snippets and annotate documents. The model can outperform current analytical tools while undergoing shorter training periods. The model is designed to comprehend the abstract concept of documents in different domains collaboratively and evaluate whether a document is knowledgeable, defined as one "containing multiple knowledgeable snippets, which describe concepts, properties of entities, or the relations among entities." The researchers say the network structure of their SSNN joint CNN-based model is "low-level Sharing, high-level Splitting," in which the low-level layers are shared for different domains while the high-level layers outside the network receive separate training to identify the differences of dissimilar domains. The team assessed SSNN's effectiveness on a dataset of real documents from three content domains on the WeChat messaging/social media/mobile payment platform. The model performed consistently better than other CNN models while saving time and memory usage due to shorter and more efficient training processes. In the future, the model could help build comprehensive knowledge databases and innovative services that answer user queries in real time. .... "

Convolutional Neural Nets

Nice intro to one of the most useful net architectures for classification today.   Direct implementation:

Convolutional Neural Networks from the ground up in Towardsdatascience

A NumPy implementation of the famed Convolutional Neural Network: one of the most influential neural network architectures to date.

When Yann LeCun published his work on the development of a new kind of neural network architecture [1], the Convolutional Neural Network (CNN), his work went largely unnoticed. It took 14 years and a team of researchers from The University of Toronto to bring CNN’s into the public’s view during the 2012 ImageNet Computer Vision competition. Their entry, which they named AlexNet after chief architect Alex Krizhevsky, achieved an error of only 15.8% when tasked with classifying millions of images from thousands of categories [2]. Fast forward to 2018 and the current state-of-the-art Convolutional Neural Networks achieve accuracies that surpass human-level performance [3]. .... " 

Sequencing Problems for Natural Language Processing

Good piece by William Vorhies

Temporal Convolutional Nets (TCNs) Take Over from RNNs for NLP Predictions   Posted by William Vorhies in DSC

Summary: Our starting assumption that sequence problems (language, speech, and others) are the natural domain of RNNs is being challenged.  Temporal Convolutional Nets (TCNs) which are our workhorse CNNs with a few new features are outperforming RNNs on major applications today.  Looks like RNNs may well be history.

It’s only been since 2014 or 2015 when our DNN-powered applications passed the 95% accuracy point on text and speech recognition allowing for whole generations of chatbots, personal assistants, and instant translators.

Convolutional Neural Nets (CNNs) are the acknowledged workhorse of image and video recognition while Recurrent Neural Nets (RNNs) became the same for all things language.

One of the key differences is that CNNs can recognize features in static images (or video when considered one frame at a time) while RNNs excelled at text and speech which were recognized as sequence or time-dependent problems.  That is where the next predicted character or word or phrase depends on those that came before (left-to-right) introducing the concept of time and therefore sequence.

Actually RNNs are good at all types of sequence problems, including speech/text recognition, language-to-language translation, handwriting recognition, sequence data analysis (forecasting), and even automatic code generation in many different configurations. .... " 

Beginners Guide to Convolutional Neural Nets

A beginners intro by a student, about how they work, how they are used and the data required.  Nicely explained.