Sampling, Pipelining Method Speeds Deep Learning on Large Graphs

Enhancing Neural Networks

Sampling, Pipelining Method Speeds Deep Learning on Large Graphs

MIT News, Lauren Hinkel, November 29, 2022

The SAmpling, sLIcing, and data movemeNT (SALIENT) methodology devised by Massachusetts Institute of Technology (MIT) and IBM Research scientists enhances graph neural networks (GNNs)' training and inference by clearing three bottlenecks in the computational pipeline. The researchers applied optimization to increase graphics processing unit (GPU) utilization in the PyTorch Geometric library for GNNs from 10% to 30%, improving performance up to double that of public benchmark codes. They addressed bottlenecks caused by graph sampling and mini-batch preparation algorithms at the beginning of the data pipeline by combining data structures and algorithmic optimizations, improving the sampling operation about threefold. MIT's Nickolas Stathas said SALIENT leveraged modern processors to further reduce per-epoch runtime via parallelizing feature slicing. ...  '

Robustly Labeled Graphs

Interesting and Technical piece in the Google AI Blog. 

Robust Graph Neural Networks  in The Google Blog

Tuesday, March 8, 2022

Posted by Bryan Perozzi, Research Scientist and Qi Zhu, Research Intern, Google Research

Graph Neural Networks (GNNs) are powerful tools for leveraging graph-structured data in machine learning. Graphs are flexible data structures that can model many different kinds of relationships and have been used in diverse applications like traffic prediction, rumor and fake news detection, modeling disease spread, and understanding why molecules smell.

As is standard in machine learning (ML), GNNs assume that training samples are selected uniformly at random (i.e., are an independent and identically distributed or “IID” sample). This is easy to do with standard academic datasets, which are specifically created for research analysis and therefore have every node already labeled. However, in many real world scenarios, data comes without labels, and labeling data can be an onerous process involving skilled human raters, which makes it difficult to label all nodes. In addition, biased training data is a common issue because the act of selecting nodes for labeling is usually not IID. For example, sometimes fixed heuristics are used to select a subset of data (which shares some characteristics) for labeling, and other times, human analysts individually choose data items for labeling using complex domain knowledge.  .... '