The Value of the FuzzBench

A means to support debugging code and thus searching for code vulnerabilities.  We used it the early days.  Google uses it and recently published some public capabilities.  See a past article here on this: Fuzzing for Testing Coding Security Vulnerabilities, from the ACM.

Below was brought to my attention here from  Steve Gibson's Security Now. #758   Podcast
https://www.grc.com/sn/SN-757-Notes.pdf


The Fuzzy Bench
Posted last week to the Google Open Source Blog:
https://opensource.googleblog.com/2020/03/fuzzbench-fuzzer-benchmarking-as-service.html

“FuzzBench: Fuzzer Benchmarking as a Service”
Monday, March 2, 2020

WikiPedia defines fuzzing this way:
     “Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. The program is then monitored for exceptions such as crashes, failing built-in code assertions, or potential memory leaks. Typically, fuzzers are used to test programs that take structured inputs. This structure is specified, e.g., in a file format or protocol and distinguishes valid from invalid input. An effective fuzzer generates semi-valid inputs that are "valid enough" in that they are not directly rejected by the parser, but do create unexpected behaviors deeper in the program and are "invalid enough" to expose corner cases that have not been properly dealt with.” .... 

So, fuzzing is an interesting way to uncover bugs in programs. Rather than having a single highly-skilled hacker with knowledge of all past vulnerabilities carefully and methodically trying this or that in attempt to exploit something that might be possible, fuzzing is like the thousand monkeys all pounding on typewriters to see whether any of them might, by pure happenstance, hit upon something novel and useful.

 In another of Google’s “working to make the world a better place because we have plenty of money, so why not?”, last week they posted the explanation of their latest initiative: We are excited to launch FuzzBench, a fully automated, open source, free service for evaluating fuzzers. The goal of FuzzBench is to make it painless to rigorously evaluate fuzzing research and make fuzzing research easier for the community to adopt.   .... 

Microsoft Makes AI Debugging Tool Open Source

And with similarities to Google's What-If tool, Microsoft has made its debugging tool open source.  Have not seen this one directly as yet,  but this is NOT a No-code solution, likely making it harder for management to understand.

Microsoft Open-Sources TensorWatch AI Debugging Tool   by Anthony Alford in InfoQ

Microsoft Research open-sourced TensorWatch, their debugging tool for AI and deep-learning. TensorWatch supports PyTorch as well as TensorFlow eager tensors, and allows developers to interactively debug training jobs in real-time via Jupyter notebooks, or build their own custom UIs in Python.

In a recent blog post, a research team led by Shital Shah announced the open-source release of TensorWatch. TensorWatch is a Python library for visualizing data from all phases of the deep-learning model-development cycle: from model structure, to training metrics, to explanation of model predictions. TensorWatch is designed to be used as an interactive tool in Jupyter notebooks or JupyterLab dashboards, but as a Python library it can also be included in custom tools and UIs. According to the development team:

"We like to think of TensorWatch as the Swiss Army knife of debugging tools with many advanced capabilities researchers and engineers will find helpful in their work."  ... " 

More from the MS Research Blog.

Checklist for Debugging Networks

Not a bad high level view.     I would consider also frequently reviewing goals, data stability, Data biases and measure or sensor errors,  trending results.  Make sure to involve domain experts in every results review.  Take their input seriously.    The worst bug you can have is solving the wrong problem and not knowing it.

Checklist for debugging neural networks

Tangible steps you can take to identify and fix issues with training, generalization, and optimization for machine learning models    By Cecelia Shao

Machine learning code can be notoriously difficult to debug with bugs that are expensive to chase. Even for simple, feedforward neural networks, you often have to make several decisions around network architecture , weight initialization, and network optimization — all of which can lead to insidious bugs in your machine learning code.

As Chase Roberts wrote in an excellent piece on ‘How to unit test machine learning code’, his frustrations stemmed from common traps like:

The code never crashes, raises an exception, or even slows down.
The network still trains and the loss will still go down.
The values converge after a few hours, but to really poor results
So what is to be done about it?  ..... "