Causal Monitoring for Distributed Systems: IOT Solutions

Spent some time looking at monitoring and troubleshooting systems, with the obvious need to infer causation.   Here an intro to 'Pivot Tracing', a means of causal monitoring I had not heard of.    Would seem to be very useful for interconnected IOT.   Technical.

Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems
By Jonathan Mace, Ryan Roelke, Rodrigo Fonseca
Communications of the ACM, March 2020, Vol. 63 No. 3, Pages 94-102
10.1145/3378933

Monitoring and troubleshooting distributed systems are notoriously difficult; potential problems are complex, varied, and unpredictable. The monitoring and diagnosis tools commonly used today—logs, counters, and metrics—have two important limitations: what gets recorded is defined a priori, and the information is recorded in a component- or machine-centric way, making it extremely hard to correlate events that cross these boundaries. This paper presents Pivot Tracing, a monitoring framework for distributed systems that addresses both limitations by combining dynamic instrumentation with a novel relational operator: the happened-before join. Pivot Tracing gives users, at runtime, the ability to define arbitrary metrics at one point of the system, while being able to select, filter, and group by events meaningful at other parts of the system, even when crossing component or machine boundaries. Pivot Tracing does not correlate cross-component events using expensive global aggregations, nor does it perform offline analysis. Instead, Pivot Tracing directly correlates events as they happen by piggybacking metadata alongside requests as they execute. This gives Pivot Tracing low runtime overhead—less than 1% for many cross-component monitoring queries.  ... "