Benchmarking AI Performance

Worth while thoughts in the Cisco Blog:

Towards an industry standard for benchmarking AI

 In tandem with this transformative process is the need for industry standards for benchmarking hardware and software systems and how they handle different workloads. These standards are used for comparison between systems, and more importantly, create benchmarks that are used to drive innovation, fueling an iterative process resulting in higher performing systems at lower cost and more efficient energy usage. The unique qualities of AI introduce new challenges, in particular, how to characterize performance and total cost of ownership (TCO). As such, it is critical for organizations like the Transaction Processing Performance Council (TCP) to develop standards that can be used by vendors, customers and researchers. ... 

 .... To this end, the TPC has announced the formation of a new Working Group (TPC-AI), and I am honored to have been elected chairman. The TPC-AI Working Group is tasked with developing industry standard benchmarks for both hardware and software platforms associated with running Artificial Intelligence based workloads. We will be working to define a level playing field for vendors, identify the areas with the greatest potential for improvement through performance optimization, and understand what are the key factors for customers when making their purchase decisions.

I encourage organizations that are interested in participating in the benchmarking development process to join the TPC.

Thank you,
  (Chairman, TPC AI)     Raghunath Nambiar ...