HPC metrics

In collaboration with the NCCS, we are listing below a range of metrics that should be evaluated for each benchmark.

• Time to solution
• Despite not aiming for production-ready code by the end of the project, we will still keep an eye on the "job-level" turn around and document improvement and potential non-numerics slowdown due to the technology swap.
• Energy
• Light software sampling to document amplitude of TPU’s chip: imprecise but can be easily ran with little overhead.
• Hardware monitoring on selected runs for precise measure: precise but requires close cooperation with NCCS sys admin and IT.
• Node-to-node
• Compare CPU node with GPU nodes
• Minimize generation difference for valid comparison
• Node usage
• Chip usage: measure in % of theoretical throughput rather than FLOP/s
• Chip idle time: important for hybrid work
• Minimal hardware requirements
• For developments
• For scientific runs