FPBench Logo

FPBench Notes for Implementors

Getting started with FPBench

As the FPBench feature set grows, implementing FPBench can look more and more intimidating. But it shouldn't. These notes are our collective advice on implementing the FPBench standards, and should clarify the standards.

What you can skip

Not every feature of the standard needs to be implemented. For example,

Focus on correctness for features you do implement, rather than the breadth of features you support. Plus, correctly implementing a feature usually only requires care, while implementing new features can require fundamental research.

Different interpretations are more harmful to interoperability than missing features. Tools should, however, publish which features they support. Users can then use tools like the FPBench filter script to select the subsets of their benchmarks supported by a tool.

What you can reject

Tools are free to reject any FPCore as unsupported. The FPCore standard is quite flexible, but this flexibility means that it is difficult for a tool to behave equally well on all inputs. For example, mixed-precision operations can be tedious to implement, while loops may be outside the purview of many techniques.

Tools should, however, make sure to raise an error when an FPCore is unsupported, instead of simply returning invalid results. For example, a tool that does not support mixed precision operations should raise an error instead of ignoring the precision annotations and producing incorrect results.

What you can add

Tools are free to add features to the standard, as long as they namespace them to avoid conflicts. For example, tools can add:

Namespaced names, like rosa-accuracy are future-proof, while it is possible that future standards might define standard functions like median. Where multiple plausible interpretations exist, namedspaced names are better.