FPBench Notes for Implementors

How to use FPBench in your project

As the FPBench feature set grows, implementing FPBench can look more and more intimidating. But it shouldn't. These notes are our collective advice on implementing the FPBench standards, and should clarify the standards.

What you can skip

Not every feature of the standard needs to be implemented. For example,

Metadata properties, such as :math-library, are optional.
Operations and constants, such as log2 or PI, are optional
Syntactic constructs, such as the different number formats, are optional
Semantic constructs, such as conditionals and loops, are optional
Rounding modes and precisions, such as binary32, are optional

Focus on correctness for features you do implement, rather than the breadth of features you support. Plus, correctly implementing a feature usually only requires care, while implementing new features can require fundamental research.

Different interpretations are more harmful to interoperability than missing features. Tools should, however, publish which features they support. Users can then use tools like the FPBench filter script to select the subsets of their benchmarks supported by a tool.

What you can reject

Tools are free to reject any FPCore as unsupported. The FPCore standard is quite flexible, but this flexibility means that it is difficult for a tool to behave equally well on all inputs. For example, mixed-precision operations can be tedious to implement, while loops may be outside the purview of many techniques.

Tools should, however, make sure to raise an error when an FPCore is unsupported, instead of simply returning invalid results. For example, a tool that does not support mixed precision operations should raise an error instead of ignoring the precision annotations and producing incorrect results.

What you can add

Tools are free to add features to the standard, as long as they namespace them to avoid conflicts. For example, tools can add:

Custom metadata properties, such as :herbie-status.
Custom operations or constants, such as median or PHI
Custom syntactic constructs, such as new number formats or strings
Custom semantic constructs, such as nondeterminism or fixpoints
Custom rounding modes and precisions, such as posit16 or (fixed 12 4)
Custom types, such as complex numbers or matrices

Namespaced names, like rosa-accuracy are future-proof, while it is possible that future standards might define standard functions like median. Where multiple plausible interpretations exist, namedspaced names are better.