Floating Point Precision in Jedox

Any computing that involves numerical calculation must face the perplexing issue of floating point precision, which is a by-product of the binary numbering system used by computers. It results from the binary format used by computers to represent a number. This format cannot accurately represent fractional numbers, so they must be rounded to a near representation of a fraction. For simple mathematical calculations, the resulting inaccuracies are negligible, but when performing aggregated operations on large data sets, errors can be more problematic.

Some systems use compensation mechanisms such as decimal computation or algorithms to reduce aggregated error, but these mechanisms take a toll on performance. For simple spreadsheets and calculators, the decrease in performance may go unnoticed, but for OLAP systems such as Jedox, the performance penalty may be too great. All Jedox-supported CPUs use the IEEE 754 Standard for Floating-Point Arithmetic, which is a technical standard for floating-point computation. These CPUs use double-precision binary floating-point format, which is precise from 15 to 17 decimal digits. Internally, Jedox operates with precision up to 15 digits, so the difference shown when raising the number of visible decimals will not influence further calculations.

Jedox users occasionally confront the issue of floating point precision when selecting Suppress zero values in Paste View or Subset Editor. Values that should be suppressed nevertheless appear in results, because they are nearly – but not quite – zero (i.e., .000000000000015). Other unexpected results may be due to rounding errors.These issues are not a fault of Jedox (or any other software) but are inherent to all technology that is based on binary encoding.

Floating Point Precision in Jedox

Suggested reading*