Jedox Database Files
The Jedox In-Memory DB is an OLAP system that holds data in RAM for fast query times while maintaining a persisted copy of the data on the file system. Data is kept in the “data” directory on the file system. Each database is represented as subdirectory in this directory. Jedox uses internal storage for persisting its own data. You will find at least two databases in every system: System and Config. These two hold meta information about system configuration.
Each individual database contains multiple files inside its directory that hold information relevant only for that database.
To prevent the system from losing its data in the event of power loss, crash, etc., every change in RAM is accompanied by change on the file system, i.e. write into the journal files. To ensure that data is really persisted every second disk cache flush is issued. In addition, a commit of journal files is executed every 5 minutes.
Jedox databases consist of the following file types:
| File type | Description |
|---|---|
| comma-separated values (*.csv) | Contains the data last saved. When saving a database, the database state is transferred (or "committed") from the memory to the .csv files. |
| binary (*.bin) | Same as *.csv, but in binary format |
| checksum (*.sha1) | Binary storage validation files |
| log (*.log) | Contains changes from the current session (since the OLAP Server was last started). These files are also called “journals”. In the case of an unexpected process kill or crash, the server will recover data from the *.log files during startup. This recovery will usually take more time than committing the same data on shutdown would have taken. |
| archive (*.archive) |
On shutdown, the server copies the data from the log file to the archive file. If your database takes up a lot of hard-disk space on the server, you can delete the archive files from the database directory after stopping the Jedox server. The server can be configured NOT to save archives by enabling the setting no-archives. |
| backup files (*.old) | These files are generated when the server has been configured to save data only in binary format (no-csv-save / no-csv-save-dim). These files are not necessary for OLAP functionality. |
| saved-TIMESTAMP (*.saved-TIMESTAMP) |
These files mark the current state of saved and unsaved data in relation to all the database files and journal files. During database transfer, they must be copied with all other files to keep the database loadable. |
In Jedox on-premises environments, we recommend that you always complete a backup before deleting any files from your database directory.
Comma-separated values (.csv) files
The structure of Jedox databases is defined in a comma-separated values (.csv) file database-*.csv. In this file, cubes, dimensions, elements, and their properties are set. The filename includes a UNIX timestamp, i.e. database-1557741467210061.csv.
Note: if the palo.ini key dimension-file-format binary has been set, then database-*.csv will contain only the dimensions, cubes, and their layout, but not all elements in all dimensions. See the section Binary files below for more information.
Each cube that exists in database-*.csv (entries in section [CUBES]) has in turn its own CSV file (if the no-csv-save ini key has not been enabled in palo.ini, as described below in the "Binary .bin files" paragraph), in which structure and cube values are specified further. These files contain the currently saved Jedox values.
Each Jedox cube also has a unique CSV file that contains the description of the rules in the cube (if any).
Below is an excerpt from the Jedox Demo database with the following entries in the file database.csv in section [CUBES]:
0;"#_GROUP_CUBE_DATA";0;1;1;0;0;
1;"#_Months";3;2;3;0;0;
2;"#_GROUP_DIMENSION_DATA_Months";0;2;1;0;0;
3;"#_Years";5;4;3;0;0;
4;"#_GROUP_DIMENSION_DATA_Years";0;4;1;0;0;
5;"#_Regions";7;6;3;0;0;
6;"#_GROUP_DIMENSION_DATA_Regions";0;6;1;0;0;
7;"#_Datatypes";9;8;3;0;0;
8;"#_GROUP_DIMENSION_DATA_Datatypes";0;8;1;0;0;
9;"#_Measures";11;10;3;0;0;
10;"#_GROUP_DIMENSION_DATA_Measures";0;10;1;0;0;
11;"#_Products";13;12;3;0;0;
12;"#_GROUP_DIMENSION_DATA_Products";0;12;1;0;0;
13;"Sales";12;6;2;4;8;10;2;1;1;
14;"#_#_CUBE_";14;1;3;0;0;
There is another csv file for each entry. The individual cubes are described briefly below:
| Cube | Description |
| 0;"#_GROUP_CUBE_DATA";0;1;1;0;0; | Contains the dimension cubes and groups, and the rights of the groups to the cubes. |
| 1;"#_Months";3;2;3;0;0; 3;"#_Years";5;4;3;0;0; 5;"#_Regions";7;6;3;0;0; 7;"#_Datatypes";9;8;3;0;0; 9;"#_Measures";11;10;3;0;0; 11;"#_Products";13;12;3;0;0; |
These six cubes contain the attribute cube for each dimension. |
| 2;"#_GROUP_DIMENSION_DATA_Months";0;2;1;0;0; 4;"#_GROUP_DIMENSION_DATA_Years";0;4;1;0;0; 6;"#_GROUP_DIMENSION_DATA_Regions";0;6;1;0;0; 8;"#_GROUP_DIMENSION_DATA_Datatypes";0;8;1;0;0; 10;"#_GROUP_DIMENSION_DATA_Measures";0;10;1;0;0; 12;"#_GROUP_DIMENSION_DATA_Products";0;12;1;0;0; |
These cubes contain the user group and the dimensions. The rights for each element for each group are registered here. |
| 13;"Sales";12;6;2;4;8;10;2;1;1; | Contains the data for the cube "Sales". |
| 14;"#_#_CUBE_";14;1;3;0;0; | This cube contains the data for the attribute cube for each cube dimension. |
Binary (.bin) files
Cube and dimension data can be stored as binary data, rather than text. The .bin files contain the same data as the corresponding .csv file, just as binary data rather than text. Binary files are usually smaller in size, so loading and saving is much faster than for the .csv files. The binary format greatly speeds up the time needed for processing files on the file system, such as when the server starts up or shuts down. Cube data is stored in a file called database_CUBE_*.bin. Dimension data is stored in a file called database_DIM_*.bin.
Dimensions are not automatically stored in binary format. This option must be enabled in your Jedox configuration with the settings key dimension-file-format binary. Cube and/or dimension data can optionally be stored only in binary format, which improves shutdown performance. This option is enabled with configuration keyno-csv-save (for cube data) and no-csv-save-dim (for dimension data).
Note: unlike .csv files, binary files are not transferable between Linux and Windows systems.
Checksum (*.sha1) files
As of Jedox 2021.3, binary storage validation includes checksums at saving time. This feature detects errors resulting from file system operations or disk failures. Checksums are stored in a "checksums" directory in each database folder and are generated during initial database load. Generating the checksums can slow down the initial database load, but future database loads will be much faster.
The option to validate binary files can be disabled with the configuration key no-checksum.
Log files
For each database defined in Jedox, one or more (depending on the journal files size, or encryption) log files are created. All values entered in the Jedox OLAP Server are protocolled in this file. As the database is saved, the values stored here are simultaneously copied to the respective .archive file. If the OLAP Server wasn't shutdown properly, the .log files are used for data recovery as described above. Note that log files contain OLAP Server version information, and the OLAP Server will only process log files of the same version. If version of a log file is significantly lower than the current OLAP Server version, it's possible that the database can't be loaded. To avoid that, it's recommended to either save all databases explicitly with the OLAP Server API command or, alternatively, restart the OLAP Server once before applying any Jedox updates.
Archive files
All changes are written into the .archive files. Both value changes as well as structural changes are logged together with the name of the user. However, if a database becomes corrupted, there is no generic possibility to recover a database exclusively from .archive files. The files can only be used for tracing changes.
Archive files may be deleted regularly (after a backup) to keep the data quantity in the server system low.
Updated September 27, 2022