19.5.3 HDF5 File Array Configuration Options

Topic Version2Published04/16/2018
For StandardPRODML v2.0

DAS data arrays can be very large (especially raw arrays) and can exceed the capacity for individual HDF5 files (i.e., exceed the space available on a disk). For this reason, it is possible to split arrays across multiple physical HDF5 files. It is currently best practice to transfer only one raw array per DAS acquisition EPC package because the raw files can become extremely large. Multiple processed arrays can be transferred within one DAS acquisition EPC package, and this makes sense because they are all expected to be processed from the one raw array.

There are several options for how to store these combinations of DAS array data in HDF5 files, which are listed in the table below and shown in Figure 19.5.3-1 and Figure 19.5.3-2 . Other combinations are of course possible.

Configuration Options

Description of Contents

Processed in separate file
  • One or more files that contain a raw array split across them.
  • The processed data is in one or more files.

Processed with own raw data
  • One or more files that contain a raw array split across them.
  • Each file contains its sub-section of the raw array and that sub-section’s corresponding processed array(s.

Hybrid of 1 and 2
  • One or more files that contain a raw array split across them.
  • The processed array(s) for the whole of the raw array are in one of the files along with one sub-section of the raw array.

Raw only

(Recommended)
  • One or more files that contain a raw array split across them and no processed arrays.
  • Typically used when one company does the acquisition and another company does any subsequent processing.

Processed only
  • One file containing the processed arrays, with no raw arrays.
  • Typically used when one company does the acquisition and another company does the processing.

FBE only

(Recommended
  • One or more files that contain FBE arrays split across them along the time axis.
  • Typically used because (1) one company does the acquisition and another company does the processing and (2) for file management purposes because FBE files are smaller than the corresponding spectra array.

Spectra only

(Recommended)
  • One or more files that contain spectra arrays split across them along the time axis.
  • Typically used because (1) one company does the acquisition and another company does the processing and (2) for file management purposes because spectra data arrays tend to be larger than FBE arrays.
Figure 19.5.3-1 Possible HDF5 (.h5) file configurations for use with DAS (1 of 2).
Figure 19.5.3-2 Possible HDF5 (.h5) file configurations for use with DAS (2 of 2).