Changes in recent versions#

The underlying raw converted data structure used by echopype (the EchoData object) has undergone revisions over time. While in most echopype releases these changes are relatively small, versions 0.8.0 and 0.6.0 incorporated significant changes, with implications for backward compatibility. Here we describe the main changes in each of these major releases. Please refer to What’s new for more complete details on specific changes that impacted the data structure.

v0.7.1 to v0.8.0 changes#

Changes introduced in version 0.8.0 were carried out to incorporate missing variables that are mandated by SONAR-netCDF4 v1 and to implement adaptations to the convention in a more consistent fashion across variables and instrument types. Some of these changes modified or reverted decisions implemented in version 0.6.0 (below) that were later found to have large impacts on performance and usability.

Highlights include:

  • Remove beam and ping_time dimensions from a whole netCDF4 group or individual variables when they were determined to be not required for specific instrument types. The beam dimension is now dropped from all Sonar/Beam_groupX groups except for EK80 complex samples, where both backscatter_r and backscatter_i exist and the beam dimension represents different sectors of split-beam transducers. The ping_time dimension is retained only with variables that are known to potentially vary with time in the instrument types supported by echopype.

  • In Sonar/Beam_groupX groups: Standardize the use of transmit_frequency_start and transmit_frequency_stop, where they were previously missing or the names being used (frequency_start and frequency_end) were not the ones specified by the convention.

  • In the Platform group: Implement variables absed on the convention more consistently across instrument types:

    • Assign default values to variables when no such variables are found in the raw data file

    • Revise the dimensions of each variable to be consistent across instrument types, with dimensions deemed unnecessary dropped from some variables.

  • In the Provenance group: Add new attributes combination_* to the “combined” EchoData object, mirroring the convention-based attributes conversion_*.

  • In the Vendor_specific group: Move filter coefficients and decimation factor from attributes to variables in EK80, to facilitate consistent provenance tracking during combine_echodata operations.

  • Improve the presence and use of variable attributes throughout EchoData groups.

Note

Version 0.8.0 does not incorporate the capability to read files converted by previous versions of echopype. We recommend using open_raw to re-convert the raw data files.

v0.5.x to v0.6.0 changes#

In order to enhance the compliance of echopype-generated datasets to the SONAR-netCDF4 version 1 convention, a number of changes were introduced in echopype v0.6.0 that create incompatibilities with the data structure used in previous versions.

To ease the transition, the open_converted function is able to open files previously converted using echopype v0.5.x (0.5.0 to 0.5.6) into the v0.6.0 data format, encapsulated in the EchoData object.

Key changes involved renaming and restructuring a couple of groups, and renaming some coordinates and data variables, as summarized below:

Type

v0.5.x

v0.6.0

Rationale and notes

Group

Beam

Sonar/Beam_group1

Convention compliance

Group

Beam_power

Sonar/Beam_group2

Convention compliance

Group

Vendor

Vendor_specific

Convention compliance

Coordinate

frequency

channel

Accommodate channels with duplicated frequencies. The new variable frequency_nominal was introduced

Coordinate

range_bin

range_sample

Better intuitive understanding of data

Coordinate

quadrant

beam

Convention compliance. This Beam_groupX coordinate was added when it did not exist.

Coordinate

location_time

time1

Convention compliance. In Platform group

Coordinate

mru_time

time2

Convention compliance. In Platform group

Variable

heave

vertical_offset

Convention compliance. In Platform group

Variable

src_filenames

source_filenames

Convention compliance. In Provenance group. Also converted from global attribute to variable

Other changes included:

  • Adding previously missing, mandatory convention variables. When no data are available to populate them, these are filled with null (NaN) values.

  • Moving variables from one group to another, particularly from the Beam groups to Platform and Vendor. These variables were not typically not part of the convention.

  • The Beam_groupX beamwidth_receive_athwartship and beamwidth_transmit_athwartship variables were consolidated into beamwidth_twoway_athwartship because the EK60 and EK80 echosounders do not store one-way transmit or receive beam widths. Likewise for beamwidth_receive_alongship and beamwidth_transmit_alongship.

More details, including Pull Requests and discussions related to these changes, can be found in the Release notes.

v0.5.x data format#

Below we provide a sample of the v0.5.x data format via a printout of the previous EchoData object.

Compare this with the v0.6.0 EchoData object to see the changes listed in the table above.

EchoData: standardized raw data from Internal Memory
    • <xarray.Dataset>
      Dimensions:  ()
      Data variables:
          *empty*
      Attributes:
          conventions:                 CF-1.7, SONAR-netCDF4-1.0, ACDD-1.3
          keywords:                    EK60
          sonar_convention_authority:  ICES
          sonar_convention_name:       SONAR-netCDF4
          sonar_convention_version:    1.0
          summary:                     EK60 raw file s3://ncei-wcsd-archive/data/ra...
          title:                       2017 Pacific Hake Acoustic Trawl Survey
          date_created:                2017-07-28T18:16:19Z
          survey_name:                 

    • <xarray.Dataset>
      Dimensions:                 (frequency: 3, ping_time: 529)
      Coordinates:
        * frequency               (frequency) float64 1.8e+04 3.8e+04 1.2e+05
        * ping_time               (ping_time) datetime64[ns] 2017-07-28T18:16:19.31...
      Data variables:
          absorption_indicative   (frequency, ping_time) float64 0.002822 ... 0.03259
          sound_speed_indicative  (frequency, ping_time) float64 1.481e+03 ... 1.48...

    • <xarray.Dataset>
      Dimensions:        (location_time: 2165, frequency: 3, ping_time: 529)
      Coordinates:
        * location_time  (location_time) datetime64[ns] 2017-07-28T18:16:21.4759997...
        * frequency      (frequency) float64 1.8e+04 3.8e+04 1.2e+05
        * ping_time      (ping_time) datetime64[ns] 2017-07-28T18:16:19.313999872 ....
      Data variables:
          latitude       (location_time) float64 dask.array<chunksize=(2165,), meta=np.ndarray>
          longitude      (location_time) float64 dask.array<chunksize=(2165,), meta=np.ndarray>
          sentence_type  (location_time) <U3 dask.array<chunksize=(2165,), meta=np.ndarray>
          pitch          (frequency, ping_time) float64 dask.array<chunksize=(3, 529), meta=np.ndarray>
          roll           (frequency, ping_time) float64 dask.array<chunksize=(3, 529), meta=np.ndarray>
          heave          (frequency, ping_time) float64 dask.array<chunksize=(3, 529), meta=np.ndarray>
          water_level    (frequency, ping_time) float64 dask.array<chunksize=(3, 529), meta=np.ndarray>
      Attributes:
          platform_type:       Research vessel
          platform_name:       Bell M. Shimada
          platform_code_ICES:  315

    • <xarray.Dataset>
      Dimensions:        (location_time: 22037)
      Coordinates:
        * location_time  (location_time) datetime64[ns] 2017-07-28T18:16:19.3140003...
      Data variables:
          NMEA_datagram  (location_time) <U73 '$SDVLW,5050.149,N,5050.149,N' ... '$...
      Attributes:
          description:  All NMEA sensor datagrams

    • <xarray.Dataset>
      Dimensions:  ()
      Data variables:
          *empty*
      Attributes:
          conversion_software_name:     echopype
          conversion_software_version:  0.5.6
          conversion_time:              2022-05-26T18:01:56Z
          src_filenames:                s3://ncei-wcsd-archive/data/raw/Bell_M._Shi...
          duplicate_ping_times:         0

    • <xarray.Dataset>
      Dimensions:  ()
      Data variables:
          *empty*
      Attributes:
          sonar_manufacturer:      Simrad
          sonar_model:             ER60
          sonar_serial_number:     
          sonar_software_name:     
          sonar_software_version:  2.4.3
          sonar_type:              echosounder

    • <xarray.Dataset>
      Dimensions:                         (frequency: 3, ping_time: 529,
                                           range_bin: 3957)
      Coordinates:
        * frequency                       (frequency) float64 1.8e+04 3.8e+04 1.2e+05
        * ping_time                       (ping_time) datetime64[ns] 2017-07-28T18:...
        * range_bin                       (range_bin) int64 0 1 2 3 ... 3954 3955 3956
      Data variables: (12/30)
          channel_id                      (frequency) <U37 'GPT  18 kHz 009072058c8...
          beam_type                       (frequency) int64 1 1 1
          beamwidth_receive_alongship     (frequency) float64 10.9 6.81 6.58
          beamwidth_receive_athwartship   (frequency) float64 10.82 6.85 6.52
          beamwidth_transmit_alongship    (frequency) float64 10.9 6.81 6.58
          beamwidth_transmit_athwartship  (frequency) float64 10.82 6.85 6.52
          ...                              ...
          data_type                       (frequency, ping_time) float64 3.0 ... 3.0
          count                           (frequency, ping_time) float64 3.957e+03 ...
          offset                          (frequency, ping_time) float64 0.0 ... 0.0
          transmit_mode                   (frequency, ping_time) float64 0.0 ... 0.0
          angle_athwartship               (frequency, ping_time, range_bin) float64 ...
          angle_alongship                 (frequency, ping_time, range_bin) float64 ...
      Attributes:
          beam_mode:              vertical
          conversion_equation_t:  type_3

    • <xarray.Dataset>
      Dimensions:           (frequency: 3, pulse_length_bin: 5)
      Coordinates:
        * frequency         (frequency) float64 1.8e+04 3.8e+04 1.2e+05
        * pulse_length_bin  (pulse_length_bin) int64 0 1 2 3 4
      Data variables:
          sa_correction     (frequency, pulse_length_bin) float64 0.0 -0.7 ... -0.3
          gain_correction   (frequency, pulse_length_bin) float64 20.3 22.95 ... 26.55
          pulse_length      (frequency, pulse_length_bin) float64 0.000512 ... 0.00...