v0.5.x to v0.6.0 changes

In order to enhance the compliance of echopype-generated datasets to the SONAR-netCDF4 version 1 convention, a number of changes were introduced in echopype v0.6.0 that create incompatibilities with the data structure used in previous versions.

To ease the transition, the open_converted function is able to open files previously converted using echopype v0.5.x (0.5.0 to 0.5.6) into the v0.6.0 data format, encapsulated in the EchoData object.

Key changes involved renaming and restructuring a couple of groups, and renaming some coordinates and data variables, as summarized below:

Type

v0.5.x

v0.6.0

Rationale and notes

Group

Beam

Sonar/Beam_group1

Convention compliance

Group

Beam_power

Sonar/Beam_group2

Convention compliance

Group

Vendor

Vendor_specific

Convention compliance

Coordinate

frequency

channel

Accommodate channels with duplicated frequencies. The new variable frequency_nominal was introduced

Coordinate

range_bin

range_sample

Better intuitive understanding of data

Coordinate

quadrant

beam

Convention compliance. This Beam_groupX coordinate was added when it did not exist.

Coordinate

location_time

time1

Convention compliance. In Platform group

Coordinate

mru_time

time2

Convention compliance. In Platform group

Variable

heave

vertical_offset

Convention compliance. In Platform group

Variable

src_filenames

source_filenames

Convention compliance. In Provenance group. Also converted from global attribute to variable

Other changes included:

  • Adding previously missing, mandatory convention variables. When no data are available to populate them, these are filled with null (NaN) values.

  • Moving variables from one group to another, particularly from the Beam groups to Platform and Vendor. These variables were not typically not part of the convention.

  • The Beam_groupX beamwidth_receive_athwartship and beamwidth_transmit_athwartship variables were consolidated into beamwidth_twoway_athwartship because the EK60 and EK80 echosounders do not store one-way transmit or receive beam widths. Likewise for beamwidth_receive_alongship and beamwidth_transmit_alongship.

More details, including Pull Requests and discussions related to these changes, can be found in the Release notes.

Convert old files to v0.6.0 format

To convert data files from v0.5.x to v0.6.0 format, simply open the old files and re-save them:

import echopype as ep
# open old v0.5.x file and convert it into a v0.6.0-format EchoData object
ed = ep.open_converted("old_format_file.nc")
ed.to_netcdf("new_format_file.nc")

v0.5.x data format

Below we provide a sample of the v0.5.x data format via a printout of the previous EchoData object.

Compare this with the v0.6.0 EchoData object to see the changes listed in the table above.

EchoData: standardized raw data from Internal Memory
    • <xarray.Dataset>
      Dimensions:  ()
      Data variables:
          *empty*
      Attributes:
          conventions:                 CF-1.7, SONAR-netCDF4-1.0, ACDD-1.3
          keywords:                    EK60
          sonar_convention_authority:  ICES
          sonar_convention_name:       SONAR-netCDF4
          sonar_convention_version:    1.0
          summary:                     EK60 raw file s3://ncei-wcsd-archive/data/ra...
          title:                       2017 Pacific Hake Acoustic Trawl Survey
          date_created:                2017-07-28T18:16:19Z
          survey_name:                 

    • <xarray.Dataset>
      Dimensions:                 (frequency: 3, ping_time: 529)
      Coordinates:
        * frequency               (frequency) float64 1.8e+04 3.8e+04 1.2e+05
        * ping_time               (ping_time) datetime64[ns] 2017-07-28T18:16:19.31...
      Data variables:
          absorption_indicative   (frequency, ping_time) float64 0.002822 ... 0.03259
          sound_speed_indicative  (frequency, ping_time) float64 1.481e+03 ... 1.48...

    • <xarray.Dataset>
      Dimensions:        (location_time: 2165, frequency: 3, ping_time: 529)
      Coordinates:
        * location_time  (location_time) datetime64[ns] 2017-07-28T18:16:21.4759997...
        * frequency      (frequency) float64 1.8e+04 3.8e+04 1.2e+05
        * ping_time      (ping_time) datetime64[ns] 2017-07-28T18:16:19.313999872 ....
      Data variables:
          latitude       (location_time) float64 dask.array<chunksize=(2165,), meta=np.ndarray>
          longitude      (location_time) float64 dask.array<chunksize=(2165,), meta=np.ndarray>
          sentence_type  (location_time) <U3 dask.array<chunksize=(2165,), meta=np.ndarray>
          pitch          (frequency, ping_time) float64 dask.array<chunksize=(3, 529), meta=np.ndarray>
          roll           (frequency, ping_time) float64 dask.array<chunksize=(3, 529), meta=np.ndarray>
          heave          (frequency, ping_time) float64 dask.array<chunksize=(3, 529), meta=np.ndarray>
          water_level    (frequency, ping_time) float64 dask.array<chunksize=(3, 529), meta=np.ndarray>
      Attributes:
          platform_type:       Research vessel
          platform_name:       Bell M. Shimada
          platform_code_ICES:  315

    • <xarray.Dataset>
      Dimensions:        (location_time: 22037)
      Coordinates:
        * location_time  (location_time) datetime64[ns] 2017-07-28T18:16:19.3140003...
      Data variables:
          NMEA_datagram  (location_time) <U73 '$SDVLW,5050.149,N,5050.149,N' ... '$...
      Attributes:
          description:  All NMEA sensor datagrams

    • <xarray.Dataset>
      Dimensions:  ()
      Data variables:
          *empty*
      Attributes:
          conversion_software_name:     echopype
          conversion_software_version:  0.5.6
          conversion_time:              2022-05-26T18:01:56Z
          src_filenames:                s3://ncei-wcsd-archive/data/raw/Bell_M._Shi...
          duplicate_ping_times:         0

    • <xarray.Dataset>
      Dimensions:  ()
      Data variables:
          *empty*
      Attributes:
          sonar_manufacturer:      Simrad
          sonar_model:             ER60
          sonar_serial_number:     
          sonar_software_name:     
          sonar_software_version:  2.4.3
          sonar_type:              echosounder

    • <xarray.Dataset>
      Dimensions:                         (frequency: 3, ping_time: 529,
                                           range_bin: 3957)
      Coordinates:
        * frequency                       (frequency) float64 1.8e+04 3.8e+04 1.2e+05
        * ping_time                       (ping_time) datetime64[ns] 2017-07-28T18:...
        * range_bin                       (range_bin) int64 0 1 2 3 ... 3954 3955 3956
      Data variables: (12/30)
          channel_id                      (frequency) <U37 'GPT  18 kHz 009072058c8...
          beam_type                       (frequency) int64 1 1 1
          beamwidth_receive_alongship     (frequency) float64 10.9 6.81 6.58
          beamwidth_receive_athwartship   (frequency) float64 10.82 6.85 6.52
          beamwidth_transmit_alongship    (frequency) float64 10.9 6.81 6.58
          beamwidth_transmit_athwartship  (frequency) float64 10.82 6.85 6.52
          ...                              ...
          data_type                       (frequency, ping_time) float64 3.0 ... 3.0
          count                           (frequency, ping_time) float64 3.957e+03 ...
          offset                          (frequency, ping_time) float64 0.0 ... 0.0
          transmit_mode                   (frequency, ping_time) float64 0.0 ... 0.0
          angle_athwartship               (frequency, ping_time, range_bin) float64 ...
          angle_alongship                 (frequency, ping_time, range_bin) float64 ...
      Attributes:
          beam_mode:              vertical
          conversion_equation_t:  type_3

    • <xarray.Dataset>
      Dimensions:           (frequency: 3, pulse_length_bin: 5)
      Coordinates:
        * frequency         (frequency) float64 1.8e+04 3.8e+04 1.2e+05
        * pulse_length_bin  (pulse_length_bin) int64 0 1 2 3 4
      Data variables:
          sa_correction     (frequency, pulse_length_bin) float64 0.0 -0.7 ... -0.3
          gain_correction   (frequency, pulse_length_bin) float64 20.3 22.95 ... 26.55
          pulse_length      (frequency, pulse_length_bin) float64 0.000512 ... 0.00...