Open converted files

Open a converted netCDF or Zarr dataset

Converted netCDF files can be opened with the open_converted function that returns a lazy loaded EchoData object (only metadata are read during opening):

import echopype as ep
file_path = "./converted_files/file.nc"      # path to a converted nc file
ed = ep.open_converted(file_path)            # create an EchoData object

Likewise, specify the path to open a Zarr dataset. To open such a dataset from cloud storage, use the same storage_options parameter as with open_raw. For example:

s3_path = "s3://s3bucketname/directory_path/dataset.zarr"     # S3 dataset path
ed = ep.open_converted(s3_path, storage_options={"anon": True})

Combine EchoData objects

Converted data found in multiple files corresponding to the same instrument deployment can be combined into a single EchoData object. First assemble a list of EchoData objects from the converted files (netCDF or Zarr). Then apply combine_echodata on this list to combine all the data into a single EchoData object in memory:

ed_list = []
for converted_file in ["convertedfile1.nc", "convertedfile2.nc"]:
   ed_list.append(ep.open_converted(converted_file))

combined_ed = ep.combine_echodata(ed_list)

EchoData object

EchoData is an object that conveniently handles raw converted data from either raw instrument files (via open_raw) or previously converted and standardized raw files (via open_converted). It is essentially a container for multiple xarray Dataset objects, where each such object corresponds to one of the netCDF4 groups specified in the SONAR-netCDF4 convention followed by echopype. EchoData objects are used for conveniently accessing and exploring the echosounder data, for calibration and other processing, and for serializing into netCDF4 or Zarr file formats.

A sample EchoData object is presented below using the Dataset HTML browser generated by xarray, collected into SONAR-netCDF4 groups. Select each group and drill down to variables and attributes to examine the structure and representative content of an EchoData object.

EchoData: standardized raw data from Internal Memory
    • <xarray.Dataset>
      Dimensions:  ()
      Data variables:
          *empty*
      Attributes:
          conventions:                 CF-1.7, SONAR-netCDF4-1.0, ACDD-1.3
          keywords:                    EK60
          sonar_convention_authority:  ICES
          sonar_convention_name:       SONAR-netCDF4
          sonar_convention_version:    1.0
          summary:                     EK60 raw file s3://ncei-wcsd-archive/data/ra...
          title:                       2017 Pacific Hake Acoustic Trawl Survey
          date_created:                2017-07-28T18:16:19Z
          survey_name:                 

    • <xarray.Dataset>
      Dimensions:                 (frequency: 3, ping_time: 529)
      Coordinates:
        * frequency               (frequency) float64 1.8e+04 3.8e+04 1.2e+05
        * ping_time               (ping_time) datetime64[ns] 2017-07-28T18:16:19.31...
      Data variables:
          absorption_indicative   (frequency, ping_time) float64 0.002822 ... 0.03259
          sound_speed_indicative  (frequency, ping_time) float64 1.481e+03 ... 1.48...

    • <xarray.Dataset>
      Dimensions:        (location_time: 2165, frequency: 3, ping_time: 529)
      Coordinates:
        * location_time  (location_time) datetime64[ns] 2017-07-28T18:16:21.4759997...
        * frequency      (frequency) float64 1.8e+04 3.8e+04 1.2e+05
        * ping_time      (ping_time) datetime64[ns] 2017-07-28T18:16:19.313999872 ....
      Data variables:
          latitude       (location_time) float64 dask.array<chunksize=(2165,), meta=np.ndarray>
          longitude      (location_time) float64 dask.array<chunksize=(2165,), meta=np.ndarray>
          sentence_type  (location_time) <U3 dask.array<chunksize=(2165,), meta=np.ndarray>
          pitch          (frequency, ping_time) float64 dask.array<chunksize=(3, 529), meta=np.ndarray>
          roll           (frequency, ping_time) float64 dask.array<chunksize=(3, 529), meta=np.ndarray>
          heave          (frequency, ping_time) float64 dask.array<chunksize=(3, 529), meta=np.ndarray>
          water_level    (frequency, ping_time) float64 dask.array<chunksize=(3, 529), meta=np.ndarray>
      Attributes:
          platform_type:       Research vessel
          platform_name:       Bell M. Shimada
          platform_code_ICES:  315

    • <xarray.Dataset>
      Dimensions:        (location_time: 22037)
      Coordinates:
        * location_time  (location_time) datetime64[ns] 2017-07-28T18:16:19.3140003...
      Data variables:
          NMEA_datagram  (location_time) <U73 '$SDVLW,5050.149,N,5050.149,N' ... '$...
      Attributes:
          description:  All NMEA sensor datagrams

    • <xarray.Dataset>
      Dimensions:  ()
      Data variables:
          *empty*
      Attributes:
          conversion_software_name:     echopype
          conversion_software_version:  0.5.5.dev72+g0bfc55b
          conversion_time:              2022-01-28T19:15:00Z
          src_filenames:                s3://ncei-wcsd-archive/data/raw/Bell_M._Shi...
          duplicate_ping_times:         0

    • <xarray.Dataset>
      Dimensions:  ()
      Data variables:
          *empty*
      Attributes:
          sonar_manufacturer:      Simrad
          sonar_model:             ER60
          sonar_serial_number:     
          sonar_software_name:     
          sonar_software_version:  2.4.3
          sonar_type:              echosounder

    • <xarray.Dataset>
      Dimensions:                         (frequency: 3, ping_time: 529, range_bin: 3957)
      Coordinates:
        * frequency                       (frequency) float64 1.8e+04 3.8e+04 1.2e+05
        * ping_time                       (ping_time) datetime64[ns] 2017-07-28T18:...
        * range_bin                       (range_bin) int64 0 1 2 3 ... 3954 3955 3956
      Data variables: (12/30)
          channel_id                      (frequency) <U37 'GPT  18 kHz 009072058c8...
          beam_type                       (frequency) int64 1 1 1
          beamwidth_receive_alongship     (frequency) float64 10.9 6.81 6.58
          beamwidth_receive_athwartship   (frequency) float64 10.82 6.85 6.52
          beamwidth_transmit_alongship    (frequency) float64 10.9 6.81 6.58
          beamwidth_transmit_athwartship  (frequency) float64 10.82 6.85 6.52
          ...                              ...
          data_type                       (frequency, ping_time) float64 3.0 ... 3.0
          count                           (frequency, ping_time) float64 3.957e+03 ...
          offset                          (frequency, ping_time) float64 0.0 ... 0.0
          transmit_mode                   (frequency, ping_time) float64 0.0 ... 0.0
          angle_athwartship               (frequency, ping_time, range_bin) float64 ...
          angle_alongship                 (frequency, ping_time, range_bin) float64 ...
      Attributes:
          beam_mode:              vertical
          conversion_equation_t:  type_3

    • <xarray.Dataset>
      Dimensions:           (frequency: 3, pulse_length_bin: 5)
      Coordinates:
        * frequency         (frequency) float64 1.8e+04 3.8e+04 1.2e+05
        * pulse_length_bin  (pulse_length_bin) int64 0 1 2 3 4
      Data variables:
          sa_correction     (frequency, pulse_length_bin) float64 0.0 -0.7 ... -0.3
          gain_correction   (frequency, pulse_length_bin) float64 20.3 22.95 ... 26.55
          pulse_length      (frequency, pulse_length_bin) float64 0.000512 ... 0.00...