Ctrl+K

Data processing functionalities

Data processing functionalities#

Summary of subpacakges#

Echopype currently contains the following data processing subpacakges:

  • calibrate: Transform raw instrument data into physicaly meaningful quantities, such as the volume backscattering strength (Sv)

    • Narrowband Sv: AZFP, EK60, and EK80 narrowband (“CW”) mode transmission

    • Broadband Sv average over the signal frequency band: EK80 broadband (“FM”) mode transmission

  • consolidate: Add additional information such as the geospatial locations, split-beam angle, etc to calibrated datasets. The source of information include:

    • Data already stored in the EchoData object, and hence already stored in the raw instrument data files

    • Quantities that can be computed from data stored in the EchoData object, e.g., for EK80 broadband data, split-beam angle can be computed from the complex samples

    • External dataset, e.g., AZFP echosounders do not record geospatial data, but typically there is a companion dataset with GPS information

  • clean: Reduce variabilities in backscatter data by perform noise removal operations. Currently contains only a simple noise removal function implementing the algorithm in [DRH07].

  • commongrid: Enhance the spatial and temporal coherence of data. Currently contains functions to compute mean volume backscattering strength (MVBS) that result in gridded data at uniform spatial and temporal intervals based on either number of indices or label values (phyiscal units).

  • qc: Handle unexpected irregularities in the data. Currently contains only functions to handle timestamp reversal in EK60/EK80 raw data.

  • mask: Create or apply mask to segment data

  • metrics: Calculate simple summary statistics from data

Note

The clean and commongrid subpackages were introduced in version 0.7.0. They contain functions previously found in the deprecated preprocess subpackage; preprocess was removed in version 0.8.0.

The use of these processing functions are summarized below:

Calibration#

nc_path = './converted_files/file.nc'  # path to a converted nc file
echodata = ep.open_converted(nc_path)  # create an EchoData object
ds_Sv = ep.calibrate.compute_Sv(echodata)  # obtain a dataset containing Sv, echo_range, and
                                           # the calibration and environmental parameters

Note

The calibration functions can require different input arguments depending on the echosounder instrument (sonar_model). See data-processing:calibration for detail.

Noise removal#

# Remove noise
ds_Sv_clean = ep.clean.remove_noise(    # obtain a denoised Sv dataset
    ds_Sv,                # calibrated Sv dataset
    range_sample_num=30,  # number of samples along the range_sample dimension for estimating noise
    ping_num=5,           # number of pings for estimating noise
)

Reduce data by computing MVBS#

# Reduce data based on physical units
ds_MVBS = ep.commongrid.compute_MVBS(
    ds_Sv,               # calibrated Sv dataset
    range_meter_bin=20,  # bin size to average along echo_range in meters
    ping_time_bin='20S'  # bin size to average along ping_time in seconds
)

# Reduce data based on sample number
ds_MVBS = ep.commongrid.compute_MVBS_index_binning(
    ds_Sv,                # calibrated Sv dataset
    range_sample_num=30,  # number of sample bins to average along the range_sample dimensionm
    ping_num=5            # number of pings to average
)

Saving results#

Typically echopype functions do not save the calculation resuls to disk, but the returned xarray Dataset can be saved using native xarray method such as to_netcdf and to_zarr. For the EchoData object we have implemented identical methods to allow saving raw converted data to disk. For example:

ed = ep.open_raw("PATH_TO_RAW_FILE", sonar_model="SONAR_MODEL")
ed.to_zarr("RAW_CONVERTED_FILENAME.zarr")  # save converted raw data to Zarr format

# Some processing functions that results in an xarray Dataset ds_Sv
ds_Sv.to_netcdf("PROCESSED_FILENAME.nc")  # save data to netCDF format

Parameter considerations for calibrating data#

Calibration transforms raw data collected by the instrument into physically meaningful units by incorporating:

  • Calibration parameters (cal_params) instrinsic to the instrument and its settings

  • Environmental parameters (env_params), such as sound speed and absorption coefficient due to the physical environment

Echopype also requires correct input argument combinations to calibrate data, due to intrinsic differences of the echosounders.

These considerations are explained below.

For more information on calibration, see [DBB+15].

Input arguments#

AZFP and EK60 data#

For data from the AZFP and EK60 echosounders, since these instruments can only transmit narrowband data, you do not have to specify any additional argument when calibrating data, and the following should always work:

ed = ep.open_raw("PATH_TO_RAW_FILE", sonar_model="EK60")
ds_Sv = ep.calibrate.compute_Sv(ed)

EK80 data#

For data from the EK80 echosounder, since both narrowband and broadband transmissions are possible for different channels, and for narrowband transmissions the data can be stored in two forms, you need to specify argument combinations corresponding to the data you want to calibrate.

For computing band-averaged Sv from broadband EK80 data, use:

ds_Sv = ep.calibrate.compute_Sv(ed, waveform_mode="BB", encode_mode="complex")

Here, waveform_mode="BB" indicates that the data you want to calibrate are the channels set to do broadband transmissions. encode_mode="complex" indicates that these data are stored as complex samples. The function will raise an error if there are no broadband data found in the provided EchoData object (ed).

For computing narrowband Sv from narrowband EK80 data:

  • If the data is stored as complex samples, use

ds_Sv = ep.calibrate.compute_Sv(ed, waveform_mode="CW", encode_mode="complex")

  • If the data is stored as power/angle samples (this is the format that is equivalent to EK60 data), use

ds_Sv = ep.calibrate.compute_Sv(ed, waveform_mode="CW", encode_mode="power")

Calibration parameters#

Echopype by default uses calibration parameters stored in the converted files. However, since careful calibration is often done separately from the data collection phase of the field work, accurate calibration parameters are often supplied in the post-processing stage. This can be done via passing in a dictionary cal_params containing calibration parameters needed to overwrite values stored in the raw data files like below:

ds_Sv = ed.calibrate.compute_Sv(
    ed,
    cal_params={
        sa_correction: sa_vals,
        equivalent_beam_angle: eba_vals,
    }
)

Calibration parameters are instrument specific, and currently we support the following:

EK60_CAL_DICT = {
    "sa_correction",
    "gain_correction",
    "equivalent_beam_angle",
    "angle_offset_alongship",
    "angle_offset_athwartship",
    "angle_sensitivity_alongship",
    "angle_sensitivity_athwartship",
    "beamwidth_alongship",
    "beamwidth_athwartship",
}

EK80_CAL_DICT = {
    "sa_correction",
    "gain_correction",
    "equivalent_beam_angle",
    "angle_offset_alongship",
    "angle_offset_athwartship",
    "angle_sensitivity_alongship",
    "angle_sensitivity_athwartship",
    "beamwidth_alongship",
    "beamwidth_athwartship",
    "impedance_transducer",   # z_et
    "impedance_transceiver",  # z_er
    "receiver_sampling_frequency",
}

AZFP_CAL_DICT = {
    "EL",
    "DS",
    "TVR",
    "VTX",
    "equivalent_beam_angle",
    "Sv_offset",
}

One could choose to bypass the above by updating only specific values in the EchoData object (calibration parameters are in the Sonar/Beam_groupX and the Vendor_specifc group). This may be useful if only one value needs to be updated among all channels.

Environmental parameters#

Environmental parameters, including temperature, salinity, pressure, and pH, are critical in processing raw data collected by the echosounders. They influence calibration through sound speed and absorption coefficients, both central to calibrating raw data to physically meaningful quantities:

  • Absorption coefficients are frequency-dependent and determine the compensation applied to the raw data to compensate for the transmission loss resulted from absortion due to seawater. The higher the frequency, the stronger the absorption and the more sensitive of the calibrated data values are to absorption.

  • Sound speed impacts the calculation of range at each echo samples (echo_range in the output of compute_Sv) recorded at discrete time intervals (sample_interval in ed["Sonar/Beam_groupX"]), which in turn affects the compensation of absorption and the transmission loss resulted from spreading loss.

Default behavior#

By default, echopype uses environmental parameters stored in the EchoData object parsed from the raw file and users need to supply any required parameters that are missing from the data file.

  • EK60 and EK80: Environmental parameters stored in the EchoData object parsed from the raw file. For these instruments, users may enter environmental parameter values into the software, where the sound speed and absorption coefficients are automatically calculated and stored; alternatively, sound speed may be entered manually. Note only sound speed and sound absorption are saved into the raw file. See relevant sections of the reference manuals for EK60 and EK80 for detail.

  • AZFP: Users need to supply at least salinity and pressure as those are not recorded by the instrument. By default echopype uses temperature values stored in the EchoData object, but as these are recorded by the instrument and do not necessarily capture the averaged properties of the water column, users are recommended to supply custom temperature values.

Echopype does not currently handle calculation based on a sound speed profile.

Formulae used#

When custom environmental values are supplied, echopype calculates sound speed and absorption coefficients based on the following:

  • EK60 and EK80:

    • sound speed: [Mac81]

    • absorption coefficients: [AM98]

  • AZFP: formulae provided in the AZFP Operator’s Manual from ASL Environmental Sciences

To supply custom environmetnal parameters, use input argument env_params:

env_params = {
    "temperature": 8,  # temperature in degree Celsius
    "salinity": 30,    # salinity in PSU
    "pressure": 50,    # pressure in dbar
}
ds_Sv = ep.calibrate.compute_Sv(ed, env_params=env_params)

Sanity checks#

Values of the calibration and environmental parameters, as well as the resultant sound speed and absorption coefficints are stored in the output of compute_Sv for easy access and verification:

# Check environmental parameters used in compute_Sv
print(
    f"Temperature used is: {ds_Sv["temperature"]}"
    f"Salinity used is: {ds_Sv["salinity"]}",
    f"Pressure used is: {ds_Sv["pressure"]}",
)

# Check absorption coefficients computed and used
ds_Sv['sound_absorption']  # absorption coefficients [dB/m]

# Check sound speed absorption coefficients computed and used in compute_Sv
ds_Sv['sound_speed']  # sound speed [m/s]

Interfacing Echoview ECS file#

Echopype is able to interface with Echoview ECS file for EK60 and EK80 echosounder starting from v0.7.0. The ECS file contains specifications for both calibration parameters and environmental parameters. When an ECS file is provided, parameters in cal_params and env_params, if provided, are ignored. To provide an ECS file, use:

ds_Sv = ep.calibrate.compute_Sv(ed, ecs_file="PATH_TO_ECS_FILE")

Note

The vocalbury currently implemented was assembled from example files we have access to and may not include all aliases. We have connected with Echoview and will add all aliases in a future release, as well as ECS support for the AZFP echosounder.

Contents