Data processing functionalities#
Summary of subpacakges#
Echopype currently contains the following data processing subpacakges:
calibrate
: Transform raw instrument data into physicaly meaningful quantities, such as the volume backscattering strength (Sv)Narrowband Sv: AZFP, EK60, and EK80 narrowband (“CW”) mode transmission
Broadband Sv average over the signal frequency band: EK80 broadband (“FM”) mode transmission
consolidate
: Add additional information such as the geospatial locations, split-beam angle, etc to calibrated datasets. The source of information include:Data already stored in the
EchoData
object, and hence already stored in the raw instrument data filesQuantities that can be computed from data stored in the
EchoData
object, e.g., for EK80 broadband data, split-beam angle can be computed from the complex samplesExternal dataset, e.g., AZFP echosounders do not record geospatial data, but typically there is a companion dataset with GPS information
clean
: Reduce variabilities in backscatter data by perform noise removal operations. Currently contains only a simple noise removal function implementing the algorithm in [DRH07].commongrid
: Enhance the spatial and temporal coherence of data. Currently contains functions to compute mean volume backscattering strength (MVBS) that result in gridded data at uniform spatial and temporal intervals based on either number of indices or label values (phyiscal units).qc
: Handle unexpected irregularities in the data. Currently contains only functions to handle timestamp reversal in EK60/EK80 raw data.mask
: Create or apply mask to segment datametrics
: Calculate simple summary statistics from data
Note
The clean
and commongrid
subpackages were introduced in version 0.7.0.
They contain functions previously found in the deprecated preprocess
subpackage;
preprocess
was removed in version 0.8.0.
The use of these processing functions are summarized below:
Calibration#
nc_path = './converted_files/file.nc' # path to a converted nc file
echodata = ep.open_converted(nc_path) # create an EchoData object
ds_Sv = ep.calibrate.compute_Sv(echodata) # obtain a dataset containing Sv, echo_range, and
# the calibration and environmental parameters
Note
The calibration functions can require different input arguments depending on the echosounder instrument (sonar_model
). See data-processing:calibration
for detail.
Noise removal#
# Remove noise
ds_Sv_clean = ep.clean.remove_noise( # obtain a denoised Sv dataset
ds_Sv, # calibrated Sv dataset
range_sample_num=30, # number of samples along the range_sample dimension for estimating noise
ping_num=5, # number of pings for estimating noise
)
Reduce data by computing MVBS#
# Reduce data based on physical units
ds_MVBS = ep.commongrid.compute_MVBS(
ds_Sv, # calibrated Sv dataset
range_meter_bin=20, # bin size to average along echo_range in meters
ping_time_bin='20S' # bin size to average along ping_time in seconds
)
# Reduce data based on sample number
ds_MVBS = ep.commongrid.compute_MVBS_index_binning(
ds_Sv, # calibrated Sv dataset
range_sample_num=30, # number of sample bins to average along the range_sample dimensionm
ping_num=5 # number of pings to average
)
Saving results#
Typically echopype functions do not save the calculation resuls to disk, but the returned xarray Dataset
can be saved using native xarray method such as to_netcdf
and to_zarr
. For the EchoData
object we have implemented identical methods to allow saving raw converted data to disk. For example:
ed = ep.open_raw("PATH_TO_RAW_FILE", sonar_model="SONAR_MODEL")
ed.to_zarr("RAW_CONVERTED_FILENAME.zarr") # save converted raw data to Zarr format
# Some processing functions that results in an xarray Dataset ds_Sv
ds_Sv.to_netcdf("PROCESSED_FILENAME.nc") # save data to netCDF format
Parameter considerations for calibrating data#
Calibration transforms raw data collected by the instrument into physically meaningful units by incorporating:
Calibration parameters (
cal_params
) instrinsic to the instrument and its settingsEnvironmental parameters (
env_params
), such as sound speed and absorption coefficient due to the physical environment
Echopype also requires correct input argument combinations to calibrate data, due to intrinsic differences of the echosounders.
These considerations are explained below.
For more information on calibration, see [DBB+15].
Input arguments#
AZFP and EK60 data#
For data from the AZFP and EK60 echosounders, since these instruments can only transmit narrowband data, you do not have to specify any additional argument when calibrating data, and the following should always work:
ed = ep.open_raw("PATH_TO_RAW_FILE", sonar_model="EK60")
ds_Sv = ep.calibrate.compute_Sv(ed)
EK80 data#
For data from the EK80 echosounder, since both narrowband and broadband transmissions are possible for different channels, and for narrowband transmissions the data can be stored in two forms, you need to specify argument combinations corresponding to the data you want to calibrate.
For computing band-averaged Sv from broadband EK80 data, use:
ds_Sv = ep.calibrate.compute_Sv(ed, waveform_mode="BB", encode_mode="complex")
Here, waveform_mode="BB"
indicates that the data you want to calibrate are the channels set to do broadband transmissions. encode_mode="complex"
indicates that these data are stored as complex samples. The function will raise an error if there are no broadband data found in the provided EchoData
object (ed
).
For computing narrowband Sv from narrowband EK80 data:
If the data is stored as complex samples, use
ds_Sv = ep.calibrate.compute_Sv(ed, waveform_mode="CW", encode_mode="complex")
If the data is stored as power/angle samples (this is the format that is equivalent to EK60 data), use
ds_Sv = ep.calibrate.compute_Sv(ed, waveform_mode="CW", encode_mode="power")
Calibration parameters#
Echopype by default uses calibration parameters stored in the converted files. However, since careful calibration is often done separately from the data collection phase of the field work, accurate calibration parameters are often supplied in the post-processing stage. This can be done via passing in a dictionary cal_params
containing calibration parameters needed to overwrite values stored in the raw data files like below:
ds_Sv = ed.calibrate.compute_Sv(
ed,
cal_params={
sa_correction: sa_vals,
equivalent_beam_angle: eba_vals,
}
)
Calibration parameters are instrument specific, and currently we support the following:
EK60_CAL_DICT = {
"sa_correction",
"gain_correction",
"equivalent_beam_angle",
"angle_offset_alongship",
"angle_offset_athwartship",
"angle_sensitivity_alongship",
"angle_sensitivity_athwartship",
"beamwidth_alongship",
"beamwidth_athwartship",
}
EK80_CAL_DICT = {
"sa_correction",
"gain_correction",
"equivalent_beam_angle",
"angle_offset_alongship",
"angle_offset_athwartship",
"angle_sensitivity_alongship",
"angle_sensitivity_athwartship",
"beamwidth_alongship",
"beamwidth_athwartship",
"impedance_transducer", # z_et
"impedance_transceiver", # z_er
"receiver_sampling_frequency",
}
AZFP_CAL_DICT = {
"EL",
"DS",
"TVR",
"VTX",
"equivalent_beam_angle",
"Sv_offset",
}
One could choose to bypass the above by updating only specific values in the EchoData
object (calibration parameters are in the Sonar/Beam_groupX
and the Vendor_specifc
group). This may be useful if only one value needs to be updated among all channels.
Environmental parameters#
Environmental parameters, including temperature, salinity, pressure, and pH, are critical in processing raw data collected by the echosounders. They influence calibration through sound speed and absorption coefficients, both central to calibrating raw data to physically meaningful quantities:
Absorption coefficients are frequency-dependent and determine the compensation applied to the raw data to compensate for the transmission loss resulted from absortion due to seawater. The higher the frequency, the stronger the absorption and the more sensitive of the calibrated data values are to absorption.
Sound speed impacts the calculation of range at each echo samples (
echo_range
in the output ofcompute_Sv
) recorded at discrete time intervals (sample_interval
ined["Sonar/Beam_groupX"]
), which in turn affects the compensation of absorption and the transmission loss resulted from spreading loss.
Default behavior#
By default, echopype uses environmental parameters stored in the EchoData
object parsed from the raw file and users need to supply any required parameters that are missing from the data file.
EK60 and EK80: Environmental parameters stored in the
EchoData
object parsed from the raw file. For these instruments, users may enter environmental parameter values into the software, where the sound speed and absorption coefficients are automatically calculated and stored; alternatively, sound speed may be entered manually. Note only sound speed and sound absorption are saved into the raw file. See relevant sections of the reference manuals for EK60 and EK80 for detail.AZFP: Users need to supply at least salinity and pressure as those are not recorded by the instrument. By default echopype uses temperature values stored in the
EchoData
object, but as these are recorded by the instrument and do not necessarily capture the averaged properties of the water column, users are recommended to supply custom temperature values.
Echopype does not currently handle calculation based on a sound speed profile.
Formulae used#
When custom environmental values are supplied, echopype calculates sound speed and absorption coefficients based on the following:
EK60 and EK80:
AZFP: formulae provided in the AZFP Operator’s Manual from ASL Environmental Sciences
To supply custom environmetnal parameters, use input argument env_params
:
env_params = {
"temperature": 8, # temperature in degree Celsius
"salinity": 30, # salinity in PSU
"pressure": 50, # pressure in dbar
}
ds_Sv = ep.calibrate.compute_Sv(ed, env_params=env_params)
Sanity checks#
Values of the calibration and environmental parameters, as well as the resultant sound speed and absorption coefficints are stored in the output of compute_Sv
for easy access and verification:
# Check environmental parameters used in compute_Sv
print(
f"Temperature used is: {ds_Sv["temperature"]}"
f"Salinity used is: {ds_Sv["salinity"]}",
f"Pressure used is: {ds_Sv["pressure"]}",
)
# Check absorption coefficients computed and used
ds_Sv['sound_absorption'] # absorption coefficients [dB/m]
# Check sound speed absorption coefficients computed and used in compute_Sv
ds_Sv['sound_speed'] # sound speed [m/s]
Interfacing Echoview ECS file#
Echopype is able to interface with Echoview ECS file for EK60 and EK80 echosounder starting from v0.7.0. The ECS file contains specifications for both calibration parameters and environmental parameters. When an ECS file is provided, parameters in cal_params
and env_params
, if provided, are ignored. To provide an ECS file, use:
ds_Sv = ep.calibrate.compute_Sv(ed, ecs_file="PATH_TO_ECS_FILE")
Note
The vocalbury currently implemented was assembled from example files we have access to and may not include all aliases. We have connected with Echoview and will add all aliases in a future release, as well as ECS support for the AZFP echosounder.