Echopype can be installed from PyPI:
$ pip install echopype
or through conda:
$ conda install -c conda-forge echopype
When creating an conda environment to work with echopype, do
$ conda create -c conda-forge --name echopype python=3.8 --file requirements.txt --file requirements-dev.txt
Echopype works for python>=3.7.
Echopype uses Git Large File Storage (Git LFS) to store the binary data and test files used. Git LFS enables the Github repository to remain small while still being able to access the large test files needed for testing. These files are only needed if you plan to work on the code and run the tests locally.
To access the test files, first install Git LFS.
Cloning echopype after installing Git LFS will automatically pull the test data, but if echopype was cloned first, then pull the files from Git LFS by running:
$ git lfs fetch
Echopype has recently migrated to using Git LFS which required removing the large datasets from the history. It is recommended that those who have previously forked echopype delete their fork and fork a new one. Otherwise, pulling form the original repository will result in twice the number of commits due to the re-written history.
Supported file types¶
Echopype currently supports conversion from
.rawfiles generated by Simrad’s EK60 echosounder
.rawfiles generated by Simrad’s EK80 echosounder
.01Afiles generated by ASL Environmental Sciences’ AZFP echosounder
into netCDF (stable) or zarr (beta) files.
We are considering implementing calibration routines for raw beam data from common-found Acoustic Doppler Current Profilers (ADCPs).
File conversion for different types of echosounders is achieved by
using a single interface through the
For data files from the EK60 echosounder, you can do the following in an interactive Python session:
from echopype import Convert dc = Convert('FILENAME.raw') dc.raw2nc()
This will generate a
FILENAME.nc file in the same directory as
For data files from the EK80 broadband echosounder, use
dc = Convert('FILENAME.raw', model='EK80')
to indicate the echosounder type, since the filename extension
.raw does not contain echosounder type information.
For data files from the AZFP echosounder, the conversion requires an
.XML file along with the
.01A data file. The
contains a lot of metadata needed for unpacking the binary data files.
Typically one single
.XML file is associated with all files from the
This can be done by:
from echopype import Convert dc = Convert('FILENAME.01A', 'XMLFILENAME.xml') dc.raw2nc()
raw2nc() to create netCDF4 files,
you should first set
patform_code_ICES, as these values are not recorded in the raw data
files but need to be specified according to the netCDF4 convention.
These parameters will be saved as empty strings unless you specify
them following the example below:
dc.platform_name = 'OOI' dc.platform_type = 'subsurface mooring' dc.platform_code_ICES = '3164' # Platform code for Moorings
platform_code_ICES attribute can be chosen by referencing
the platform code from the
ICES SHIPC vocabulary.
For conversion to zarr files, call method
.raw2zarr()from the same
Convertobject as shown above.
Convertinstance contains all the data unpacked from the raw file, so it is a good idea to clear it from memory once done with conversion.
More conversion options¶
There are optional arguments that you can pass into
that may come in handy.
Save converted files into another folder:
By default the converted
.ncfiles are saved into the same folder as the input files. This can be changed by setting
save_pathto path to a directory.
raw_file_path = ['./raw_data_files/file_01.raw', # a list of raw data files './raw_data_files/file_02.raw', ...] dc = Convert(raw_file_path) # create a Convert object dc.raw2nc(save_path='./unpacked_files') # set the output directory
Each input file will be converted to individual
.ncfiles and stored in the specified directory.
Combine multiple raw data files into one
.ncfile when unpacking:
raw_file_path = ['./raw_data_files/file_01.raw', # a list of raw data files './raw_data_files/file_02.raw', ...] dc = Convert(raw_file_path) # create a Convert object dc.raw2nc(combine_opt=True, # combine all input files when unpacking save_path='./unpacked_files/combined_file.nc')
save_pathhas to be given explicitly when combining multiple files. If
save_pathis only a filename instead of a full path, the combined output file will be saved in the same folder as the raw data files.
Due to flexibility in echosounder settings, some dimensional parameters can change in the middle of the file. For example:
The maximum depth range to which data are collected can change in the middle of a data file in EK60. This happens often when the bottom depth changes.
The sampling interval, which translates to temporal resolution, and thus range resolution, can also change in the middle of the file.
Data from different frequency channels can also be collected with different sampling intervals.
These changes produce different number of samples along range (the
dimension in the converted
.nc file), which are incompatible with the goal
to save the data as a multi-dimensional array that can be easily indexed using xarray.
Echopype accommodates these cases in the following two ways:
When there are changes in the
range_bindimension in the middle of a data file, echopype creates separate files for each consecutive chunk of data with the same number of samples along range and append
_partXXto the converted filename to indicate the existence of such changes. For example, if
datafile.rawcontains changes in the number of samples along range, the converted output will be
When the number of samples along the
range_bindimensions are different for different frequency channels, echopype pads the shorter channels with
NaNto form a multi-dimensional array. We use the data compression option in
xarray.to_zarr()to avoid dramatically increasing the output file size due to padding.
model subpackage and the data processing interface
have been renamed to
Attempts to import
echopype.model and use
EchoData will still
work at the moment but will be deprecated in the future.
EK60 and AZFP narrowband echosounders:
calibration and echo-integration to obtain volume backscattering strength (Sv) from power data.
Simple noise removal by removing data points (set to
NaN) below an adaptively estimated noise floor 1.
Binning and averaging to obtain mean volume backscattering strength (MVBS) from the calibrated data.
EK80 broadband echosounder:
calibration based on pulse compression output in the form of average over frequency.
The steps of performing these analysis for EK60 and AZFP echosounders are summarized below. Additional information will be added for broadband EK80 echosounder as additional functionality is developed.
from echopype import Process nc_path = './converted_files/convertedfile.nc' # path to a converted nc file ed = Process(nc_path) # create a processing object ed.calibrate() # Sv ed.remove_noise() # denoise ed.get_MVBS() # calculate MVBS
By default, these methods do not save the calculation results to disk.
The computation results can be accessed from
ed.MVBS as xarray Datasets with proper dimension labels.
To save results to disk:
ed.calibrate(save=True) # output: convertedfile_Sv.nc ed.remove_noise(save=True) # output: convertedfile_Sv_clean.nc ed.get_MVBS(save=True) # output: convertedfile_MVBS.nc
There are various options to save the results:
# Overwrite the output postfix from _Sv to_Cal: convertedfile_Cal.nc ed.calibrate(save=True, save_postfix='_Cal') # Save output to another directory: ./cal_results/convertedfile_Sv.nc ed.calibrate(save=True, save_path='./cal_results') # Save output to another directory with an arbitrary name ed.calibrate(save=True, save_path='./cal_results/somethingnew.nc')
By default, for noise removal and MVBS calculation, echopype tries to load Sv
already stored in memory (
ed.Sv), or tries to calibrate the raw data to
obtain Sv. If
ed.Sv is empty (i.e., whe calibration operation has not been
performed on the object), echopype will try to load Sv from
the directory containing the converted
.nc file or from the user-specified
path. For example:
Try to do MVBS calculation without having previously calibrated data
from echopype import Process nc_path = './converted_files/convertedfile.nc' # path to a converted nc file ed = Process(nc_path) # create a processing object ed.get_MVBS() # echopype will call .calibrate() automatically
Try to do MVBS calculation with _Sv_clean.nc file previously created in folder ‘another_directory’
from echopype import Process nc_path = './converted_files/convertedfile.nc' # path to a converted nc file ed = Process(nc_path) # create a data processing object ed.get_MVBS(source_path='another_directory', source_postfix='_Sv_clean')
Echopype’s data processing functionality is being developed actively. Be sure to check back here often!
Environmental parameters, including temperature, salinity and pressure, are critical in biological interpretation of ocean sonar data. They influence
Transducer calibration, through seawater absorption. This influence is frequency-dependent, and the higher the frequency the more sensitive the calibration is to the environmental parameters.
Sound speed, which impacts the conversion from temporal resolution of (of each data sample) to spatial resolution, i.e. the sonar observation range would change.
By default, echopype uses the following for calibration:
EK60: Environmental parameters saved with the data files
AZFP: salinity = 29.6 PSU, pressure = 60 dbar, and temperature recorded at the instrument
These parameters should be overwritten when they differ from the actual
environmental condition during data collection.
To update these parameters, simply do the following before
ed.temperature = 8 # temperature in degree Celsius ed.salinity = 30 # salinity in PSU ed.pressure = 50 # pressure in dbar ed.recalculate_environment() # recalculate related parameters
This will trigger recalculation of all related parameters, including sound speed, seawater absorption, thickness of each sonar sample, and range. The updated values can be retrieved with:
ed.seawater_absorption # absorption in [dB/m] ed.sound_speed # sound speed in [m/s] ed.sample_thickness # sample spatial resolution in [m] ed.range # range for each sonar sample in [m]
For AZFP data, echopype updates the sound speed and seawater absorption using the formulae provided by the manufacturer ASL Environmental Sci.
Calibration here refers to the calibration of transducers on an echosounder, which finds the mapping between the voltage signal recorded by the echosounder and the actual (physical) acoustic pressure received at the transducer. This mapping is critical in deriving biological quantities from acoustic measurements, such as estimating biomass. More detail about the calibration procedure can be found in 4.
Echopype by default uses calibration parameters stored in the converted
files along with the backscatter measurements and other metadata parsed
from the raw data file.
However, since careful calibration is often done separately from the
data collection phase of the field work, accurate calibration parameters
are often supplied in the post-processing stage.
Currently echopypy allows users to overwrite calibration parameters for
EK60 data, including
As an example, to reset the equivalent beam angle for 18 kHz only, one can do:
ed.equivalent_beam_angle.loc[dict(frequency=18000)] = -18.02 # set value for 18 kHz only
To set the equivalent beam angle for all channels at once, do:
ed.equivalent_beam_angle = [-17.47, -20.77, -21.13, -20.4 , -30] # set all channels at once
Make sure you use
ed.equivalent_beam_angle.frequency to check
the sequence of the frequency channels first, and always double
check after setting these parameters!
De Robertis A, Higginbottoms I. (2007) A post-processing technique to estimate the signal-to-noise ratio and remove echosounder background noise. ICES J. Mar. Sci. 64(6): 1282–1291.
Mackenzie K. (1981) Nine‐term equation for sound speed in the oceans. J. Acoust. Soc. Am. 70(3): 806-812
Ainslie MA, McColm JG. (1998) A simplified formula for viscous and chemical absorption in sea water. J. Acoust. Soc. Am. 103(3): 1671-1672
Demer DA, Berger L, Bernasconi M, Bethke E, Boswell K, Chu D, Domokos R, et al. (2015) Calibration of acoustic instruments. ICES Cooperative Research Report No. 326. 133 pp.