Working with converted files

Open a converted netCDF or Zarr dataset

Converted netCDF files can be opened with the open_converted function that returns a lazy-loaded EchoData object (only metadata are read during opening):

import echopype as ep
file_path = "./converted_files/file.nc"      # path to a converted nc file
ed = ep.open_converted(file_path)            # create an EchoData object

Likewise, specify the path to open a Zarr dataset. To open such a dataset from cloud storage, use the same storage_options parameter as with open_raw. For example:

s3_path = "s3://s3bucketname/directory_path/dataset.zarr"     # S3 dataset path
ed = ep.open_converted(s3_path, storage_options={"anon": True})

Combine EchoData objects

Warning

The combine_echodata function will be deprecated in version v0.12.1. Instead, we recommend using ep.calibrate.compute_Sv to compute volume backscattering strength (Sv) for each EchoData object individually, then combining the resulting Sv datasets rather than combining EchoData objects directly. See Combine volume backscattering strength (Sv) datasets for more details.

Data collected by the same instrument deployment across multiple files can be combined into a single EchoData object using combine_echodata. With the release of echopype version 0.6.3, one can now combine a large number of files in parallel (using Dask) while maintaining a stable memory usage. This is done under-the-hood by concatenating data directly into a Zarr store, which corresponds to the final combined EchoData object.

To use combine_echodata, the following criteria must be met:

• Each EchoData object must have the same sonar_model

• The EchoData objects to be combined must correspond to different raw data files (i.e., no duplicated files)

• The EchoData objects in the list must be of sequential order in time. Specifically, the first timestamp of each EchoData object must be smaller (earlier) than the first timestamp of the subsequent EchoData object

• The EchoData objects must contain the same frequency channels and the same number of channels

• The following attribute criteria must be satisfied for all groups under each of the EchoData objects to be combined:

  • the names of all attributes must be the same

  • the values of all attributes must be identical (other than the attributes date_created or conversion_time; these attributes should have the same data type)

Attention

In previous versions, combine_echodata corrected reversed timestamps and stored the uncorrected timestamps in the Provenance group. Starting from 0.6.3, combine_echodata will preserve time coordinates that have reversed timestamps and no correction is performed.

The first step in combining data is to establish a Dask client with a scheduler. On a local machine, this can be done as follows:

client = Client()  # create client with local scheduler

With distributed resources, we highly recommend reviewing the Dask documentation for deploying Dask clusters.

Next, we assemble a list of EchoData objects. This list can be from converted files (netCDF or Zarr) as in the example below, or from in-memory EchoData objects:

ed_list = []
for converted_file in ["convertedfile1.zarr", "convertedfile2.zarr"]:
    ed_list.append(ep.open_converted(converted_file))  # already converted files are lazy-loaded

Finally, we apply combine_echodata on this list to combine all the data into a single EchoData object. Here, we will store the final combined form in the Zarr path path_to/combined_echodata.zarr and use the client we established above:

combined_ed = ep.combine_echodata(
    ed_list, 
    zarr_path="path_to/combined_echodata.zarr", 
    client=client
)

Once executed, combine_echodata returns a lazy loaded EchoData object (obtained from zarr_path) with all data from the input EchoData objects combined.

Note

As shown in the above example, the path of the combined Zarr store is given by the keyword argument zarr_path, and the Dask client that parallel tasks will be submitted to is given by the keyword argument client. When either (or both) of these are not provided, default values listed in the Notes section in combine_echodata will be used.

Combine volume backscattering strength (Sv) datasets

For handling dataset consisting of many individual raw instrument-generated data files, we recommend compute Sv separately for each EchoData object (each corresponding to a raw data file), save each Sv dataset to Zarr, and combine each Sv dataset. The operations below use tools from Xarray and Zarr to efficiently manage large datasets. For more information on working with Zarr datasets, see the Xarray Zarr documentation and the Zarr documentation.

The recommended order of operations:

1. Compute Sv for each EchoData object

2. Rechunk each Sv dataset before writing

3. Save each Sv dataset as a Zarr store

4. Combine the individual Zarr stores using xarray.open_mfdataset

5. Save the combined Sv dataset as a Zarr store

6. Read the final combined Sv dataset

import echopype as ep
import xarray as xr

sv_paths = []

# Save each echodata object to an Sv zarr file
for file_index, ed in enumerate(echodata_list):
    ds_Sv = ep.calibrate.compute_Sv(ed) # Insert necessary parameters

    # Uncomment this rechunking code if the user needs to optimize speed or memory usage.
    # Chunk sizes are chosen based on dataset size, access patterns, and available RAM.
    # ds_Sv = ds_Sv.chunk({"channel": 1, "ping_time": 500, "range_sample": 500})

    # Save to zarr
    file_path = f"Sv_{file_index}.zarr"
    ds_Sv.to_zarr(
        file_path,
        mode="w",
        overwrite=True,
        compute=True,
    )

    sv_paths.append(file_path)

# Open all Sv datasets
ds_Sv_combined = xr.open_mfdataset(
    sv_paths,
    engine="zarr",
    data_vars="minimal",
    coords="minimal",
)

# Can similarly rechunk `ds_Sv_combined` as was done previously

# Save the combined Sv dataset as a Zarr store
ds_Sv_combined.to_zarr(
    "combined_Sv.zarr",
    mode="w",
    compute=True,
    consolidated=True,
)

# Read the combined Sv dataset
ds_Sv_combined = xr.open_zarr(
    "combined_Sv.zarr",
    chunks={},
)