Converting Multiple Sessions into a Dataset

The NWBConverter page covers combining several streams into one NWB file, which is one session. An experiment is usually many sessions, and converting all of them means running that same conversion once per session and then organizing the collection of files that comes out.

Whatever that loop looks like, it should write a flat collection of uniquely named NWB files into one output folder. The organization steps at the end of this page all take that folder as their input.

Finding and Converting the Sessions

LocalPathExpander finds the sessions and reads subject_id, session_id and session_start_time out of the paths, as described in Path Expansion. Each entry it returns carries a source_data dictionary shaped for the converter and a metadata dictionary holding what the path names.

from pathlib import Path

from neuroconv import NWBConverter
from neuroconv.datainterfaces import SpikeGLXRecordingInterface, PhySortingInterface
from neuroconv.tools.path_expansion import LocalPathExpander
from neuroconv.utils.dict import load_dict_from_file, dict_deep_update

class ExampleNWBConverter(NWBConverter):
    data_interface_classes = dict(
        SpikeGLXRecording=SpikeGLXRecordingInterface,
        PhySorting=PhySortingInterface
    )

source_data_spec = {
    "SpikeGLXRecording": {
        "base_directory": "/path/to/raw_data",
        "file_path": "{subject_id}/{session_id}/{session_id}_g0_imec0/{session_id}_g0_imec0.ap.bin"
    },
    "PhySorting": {
        "base_directory": "/path/to/processed_data",
        "folder_path": "{subject_id}/{session_id}/phy"
    }
}

output_folder_path = Path("/path/to/output")
output_folder_path.mkdir(exist_ok=True)

lab_metadata = load_dict_from_file(file_path="my_lab_metadata.yml")

sessions = LocalPathExpander().expand_paths(source_data_spec)
for session in sessions:
    converter = ExampleNWBConverter(source_data=session["source_data"])

    metadata = converter.get_metadata()
    metadata = dict_deep_update(metadata, lab_metadata)
    metadata = dict_deep_update(metadata, session["metadata"])

    subject_id = metadata["Subject"]["subject_id"]
    session_id = metadata["NWBFile"]["session_id"]
    converter.run_conversion(
        nwbfile_path=output_folder_path / f"sub-{subject_id}_ses-{session_id}.nwb",
        metadata=metadata
    )

The metadata is merged in three passes so that the more specific source wins: what the interfaces read from the data files, then the fields shared by every session in the dataset (see Using YAML to specify metadata), then what the path expander recovered for this particular session.

Organizing the Converted Files

The output folder is now a flat collection of NWB files, which is the input format the tools below expect. NeuroConv stops here: the layouts these produce are maintained elsewhere, and which one you want depends on where the data is going.

Uploading to DANDI

Upload to DANDI when the dataset is going to be published, so that the archive stores it, gives it a citable identifier and serves it for streaming.

automatic_dandi_upload() organizes the folder into the DANDI layout and uploads it to a Dandiset you have already created. It needs your API token in the DANDI_API_KEY environment variable, and the dandi extra installed.

from neuroconv.tools.data_transfers import automatic_dandi_upload

automatic_dandi_upload(dandiset_id="123456", nwb_folder_path="/path/to/output")

Every file needs a session_id in its metadata, since DANDI requires one.

Reorganizing into BIDS

Reorganize into BIDS when the dataset feeds tooling that expects that layout, or has to sit beside other modalities of the same study in a single dataset.

NeuroConv does not write a BIDS (Brain Imaging Data Structure) directory layout. BIDS organization is a separate step over finished NWB files, so convert first and reorganize afterwards with nwb2bids:

pip install nwb2bids
nwb2bids convert /path/to/output --bids-directory /path/to/bids

nwb2bids renames the files and directories to BIDS conventions and fills the sidecar TSV and JSON files from metadata already in the NWB files, so the more complete your conversion metadata is, the more complete the BIDS output will be. Its participants.tsv and sessions.tsv are tables across the dataset, which is why this step belongs to a collection of files rather than to a single conversion. It targets BEP032, the microelectrode electrophysiology extension covering extracellular and intracellular recordings and their associated behavioral events. That extension is still under formal review and has not been merged into the BIDS specification, so the layout it produces may still change.