.. _fiber_photometry_metadata_structure: Fiber Photometry Metadata Structure =================================== This document describes the fiber photometry metadata system in NeuroConv, used by the single-series fiber photometry interfaces built on ``BaseFiberPhotometryInterface`` (currently ``TDTFiberPhotometryInterface`` and ``DoricFiberPhotometryInterface``). It is intended as a reference for developers contributing new fiber photometry interfaces or modifying existing ones, and to document the design decisions behind the format. A single-series interface writes exactly one ``FiberPhotometryResponseSeries`` (the ndx-fiber-photometry neurodata type), assembled from one or more input *streams* (atomic source signals — a TDT store, a Doric dataset). That series references a *region* of a ``FiberPhotometryTable``, and each row of the table references the physical hardware (optical fiber, excitation source, photodetector, filters) and the indicator that produced the trace — much as an ecephys ``ElectricalSeries`` references a region of the electrodes table. The typical conversion is a single interface writing one response series. That one series can still be multi-channel — a multi-fiber, dual-wavelength recording (a calcium signal and an isosbestic control per fiber) is usually written as one series whose data columns map to several table rows. NeuroConv also supports running several fiber photometry interfaces together, each writing its own series into one shared table (as several ecephys recording interfaces share one electrodes table), but that is the exception, used when you deliberately want distinct response series rather than channels of one series. The purpose of the fiber photometry metadata dict is to describe the hardware once and let each interface point its response series at the table rows it needs. Design Principles ----------------- The fiber photometry metadata system follows the same spirit as every modality (see :ref:`metadata_principles`), specialized to the ndx-fiber-photometry data model: 1. **Its own top-level block, but shared devices.** The fiber-photometry-specific metadata (indicators, viruses/injections, the ``FiberPhotometryTable``, commanded voltage, and the response series) lives under ``metadata["FiberPhotometry"]``, not nested under ``metadata["Ophys"]``. Fiber photometry is technically an optical method, but — like the ecephys/icephys split — it has its own interfaces, its own extension, and its own metadata block. The one exception is the hardware itself: device models and device instances live in the shared, cross-modality top-level registry ``metadata["Devices"]`` / ``metadata["DeviceModels"]``, because ndx-ophys-devices classes such as ``ExcitationSource`` are reused by other modalities and should not grow a fiber-photometry-only writing path. .. code-block:: python metadata["FiberPhotometry"]["FiberPhotometryIndicators"]["gcamp"]["label"] = "GCaMP7b" metadata["DeviceModels"]["ofm"]["numerical_aperture"] = 0.48 2. **Dict-keyed containers, keyed by ``metadata_key``.** Following the ophys metadata system, every container is a dict whose keys are ``metadata_key`` handles. The hardware and biology shared across a file (device models, devices, optical fibers, indicators, viral vectors and injections, commanded voltage, and the table's rows) are keyed this way, and each interface's response series is keyed by its own ``metadata_key``. Keying every level lets several interfaces run in one conversion without clashing and merges cleanly across interfaces (dicts merge by key), and it decouples an object's routing handle from its NWB ``name`` (see :ref:`metadata_key_naming`). 3. **Linking is by ``_metadata_key``, resolved at write time.** Entries reference each other with ``_metadata_key`` fields holding the *key* of the referenced entry — a device names its model via ``device_model_metadata_key`` (into ``metadata["DeviceModels"]``), a table row names its optical fiber via ``optical_fiber_metadata_key`` (into ``metadata["Devices"]``), and a response series names the table rows it spans via ``fiber_photometry_table_region`` (a list of row keys). This matches the ophys convention. At write time each key is resolved through its container to the entry's ``name`` and then to the actual NWB object; row references resolve to integer row indices (their position in the merged rows dict). 4. **One shared, keyed-row table.** A file has a single ``FiberPhotometryTable``. Every response series references a *region* of its rows by row key, so regions never depend on fragile integer indices. 5. **No fabricated provenance.** ``get_metadata()`` returns only what the interface can source from the files — the response-series entry (and the session start time where the format embeds it) — and never fabricates devices, indicators, or table rows the data does not contain (see :ref:`metadata_principles`). With no user metadata an interface writes a bare ``FiberPhotometryResponseSeries``, which is a valid file. The full provenance chain is opt-in: supply it yourself. When a table *is* supplied, ``add_to_nwbfile`` requires the chain to be complete and raises on missing or dangling pieces rather than writing partial provenance. Metadata Structure Overview --------------------------- The complete fiber photometry metadata structure (one interface with ``metadata_key="gcamp_dms"``): .. code-block:: python metadata = { "NWBFile": {...}, # Session-level metadata "Subject": {...}, # Subject information # ----- Shared, cross-modality device registry (top-level) ----- # Device models. Each entry carries a `type` naming its concrete ndx-ophys-devices class. "DeviceModels": { "ofm": {"type": "OpticalFiberModel", "name": "optical_fiber_model", "manufacturer": "Doric Lenses", "numerical_aperture": 0.48}, "esm": {"type": "ExcitationSourceModel", "name": "excitation_source_model", "manufacturer": "Doric Lenses", "source_type": "LED", "excitation_mode": "one-photon"}, "pdm": {"type": "PhotodetectorModel", "name": "photodetector_model", "manufacturer": "Doric Lenses", "detector_type": "photodiode"}, # (also BandOpticalFilterModel, EdgeOpticalFilterModel, DichroicMirrorModel) }, # Device instances. Each carries a `type` and links its model by `device_model_metadata_key`. "Devices": { "fiber_dms": {"type": "OpticalFiber", "name": "optical_fiber_dms", "device_model_metadata_key": "ofm", "fiber_insertion": {"depth_in_mm": 4.2, "insertion_position_ap_in_mm": 0.8}}, "led_465": {"type": "ExcitationSource", "name": "excitation_source_465", "device_model_metadata_key": "esm"}, "pd": {"type": "Photodetector", "name": "photodetector", "device_model_metadata_key": "pdm"}, # (also BandOpticalFilter, EdgeOpticalFilter, DichroicMirror) }, "FiberPhotometry": { # ----- Shared containers: dicts keyed by metadata_key (built once per file) ----- # Indicators, and optionally the viruses/injections that delivered them "FiberPhotometryViruses": { "aav_gcamp": {"name": "aav_gcamp", "construct_name": "AAV-GCaMP7b"}, }, "FiberPhotometryVirusInjections": { "inj_dms": {"name": "injection_dms", "viral_vector_metadata_key": "aav_gcamp"}, }, "FiberPhotometryIndicators": { "gcamp": {"name": "gcamp", "label": "GCaMP7b", "viral_vector_injection_metadata_key": "inj_dms"}, }, # Optional commanded-voltage drive signals (data-bearing; read from a stream) "CommandedVoltageSeries": { "cv_dms": {"name": "commanded_voltage_dms", "stream_name": "Fi1r", "index": 0, "unit": "volts", "frequency": 211.0}, }, # The single shared table, rows keyed by metadata_key. Each row = one fiber x one channel. "FiberPhotometryTable": { "name": "fiber_photometry_table", "description": "Fiber photometry setup.", "rows": { "dms_465": { "location": "DMS", "excitation_wavelength_in_nm": 465.0, "emission_wavelength_in_nm": 520.0, "indicator_metadata_key": "gcamp", "optical_fiber_metadata_key": "fiber_dms", "excitation_source_metadata_key": "led_465", "photodetector_metadata_key": "pd", "commanded_voltage_series_metadata_key": "cv_dms", # optional }, }, }, # ----- Per-interface response series, keyed by metadata_key ----- # (unit is not metadata — it is set when the series is built; see below) "gcamp_dms": { "name": "FiberPhotometryResponseSeries", "description": "GCaMP7b calcium signal in DMS.", "fiber_photometry_table_region": ["dms_465"], # list of ROW metadata_keys "fiber_photometry_table_region_description": "DMS calcium signal.", }, }, } The same structure with its blanks unfilled, in YAML and JSON, is at :ref:`fiber_photometry_metadata_template`. The metadata_key Parameter -------------------------- Every single-series interface accepts a keyword-only ``metadata_key`` parameter, following the same pattern as the ophys interfaces (see :ref:`ophys_metadata_structure`). When ``None`` (the default), the interface generates a key from the parameters that make it unique — for fiber photometry, the ``stream_names`` — so two interfaces over different streams get distinct keys automatically: .. code-block:: python interface = TDTFiberPhotometryInterface(folder_path=..., stream_names="_405R") interface.metadata_key # -> "fiber_photometry_405r" ``metadata_key`` scopes only this interface's **response series** entry. The shared containers are keyed independently: devices, indicators, and table rows each have their own ``metadata_key`` handles, so if several interfaces are combined they can reference the same optical fiber or table row by key regardless of their own response-series keys. This split is deliberate — the response series is the interface's own output (one per interface), while the hardware is shared file-wide. Note that the response series' NWB ``name`` (``"FiberPhotometryResponseSeries"`` by default) is distinct from its ``metadata_key`` (see :ref:`metadata_key_naming`). The key identifies the interface's entry; the ``name`` names the object written to the file. Combine multiple interfaces in a converter and give each a distinct ``metadata_key`` (and typically a distinct series ``name``) to write several series into one file. Shared Containers and Metadata-Key Linking ------------------------------------------ Following the ophys system, entries reference each other with ``_metadata_key`` fields holding the *key* of the referenced entry (not its NWB name). The reference chain is: - A **device instance** (in ``metadata["Devices"]``) references its **model** (in ``metadata["DeviceModels"]``): ``"device_model_metadata_key": "ofm"``. - A **virus injection** references its **viral vector**: ``"viral_vector_metadata_key": "aav_gcamp"``. - An **indicator** optionally references its **injection**: ``"viral_vector_injection_metadata_key": "inj_dms"``. - A **table row** references its **devices and indicator** via ``optical_fiber_metadata_key``, ``excitation_source_metadata_key``, ``photodetector_metadata_key``, ``indicator_metadata_key``, and optionally ``dichroic_mirror_metadata_key``, ``excitation_filter_metadata_key``, ``emission_filter_metadata_key``, and ``commanded_voltage_series_metadata_key``. The device keys point into the shared ``metadata["Devices"]`` registry. - A **response series** references the **table rows** it spans: ``"fiber_photometry_table_region": ["dms_465", "dms_405"]`` (a list of row keys). At write time each ``_metadata_key`` is resolved through its container to the entry's ``name`` and then to the actual NWB object (e.g. an ``optical_fiber_metadata_key`` resolves to ``Devices[key]["name"]`` and then ``nwbfile.devices[name]``). A device instance and its model are separate entries linked by ``device_model_metadata_key``, written through the shared device helpers (``_add_device_to_nwbfile`` / ``_add_device_model_to_nwbfile``); an entry's ``type`` selects the concrete ndx-ophys-devices class. Optical fibers are the one special case: each carries a nested ``fiber_insertion`` block (stereotaxic coordinates and insertion geometry), a *contained* component specified inline rather than a separately-keyed object. The FiberPhotometryTable and Regions ------------------------------------ A file has exactly one ``FiberPhotometryTable``. Each row describes one fiber × one excitation channel — its brain ``location``, its excitation/emission wavelengths, the indicator, and the hardware that recorded it. A multi-fiber, dual-wavelength setup therefore has several rows (e.g. ``dms_465``, ``dms_405``, ``dls_465``, ``dls_405``). Each ``FiberPhotometryResponseSeries`` references a **region** of the table — the subset of rows whose channels make up that series' data columns — via ``fiber_photometry_table_region``, a list of **row metadata keys**. At write time these keys are resolved to integer row indices (their position in the merged ``rows`` dict), so the region ordering matches the data's channel ordering and no interface needs to know the absolute index a row landed at. Referencing a row key that is not present in the table is a loud error. This key-based resolution is what makes multi-interface conversions tractable: interface B can point its region at rows that interface A contributed, purely by key. Object Creation and Idempotency ------------------------------- Like the ophys system, objects are not created when the metadata dict is assembled — they are created during ``add_to_nwbfile``. The shared containers (devices, indicators, and the table) are built **once per file**, which matters in the less-common case where several interfaces contribute to one table: 1. The device/indicator/table helpers are idempotent by NWB name. The interface assembles the whole ``FiberPhotometry`` lab-metadata (devices, indicators, viruses, injections, and every table row) from its metadata; if a later interface finds it already present, it reuses it and adds only its own response series. 2. When several interfaces run together, their metadata is merged before writing (by the converter's ``dict_deep_update``, which merges the keyed containers by key), so every interface sees the full set of shared rows and devices and whichever runs first can build the complete table. Because devices and device models live in the single shared ``metadata["Devices"]`` / ``metadata["DeviceModels"]`` registry keyed by ``metadata_key``, every interface references the same entry by key — there is no per-interface copy of a shared object that could diverge. Commanded voltage is a special case among the shared containers: it is data-bearing (it reads samples from a stream, not just static metadata). Each ``CommandedVoltageSeries`` entry names the input ``stream_name`` (and optional channel ``index``) to read; a table row then references the resulting series through its ``commanded_voltage_series_metadata_key``. This is how frequency-multiplexed setups associate each demodulated signal with the sinusoidal drive that produced it. Default Metadata ---------------- ``get_metadata()`` returns only what the interface can derive from the source files: the NWBFile basics, the session start time where the format embeds it, and a single response-series entry under the interface's ``metadata_key`` holding just its ``name`` (see :ref:`metadata_principles`). It fabricates no devices, indicators, or table rows — none of that is contained in the data — and no ``description`` (left empty unless a source or user supplies one). ``unit`` is deliberately absent: it is a property of the data, not editable metadata, so it is set when the ``FiberPhotometryResponseSeries`` is built (defaulting to ``"a.u."`` for uncalibrated signals, or the source's calibrated unit if one is reported). With no further metadata, ``add_to_nwbfile`` writes a bare ``FiberPhotometryResponseSeries``; this is a valid NWB file, since the series' ``fiber_photometry_table_region`` is optional. To record the optical hardware, indicator, and table, supply that metadata yourself, following the complete structure shown in "Metadata Structure Overview" above. Provenance is all-or-nothing. ``add_to_nwbfile`` requires the response series' ``fiber_photometry_table_region`` and the ``FiberPhotometryTable`` to be present together — each row with its required fields and resolvable device/indicator references — and raises on anything missing or dangling. The writer records real provenance or writes none.