.. _annotate_events_metadata: How to Annotate Discrete Events Metadata ======================================== This guide provides instructions for annotating discrete-event data (TTL lines, strobes, epocs, markers) using NeuroConv's dict-based events metadata format. It covers what the events are called and how they are laid out in the file. Which events exist is decided when the interface is constructed, and for a source that carries its events as a sampled signal rather than as a list, that is the ``detection_configuration`` argument covered in :ref:`extract_events_from_signals`. .. _annotate_events_single_interface: How to Annotate Events from a Single Interface ---------------------------------------------- In the simplest case one interface reads one source. Construct it and convert with no metadata editing at all; the interface extracts everything needed for a faithful file. .. code-block:: python from neuroconv.datainterfaces import TDTEventsInterface interface = TDTEventsInterface(folder_path="path/to/tank", metadata_key="behavioral_session") nwbfile = interface.create_nwbfile() For a tank with two epoc stores, ``PtAB`` (port-entry codes) and ``PC0_`` (a reward marker), this writes one ``EventsTable`` per store, **named after the store**, with the raw values written as-is and no value labels: .. code-block:: text PtAB (EventsTable in /events; the table is named after the store's event_type_source_id) ┌───────────┬────────┐ │ timestamp │ strobe │ <- the value column, named after its field_source_id; raw codes, no MeaningsTable ├───────────┼────────┤ │ 12.084 │ 64959 │ │ 13.553 │ 65535 │ │ ... │ ... │ └───────────┴────────┘ PC0_ (EventsTable; a bare marker, so timestamps only) ┌───────────┐ │ timestamp │ ├───────────┤ │ 8.500 │ │ 19.310 │ │ ... │ └───────────┘ This is already correct: every event, every timestamp, every raw value is preserved. What it lacks is *meaning*, the table is named ``PtAB``, its value column ``strobe``, and the code ``64959`` has no human label. Annotation supplies that, and it is just editing the dict ``get_metadata()`` seeds. Each recipe below edits that dict; apply the ones you need, then pass the result to ``create_nwbfile``. **Inspect what the interface extracted.** Print the seeded dict to see the event types the interface found (each keyed by its ``event_type_source_id``) and the seeded defaults: .. code-block:: python metadata = interface.get_metadata() print(metadata["Events"]) .. code-block:: python { "behavioral_session": { # the interface's metadata_key "event_types": { "PtAB": { # one entry per event type found (its event_type_source_id) "event_name": "PtAB", # seeded to the source id; names its own table "columns": { # value columns, keyed by field_source_id "strobe": { # the strobe value field "column_name": "strobe", # seeded to the field_source_id }, }, }, "PC0_": { # constant value -> a bare marker (timestamps only) "event_name": "PC0_", }, # no columns }, }, } There is no ``EventTables`` block: a solo event type names its own table from its ``event_name`` (here the seeded source ids ``PtAB``/``PC0_``), so ``EventTables`` appears only once you merge several event types into one shared table (shown further down). **Rename an event column.** ``column_name`` is the column header in the output table, seeded to the ``field_source_id``. Reach the value column through its event type and rename it: .. code-block:: python column = metadata["Events"]["behavioral_session"]["event_types"]["PtAB"]["columns"]["strobe"] column["column_name"] = "choice" The column header changes; the values are still the raw codes (this edit only renamed the column): .. code-block:: text PtAB (table not renamed yet) ┌───────────┬────────┐ │ timestamp │ choice │ ├───────────┼────────┤ │ 12.084 │ 64959 │ │ 13.553 │ 65535 │ │ ... │ ... │ └───────────┴────────┘ **Label categorical values.** For a code-like column, ``column_categories["labels"]`` maps each raw value the hardware emitted to the display text written in its place: .. code-block:: python column["column_categories"] = {"labels": {64959: "left", 65023: "center", 65535: "right"}} The cells now show the labels instead of the raw codes: .. code-block:: text PtAB ┌───────────┬────────┐ │ timestamp │ choice │ ├───────────┼────────┤ │ 12.084 │ left │ │ 13.553 │ right │ │ ... │ ... │ └───────────┴────────┘ **Describe what each value means.** ``column_categories["meanings"]`` gives each label a longer description. The descriptions become a ``MeaningsTable`` that is **contained in the events table** (in its ``meanings_tables`` group), named after the column it describes (``choice_meanings``): .. code-block:: python column["column_categories"]["meanings"] = { 64959: "Subject chose the left port", 65023: "Subject chose the center port", 65535: "Subject chose the right port", } The ``choice`` column is unchanged; the events table now holds a ``choice_meanings`` MeaningsTable describing each value: .. code-block:: text PtAB (EventsTable) ┌───────────┬────────┐ │ timestamp │ choice │ ├───────────┼────────┤ │ 12.084 │ left │ │ 13.553 │ right │ │ ... │ ... │ └───────────┴────────┘ meanings_tables/ choice_meanings (MeaningsTable, describes the choice column) ┌────────┬───────────────────────────────┐ │ value │ meaning │ ├────────┼───────────────────────────────┤ │ left │ Subject chose the left port │ │ center │ Subject chose the center port │ │ right │ Subject chose the right port │ └────────┴───────────────────────────────┘ **Rename or describe the output table.** A solo event type names its own table from its ``event_name`` (CamelCased for the NWB object name) and ``event_description``. Edit those to rename or describe the table: .. code-block:: python event_type = metadata["Events"]["behavioral_session"]["event_types"]["PtAB"] event_type["event_name"] = "choices" # table becomes "Choices" event_type["event_description"] = "Subject's choice (left, center, right)." The table is now named ``Choices``. The final state is the events table with its contained ``choice_meanings`` MeaningsTable: .. code-block:: text Choices (EventsTable) ┌───────────┬────────┐ │ timestamp │ choice │ ├───────────┼────────┤ │ 12.084 │ left │ │ 13.553 │ right │ │ ... │ ... │ └───────────┴────────┘ meanings_tables/ choice_meanings (MeaningsTable) ┌────────┬───────────────────────────────┐ │ value │ meaning │ ├────────┼───────────────────────────────┤ │ left │ Subject chose the left port │ │ center │ Subject chose the center port │ │ right │ Subject chose the right port │ └────────┴───────────────────────────────┘ **Apply the edits.** Pass the edited metadata to the conversion: .. code-block:: python nwbfile = interface.create_nwbfile(metadata=metadata) The events are unchanged from the first run, same timestamps and same occurrences; the edits only renamed the table and column, replaced the displayed codes with labels, and added the contained ``choice_meanings`` MeaningsTable. Annotation never alters the events themselves, it makes them legible. **Write a continuous (numeric) column.** When a column carries a self-describing measurement rather than codes (e.g. a TDT ``Freq`` store of stimulus frequencies in Hz), omit ``column_categories`` entirely and the raw numbers are written as a plain numeric column. Carry the unit in the ``column_name`` (e.g. ``"frequency_hz"``). A bare marker like ``PC0_`` carries no value at all: its ``columns`` is empty, so it is already timestamps-only and needs no edit of this kind. .. _annotate_events_shared_table: How to Write Multiple Event Types to a Single EventsTable --------------------------------------------------------- In :ref:`annotate_events_single_interface` we annotated the tank's events. Give the reward marker its own ``Rewards`` table the same way (rename its ``EventTables`` entry) and the interface writes **two clean, separate tables**: .. code-block:: text Choices (EventsTable) Rewards (EventsTable) ┌───────────┬────────┐ ┌───────────┐ │ timestamp │ choice │ │ timestamp │ ├───────────┼────────┤ ├───────────┤ │ 12.084 │ left │ │ 8.500 │ │ 13.553 │ right │ │ 19.310 │ │ ... │ ... │ │ ... │ └───────────┴────────┘ └───────────┘ That is fine when the events are genuinely separate kinds. But sometimes you want the different event types of one interface in **one** table, for semantic grouping or to simplify a downstream read. Point each event type's ``table_metadata_key`` at one shared ``EventTables`` entry, and give each a friendly ``event_name`` (it becomes the type's label in the shared table's ``event_type`` column): .. code-block:: python # Continuing with the annotated metadata (PtAB's column renamed to "choice" and labeled, and its # event_name set to "choices"). Give the reward marker a friendly name too. event_types = metadata["Events"]["behavioral_session"]["event_types"] event_types["PC0_"]["event_name"] = "reward" event_types["PC0_"]["event_description"] = "Reward delivery." # Define the shared table once, then point both event types at it instead of their own tables. metadata["Events"]["EventTables"]["behavior"] = { "table_name": "BehavioralEvents", "description": "Choices and rewards in one table.", } event_types["PtAB"]["table_metadata_key"] = "behavior" # the choice event type event_types["PC0_"]["table_metadata_key"] = "behavior" # the reward marker nwbfile = interface.create_nwbfile(metadata=metadata) Both event types now land in one ``BehavioralEvents`` table, one row per event, ordered by ``timestamp``. Because the rows come from different event types, the shared table carries an ``event_type`` column holding each row's ``event_name``, and each event type contributes only the value columns it carries: the choice event type adds a ``choice`` column, while the ``reward`` marker is a bare timestamp and adds no value column. A reward row is therefore identified by ``event_type = "reward"`` with the ``choice`` cell empty: .. code-block:: text BehavioralEvents (EventsTable) ┌───────────┬────────────┬────────┐ │ timestamp │ event_type │ choice │ ├───────────┼────────────┼────────┤ │ 8.500 │ reward │ │ <- a bare marker: identified by event_type, no choice value │ 12.084 │ choices │ left │ │ 13.553 │ choices │ right │ │ 19.310 │ reward │ │ │ ... │ ... │ ... │ └───────────┴────────────┴────────┘ meanings_tables/ event_type_meanings (MeaningsTable, describes each event type) ┌─────────┬─────────────────────────────────────────┐ │ value │ meaning │ ├─────────┼─────────────────────────────────────────┤ │ choices │ Subject's choice (left, center, right). │ │ reward │ Reward delivery. │ └─────────┴─────────────────────────────────────────┘ The ``event_type`` column is what makes sharing a table safe: without it a bare marker (which adds no value column) would be an all-blank row, indistinguishable from any other, and its identity would be lost. With it, every row knows its type regardless of which value columns it fills. Each row's ``event_type`` value is the type's ``event_name`` (``choices`` / ``reward``), and the column carries an ``event_type_meanings`` MeaningsTable mapping each name to its ``event_description``, so the merged table documents its own types. The ``choice`` column keeps its ``choice_meanings`` MeaningsTable, now also contained in ``BehavioralEvents``. By default each event type stays in its own table (no ``event_type`` column needed, since the table *is* the type); you share a table when the grouping is worth it, and the writer adds the ``event_type`` column (with its meanings) to keep the rows distinguishable and named. Putting event types from *different* interfaces into one table works exactly the same way and is shown next; the only difference is the columns live under different ``metadata_key`` s. How to Annotate Multiple Events Interfaces ------------------------------------------ A single conversion often runs several event interfaces, here a TDT tank and a SpikeGLX NIDQ stream, wired together in an ``NWBConverter``. Each interface gets its own ``metadata_key``, and that key is what keeps them apart: two sources can expose the **same** ``event_type_source_id`` (two tanks both with a store ``PtAB``, two boards both with a line ``XD0``), and the ``metadata_key`` namespaces each interface's ``event_types`` block so those identical ids never clash. .. code-block:: python from neuroconv.datainterfaces import TDTEventsInterface, SpikeGLXNIDQInterface from neuroconv import NWBConverter tdt_interface = TDTEventsInterface(folder_path="path/to/tank", metadata_key="tdt") # Signal-encoded interfaces also take an events-extraction config at construction # (which lines/levels, and how to read them); elided here to keep the focus on metadata. nidq_interface = SpikeGLXNIDQInterface(file_path="path/to/run.nidq.bin", metadata_key="nidq") converter = NWBConverter(data_interfaces={"tdt": tdt_interface, "nidq": nidq_interface}) metadata = converter.get_metadata() # Each interface's event types live under its own metadata_key, so identical event_type_source_ids # from different interfaces never collide: metadata["Events"]["tdt"] vs ["nidq"]. metadata["Events"]["tdt"]["event_types"]["PtAB"]["columns"]["strobe"]["column_name"] = "choice" # XD0 is a bare marker (no value column); name its table via event_name. metadata["Events"]["nidq"]["event_types"]["XD0"]["event_name"] = "camera_frame" # table -> "CameraFrame" So far each event type keeps its own default table, giving separate tables per interface. Sharing a table **across** interfaces works exactly like sharing within one (above): point each event type's ``table_metadata_key`` at a shared ``EventTables`` entry. The only difference is the event types live under different ``metadata_key`` s; the shared ``EventTables`` block is global, so event types from any interface can route into it. .. code-block:: python # A shared table for events from both interfaces. metadata["Events"]["EventTables"]["behavior"] = { "table_name": "BehavioralEvents", "description": "Rewards (TDT) and licks (NIDQ) sharing one table across interfaces.", } # A TDT reward marker and a NIDQ lick line, both bare markers, routed into the one shared table, # each given a friendly event_name that becomes its label in the event_type column. reward = metadata["Events"]["tdt"]["event_types"]["PC0_"] reward["event_name"], reward["event_description"] = "reward", "Reward delivery." reward["table_metadata_key"] = "behavior" lick = metadata["Events"]["nidq"]["event_types"]["XD1"] lick["event_name"], lick["event_description"] = "lick", "Lick detection." lick["table_metadata_key"] = "behavior" converter.run_conversion(nwbfile_path="session.nwb", metadata=metadata) Both event types land in one ``BehavioralEvents`` table. As with sharing within a single interface, the shared table carries an ``event_type`` column holding each row's ``event_name`` (with an ``event_type_meanings`` MeaningsTable built from the ``event_description`` s), and each type contributes only the value columns it has. Both ``reward`` and ``lick`` here are bare markers (timestamps only, no value), so they add no value column at all, the table is just timestamps tagged by type: .. code-block:: text BehavioralEvents (EventsTable) ┌───────────┬────────────┐ │ timestamp │ event_type │ ├───────────┼────────────┤ │ 8.500 │ reward │ <- from the TDT interface │ 11.200 │ lick │ <- from the NIDQ interface │ 19.310 │ reward │ │ ... │ ... │ └───────────┴────────────┘ If a shared type did carry a value (e.g. a labeled choice), it would add its own column, filled only on its rows, exactly as in the single-interface case above. When several pooled types record the *same* quantity, give them one shared column by declaring the identical ``column_name`` (with matching ``column_categories`` and ``description``) on each; the writer merges them into a single column filled from every type's rows instead of one column per type. Declaring the same ``column_name`` with conflicting meanings or descriptions is an error. If you have a use case not covered here, please open an issue at `NeuroConv GitHub Issues `_.