Annotate across levels

Some decisions are not about a note but about a patient: is this patient eligible for the study, given all their notes? Annotators then need to read every note of a patient before answering. In this tutorial, you add a patients table above the notes, and a view where selecting a patient shows their notes, and selecting a note its entities.

Before you start

Follow Design your view first: this tutorial adds patients to its oncology.pret file.

Add the patients

Build one row per patient from the notes, with the number of notes and an eligible column to fill:

from pret import DataFrame, load_store

store = load_store("~/metanno-home/annotations/oncology.pret")
notes = store["notes"].materialize().to_pandas()
patients = notes.groupby("patient_id", as_index=False).agg(note_count=("note_id", "count"))
store["patients"] = DataFrame(patients.assign(eligible=False))
store["patients"].materialize().to_pandas()

	patient_id	note_count	eligible
0	10	2	False
1	12	2	False

The file now holds three tables, linked by two columns: patient_id between patients and notes, note_id between notes and entities.

Write the view

%%writefile ~/metanno-home/recipes/patient_review.py
from typing import Annotated, TypedDict

from pret import DataFrame
from pret_joy import Stack
from pret_simple_dock import Layout, Panel

from metanno import DataRelation, DataWidgetFactory, RecipeTable

LABELS = {
    "tumeur": {"name": "Tumeur", "color": "rgb(255,200,206)", "shortcut": "t"},
    "tnm": {"name": "TNM", "color": "rgb(210,236,247)", "shortcut": "n"},
}


class Tables(TypedDict):
    patients: Annotated[DataFrame, RecipeTable("One row per patient", keywords=["patient", "person"])]
    notes: Annotated[DataFrame, RecipeTable("Notes of the patients", keywords=["note", "doc"])]
    entities: Annotated[DataFrame, RecipeTable("Entities found in the notes", keywords=["entit"])]


def main(store: Tables):
    """
    Decide on each patient while reading their notes and entities
    """
    # One relation per link between two tables. The notes table shows the
    # notes of every listed patient (working_set), the text the entities
    # of the selected note (selection, the default)
    factory = DataWidgetFactory(
        store,
        relations=[
            DataRelation("patients", "notes", on="patient_id", load="working_set"),
            DataRelation("notes", "entities", on="note_id"),
        ],
    )

    # The patient level decision
    patients = factory.create_table_widget(
        store_key="patients",
        primary_key="patient_id",
        fields=[
            {"key": "patient_id", "kind": "text"},
            {"key": "note_count", "kind": "text"},
            {"key": "eligible", "kind": "boolean", "editable": True},
        ],
        style={"height": "100%"},
    )

    # Notes of the listed patients, starting with those of the selected
    # patient only, All above the table shows them all
    notes = factory.create_table_widget(
        store_key="notes",
        primary_key="note_id",
        fields=[
            {"key": "note_id", "kind": "text"},
            {"key": "patient_id", "kind": "text"},
            {"key": "department", "kind": "text"},
        ],
        follow_parent_selection=True,
        parent_row_name="patient",
        style={"height": "100%"},
    )

    # Text of the selected note with its entities
    text, toolbar = factory.create_text_widget(
        store_text_key="notes",
        store_spans_key="entities",
        text_key="text",
        text_primary_key="note_id",
        spans_primary_key="entity_id",
        fields=[{"key": "label", "kind": "select", "editable": True, "options": list(LABELS)}],
        labels=LABELS,
    )

    # Patients above their notes on the left, the text on the right
    return Layout(
        Panel(patients, key="Patients"),
        Panel(notes, key="Notes"),
        Panel(Stack(toolbar, text, spacing=1), key="Text"),
        default_config={
            "kind": "row",
            "children": [
                {"kind": "column", "size": 35, "children": [{"tabs": ["Patients"]}, {"tabs": ["Notes"]}]},
                {"tabs": ["Text"], "size": 65},
            ],
        },
    )

As in Design your view, Tables names the tables main reads, so patient_review also opens from Recipes in the sidebar, on the tables of your sources.

A DataRelation links a parent table to a child table through a column they share, and decides which child rows the child widgets show:

`load`	The child table shows	Use it when
`selection`, the default	the rows of the selected parent row only	you work on one parent at a time, such as the entities of a note
`working_set`	the rows of every parent row listed, after its filters	you also compare parents, such as the notes of all eligible patients

With working_set, follow_parent_selection=True starts the child table on the selected parent row only, with All and Patient above it to switch to the whole working set and back. A table of the entities would also offer Note once a note is selected, and show the entities of the selected patient until then.

Open it

from metanno import view_opener

view_opener(store, view="patient_review")

Then go through the patients:

select patient 12: notes 4 and 5 are listed, select note 4 to read it
click All to see the notes of every patient, then type oncologie under department to keep the oncology notes only
tick eligible for the patients you would include

Small data as nested lists

For a few hundred records built in Python, a Store can also hold nested lists of dictionaries, without DataFrames, such as {"patients": [{"patient_id": 10, "notes": [...]}]}. Widgets then point to the nested level with a dotted path, store_key="patients.notes", and need no DataRelation. Prefer DataFrames for data coming from a database: they are what Prepare annotation produces, and they stay fast on large tables.

Annotate across levels

Add the patients

Write the view

Open it

Next