Nested and linked collections

This tutorial explains how to model multi-level data in Metanno and keep views synchronized. Two patterns are common:

Nested collections (for example stays[].notes[]).
Linked collections at different paths (for example stays[], notes[] and entities[]).

In both cases, DataWidgetFactory views are connected through shared keys and referenced with path strings such as stay.notes (or stays.notes when your top-level key is plural).

Widgets read from a store_key path:

stays points to the top-level collection.
stays.notes points to nested notes.
stays.evidences points to another nested collection.

When collections are not nested, synchronization still works if rows share linking keys:

stays.notes and stays.entities point to data nested in stays: this is "nesting"
stays.evidences and stays.notes are not nested, but they share some primary keys: this is "linking".
Every entity in stays.entities should have note_id so it can align with notes.

Show nested collections

Nested can be easily displayed by providing paths to the sub-collections.
Let's work with the following toy dataset, with entities and notes nested in stays.

def build_nested_data():
    return {
        "stays": [
            {
                "stay_id": "S1",
                "service": "oncology",
                "notes": [
                    {"note_id": "S1-N1", "note_text": "Mammographie le 12/06."},
                    {"note_id": "S1-N2", "note_text": "Consultation de suivi."},
                ],
                "entities": [
                    {
                        "id": "S1-N1-E1",
                        "note_id": "S1-N1",
                        "begin": 16,
                        "end": 21,
                        "text": "12/06",
                        "label": "date",
                        "mammography": True,
                    }
                ],
            },
            {
                "stay_id": "S2",
                "service": "cardiology",
                "notes": [
                    {"note_id": "S2-N1", "note_text": "Cardio chez M. Dupont."},
                ],
                "entities": [
                    {
                        "id": "S2-N1-E1",
                        "note_id": "S2-N1",
                        "begin": 0,
                        "end": 6,
                        "text": "Cardio",
                        "label": "procedure",
                    }
                ],
            },
        ]
    }

We'll first import the widget factory.

from metanno.recipes.data_widget_factory import DataWidgetFactory, infer_fields

factory = DataWidgetFactory(
    data=build_nested_data,
    # sync=True/path, enable sync to sync the user edits with the kernel/server
)

We'll create a table view to display stays, pointing at stays in our data collection

stays_view = factory.create_table_widget(
    store_key="stays",
    primary_key="stay_id",
    fields=infer_fields(factory.data["stays"], visible_keys=["stay_id", "service"]),
)

Then we'll add another table view to display notes. Note that these notes are "conditional" on stays: only the notes of the selected stay are shown

notes_view = factory.create_table_widget(
    store_key="stays.notes",
    primary_key="note_id",
    fields=infer_fields(
        [n for s in factory.data["stays"] for n in s["notes"]],
        visible_keys=["note_id", "note_text"],
        id_keys=["note_id"],
    ),
)

Then we'll add another table view to display entities. Note that these entities are "conditional" on stays: only the entities of the stay stay are shown.
They will be auto-linked to notes as they contain note_id field which is the primary key of the stays.notes view (see above).

entities_view = factory.create_table_widget(
    store_key="stays.entities",
    primary_key="id",
    fields=infer_fields(
        [e for s in factory.data["stays"] for e in s["entities"]],
        visible_keys=["id", "note_id", "label", "mammography"],
        id_keys=["id"],
        editable_keys=["label", "mammography"],
    ),
)

We could stop here if we only wanted table views, but we'll add another view to display entities annotated on texts.
Note how:

the location of the texts is the same as the one we provided earlier for the table view: store_text_key="stays.notes"
the location of the entities is the same as the one we provided earlier for the table view: store_spans_key="stays.entities"

note_text_view, ent_toolbar = factory.create_text_widget(
    store_text_key="stays.notes",
    store_spans_key="stays.entities",
    text_key="note_text",
    text_primary_key="note_id",
    spans_primary_key="id",
    fields=infer_fields(
        [e for s in factory.data["stays"] for e in s["entities"]],
        visible_keys=["label", "mammography"],
        editable_keys=["label", "mammography"],
    ),
    labels={
        "date": {"name": "Date", "color": "lightblue"},
    },
)

Finally, let's compose everything in a single view:

from pret.react import div
from pret_joy import Box, Divider, Stack
from pret_simple_dock import Layout, Panel

layout = div(
    Layout(
        Panel(stays_view, key="Stays"),
        Panel(notes_view, key="Notes"),
        Panel(entities_view, key="Entities"),
        Panel(Stack(Box(ent_toolbar, sx={"m": 1}), Divider(), note_text_view), key="Note Text"),
        default_config={
            "kind": "row",
            "children": [
                {"kind": "column", "children": ["Stays", "Notes", "Entities"], "size": 50},
                {"tabs": ["Note Text"], "size": 50},
            ],
        },
    ),
    style={
        "background": "var(--joy-palette-background-level2, #f0f0f0)",
        "width": "100%",
        "height": "100%",
        "minHeight": "500px",
        "--sd-background-color": "transparent",
    },
)

And display it:

layout

Linked collections stored separately

If you prefer separate top-level arrays, keep explicit foreign keys.

Note that this is the same data, just viewed as separate collections !

def build_split_data():
    return {
        "stays": [
            {"stay_id": "S1", "service": "oncology"},
            {"stay_id": "S2", "service": "cardiology"},
        ],
        "notes": [
            {"stay_id": "S1", "note_id": "S1-N1", "note_text": "Mammographie le 12/06."},
            {"stay_id": "S1", "note_id": "S1-N2", "note_text": "Consultation de suivi."},
            {"stay_id": "S2", "note_id": "S2-N1", "note_text": "Cardio chez M. Dupont."},
        ],
        "entities": [
            {
                "stay_id": "S1",
                "id": "S1-N1-E1",
                "note_id": "S1-N1",
                "begin": 16,
                "end": 21,
                "text": "12/06",
                "label": "date",
                "mammography": True,
            },
            {
                "stay_id": "S2",
                "id": "S2-N1-E1",
                "note_id": "S2-N1",
                "begin": 0,
                "end": 6,
                "text": "Cardio",
                "label": "procedure",
            },
        ],
    }

Let's create a compose the views as we did before. The trick is now to refer to collections using their new path, and to add foreign keys to the tables: whenever a user clicks a view, Metanno will automatically decide which view's filters should be changed to reflect the new selection on dependent views.

The code below also demonstrate how fields can be defined manually instead of using infer_fields.

from pret.react import div
from pret_joy import Box, Divider, Stack
from pret_simple_dock import Layout, Panel

from metanno.recipes.data_widget_factory import DataWidgetFactory, infer_fields

split_factory = DataWidgetFactory(
    data=build_split_data,
    # sync=True/path, enable sync to sync the user edits with the kernel/server
)

split_stays_view = split_factory.create_table_widget(
    store_key="stays",
    primary_key="stay_id",
    fields=infer_fields(split_factory.data["stays"], visible_keys=["stay_id", "service"]),
)

# fmt: off
split_notes_view = split_factory.create_table_widget(
    store_key="notes",
    primary_key="note_id",
    # or we could use infer_fields
    fields=[
        {"key": "note_id", "name": "note_id", "kind": "hyperlink", "editable": False, "filterable": True, "options": None},
        {"key": "stay_id", "name": "stay_id", "kind": "text", "editable": False, "filterable": True, "options": None},
        {"key": "note_text", "name": "note_text", "kind": "text", "editable": False, "filterable": True, "options": None},
    ],
)
# fmt: on

# fmt: off
split_entities_view = split_factory.create_table_widget(
    store_key="entities",
    primary_key="id",
    # or we could use infer_fields
    fields=[
        {"key": "id", "name": "id", "kind": "hyperlink", "editable": False, "filterable": True, "options": None},
        {"key": "stay_id", "name": "stay_id", "kind": "text", "editable": False, "filterable": True, "options": None},
        {"key": "note_id", "name": "note_id", "kind": "text", "editable": False, "filterable": True, "options": None},
        {"key": "label", "name": "label", "kind": "text", "editable": True, "filterable": True, "options": ["date", "procedure"]},
        {"key": "mammography", "name": "mammo", "kind": "boolean", "editable": True, "filterable": True, "options": None},
    ],
)
# fmt: on

# fmt: off
split_note_text_view, split_ent_toolbar = split_factory.create_text_widget(
    store_text_key="notes",
    store_spans_key="entities",
    text_key="note_text",
    text_primary_key="note_id",
    spans_primary_key="id",
    fields=[
        {"key": "label", "name": "label", "kind": "text", "editable": True, "filterable": True, "options": ["date", "procedure"]},
        {"key": "mammography", "name": "mammo", "kind": "boolean", "editable": True, "filterable": True, "options": None},
    ],
    labels={
        "date": {"name": "Date", "color": "lightblue"},
    },
)
# fmt: on

split_layout = div(
    Layout(
        Panel(split_stays_view, key="Stays"),
        Panel(split_notes_view, key="Notes"),
        Panel(split_entities_view, key="Entities"),
        Panel(
            Stack(Box(split_ent_toolbar, sx={"m": 1}), Divider(), split_note_text_view),
            key="Note Text",
        ),
        default_config={
            "kind": "row",
            "children": [
                {"kind": "column", "children": ["Stays", "Notes", "Entities"], "size": 50},
                {"tabs": ["Note Text"], "size": 50},
            ],
        },
    ),
    style={
        "background": "var(--joy-palette-background-level2, #f0f0f0)",
        "width": "100%",
        "height": "100%",
        "minHeight": "500px",
        "--sd-background-color": "transparent",
    },
)

And display it:

split_layout

Note how, before anything is selected, all notes and all entities are displayed in the table views.