Pre-annotate

Finding every tumor mention by hand is slow, while checking a mention that is already highlighted is quick. In this tutorial, you run an EDS-NLP pipeline on the notes you picked, and save the entities it finds, tumors and TNM stages, next to the notes, ready to be reviewed.

Before you start

Follow Pick documents first: this tutorial adds entities to its oncology.pret file. Install EDS-NLP in your notebook environment:

pip install edsnlp

Build the pipeline

The pipeline splits the notes into sentences, matches a few tumor terms, detects TNM stages, then flags the entities that are negated, such as the metastasis in "pas de métastase":

import edsnlp
import edsnlp.pipes as eds

nlp = edsnlp.blank("eds")
nlp.add_pipe(eds.sentences())
nlp.add_pipe(eds.normalizer())
nlp.add_pipe(eds.matcher(terms={"tumeur": ["cancer", "tumeur", "métastase"]}, attr="NORM"))
nlp.add_pipe(eds.tnm())
nlp.add_pipe(eds.negation())
nlp

Pipeline(lang=eds, pipes={
  "sentences": eds.sentences,
  "normalizer": eds.normalizer,
  "matcher": eds.matcher,
  "tnm": eds.tnm,
  "negation": eds.negation
})

Try it on a sentence before running it on your notes:

doc = nlp("Tumeur cT3N1M0, pas de métastase.")
[(ent.text, ent.label_, ent._.negation) for ent in doc.ents]

[('Tumeur', 'tumeur', False), ('cT3N1M0', 'tnm', False), ('métastase', 'tumeur', True)]

Pre-annotate the notes

Open the annotation file and read its notes. EDS-NLP then streams them through the pipeline and returns one row per entity, which you save in a new entities table of the file:

from pret import DataFrame, load_store

store = load_store("~/metanno-home/annotations/oncology.pret")
notes = store["notes"].materialize().to_pandas()

entities = (
    edsnlp.data.from_pandas(
        notes.rename(columns={"text": "note_text"}),
        converter="omop",
        doc_attributes=["note_id"],
        nlp=nlp,
    )
    .map_pipeline(nlp)
    .to_pandas(converter="ents", span_attributes={"negation": "negated"})
    .rename(columns={"start": "begin", "lexical_variant": "text"})
    .drop(columns="span_type")
    .assign(
        entity_id=lambda df: df["note_id"].astype(str) + "-" + df["begin"].astype(str),
        checked=False,
    )
)
store["entities"] = DataFrame(entities)
store["entities"].materialize().to_pandas()

	note_id	begin	end	label	text	negated	entity_id	checked
0	1	0	6	tumeur	Cancer	False	1-0	False
1	1	22	29	tnm	pT2N0M0	False	1-22	False
2	4	0	6	tumeur	Tumeur	False	4-0	False
3	4	7	14	tnm	cT3N1M0	False	4-7	False
4	4	23	32	tumeur	métastase	True	4-23	False

from_pandas reads the notes, keeping their note_id, map_pipeline runs the pipeline on them, by batches for large corpora, and to_pandas returns their entities, with negated from the negation step. Each row points to its note with note_id, and to its characters with begin, included, and end, excluded. checked is the column annotators will tick once they reviewed an entity.

Saved at once

Like any change to a saved Store, the new table is written to oncology.pret as soon as you add it, with no save step. It is also an editable copy: changing a row in Metanno never reruns the pipeline.

Pre-annotate

Build the pipeline

Pre-annotate the notes

Next