MIS 752 · Lab 5 · winnetka

Speak the Patient's Language: semantic search vs keyword search

A 20-sentence clinical corpus embedded with all-MiniLM-L6-v2, the full cosine similarity matrix, and a scored comparison of semantic search against a keyword baseline on a written gold standard. Runs locally on a free CPU: no API key, no PHI, $0.

The 30-second read

THE MODEL encoder : sentence-transformers/all-MiniLM-L6-v2 dimensions : 384 numbers per sentence corpus : 20 sentences, 4 topics, 2 voices embed time : 24 ms for the whole corpus API cost : $0.00 (runs locally, no key, no network after load) DOES IT CAPTURE MEANING? mean cosine within a topic : 0.235 mean cosine across topics : 0.129 separation : +0.106 RETRIEVAL vs THE GOLD STANDARD (12 queries, top-3) keyword (word overlap) precision 0.25 · recall 0.32 · MRR 0.50 semantic (embeddings) precision 0.42 · recall 0.54 · MRR 0.92 overall, semantic beats keyword on MRR by +0.42 THE FINDING - MRR SPLIT BY WHETHER THE PATIENT GUESSED OUR WORDS patient's own words keyword 0.00 · semantic 0.80 patient happened to use our vocabulary keyword 0.86 · semantic 1.00 EXAMPLE QUERY "chest hurts when i climb stairs" 1. car1 [cardiac/patient] cosine 0.787 2. car2 [cardiac/clinical] cosine 0.392 3. car4 [cardiac/patient] cosine 0.374

Do embeddings capture meaning?

The 20×20 cosine similarity matrix, rows and columns sorted by topic and boxed into the four topic blocks. Each label shows the sentence id and its voice (pat = patient, cli = clinical). The diagonal is blanked and the colour scale spans the off-diagonal range: every sentence scores 1.00 against itself, and leaving that in flattens everything else into one shade. A dark cell spanning both voices is the whole point of the week: same meaning, no shared words. A pale row inside a box is a sentence this model cannot place.

Meaning beats spelling only where spelling runs out

Scored against the written-down gold standard, not by eyeball. MRR asks how high the first right answer was, and it is the metric a patient on a phone actually feels: 1.0 means rank one. The two groups differ by whether the patient happened to guess the corpus's own vocabulary — a split computed from the text, not labelled by hand. In the left group keyword search is fine, because there are words to match. In the right group there are none and it collapses. Semantic search never needed the words to match in the first place.

Per-query detail

Where each method wins and loses, query by query. The keyword baseline is deliberately naive word overlap, so this is not the strongest possible keyword system — BM25 would do better, and a reranker better still. It is enough to show that the vocabulary gap, not the algorithm, is what defeats word matching.

query_type query method retrieved p_at_k recall_at_k rr first_rank patient happened to use our vocabulary chest hurts when i climb stairs semantic car1, car2, car4 0.666667 0.666667 1.0 1.0 patient happened to use our vocabulary chest hurts when i climb stairs keyword car1, car2 0.666667 0.666667 1.0 1.0 patient happened to use our vocabulary my refill is not ready at the pharmacy semantic med5, dia4, car5 0.333333 0.500000 1.0 1.0 patient happened to use our vocabulary my refill is not ready at the pharmacy keyword med5 0.333333 0.500000 1.0 1.0 patient happened to use our vocabulary is it safe to take ibuprofen with warfarin semantic med1, med2, med5 0.666667 1.000000 1.0 1.0 patient happened to use our vocabulary is it safe to take ibuprofen with warfarin keyword med1, dia1, med2 0.666667 1.000000 1.0 1.0 patient happened to use our vocabulary i feel low and cannot sleep semantic men1, dia3, dia5 0.333333 0.333333 1.0 1.0 patient happened to use our vocabulary i feel low and cannot sleep keyword men1, men3 0.333333 0.333333 1.0 1.0 patient's own words it aches to climb to the second floor semantic car1, dia5, men1 0.333333 0.333333 1.0 1.0 patient's own words it aches to climb to the second floor keyword 0.000000 0.000000 0.0 NaN patient's own words the reading on my meter jumped since supper semantic car5, dia5, car2 0.000000 0.000000 0.0 NaN patient's own words the reading on my meter jumped since supper keyword 0.000000 0.000000 0.0 NaN patient happened to use our vocabulary would the anti-inflammatory clash with what thins my blood semantic med1, med2, men3 0.666667 1.000000 1.0 1.0 patient happened to use our vocabulary would the anti-inflammatory clash with what thins my blood keyword car5, med1 0.333333 0.500000 0.5 2.0 patient happened to use our vocabulary i have been down and wide awake at 3am for weeks semantic men1, dia3, men2 0.666667 0.666667 1.0 1.0 patient happened to use our vocabulary i have been down and wide awake at 3am for weeks keyword med5, men1, men5 0.333333 0.333333 0.5 2.0 patient's own words i cannot pay for the name brand any more semantic med3, med5, car5 0.333333 0.500000 1.0 1.0 patient's own words i cannot pay for the name brand any more keyword 0.000000 0.000000 0.0 NaN patient happened to use our vocabulary my feet feel like pins and needles semantic dia5, car1, car5 0.333333 0.500000 1.0 1.0 patient happened to use our vocabulary my feet feel like pins and needles keyword dia5, men3 0.333333 0.500000 1.0 1.0 patient's own words the drug store says come back next week semantic med5, men5, dia4 0.333333 0.500000 1.0 1.0 patient's own words the drug store says come back next week keyword 0.000000 0.000000 0.0 NaN patient's own words i get winded carrying the groceries in semantic car4, car1, dia1 0.333333 0.500000 1.0 1.0 patient's own words i get winded carrying the groceries in keyword 0.000000 0.000000 0.0 NaN

How this was measured

Sentences were embedded with sentence-transformers/all-MiniLM-L6-v2 (384 dimensions, ungated, run locally on a free Colab CPU -- no API key and no network traffic after the model loads). Cosine similarity is computed directly as E @ E.T on L2-normalised rows, not via a library convenience method, so every number is reproducible from the saved embeddings.npy. Retrieval is scored against 12 queries carrying 29 written relevance judgements (GOLD in the notebook). The keyword baseline counts shared content words (stopwords dropped, length > 2). Metrics are precision@k, recall@k and mean reciprocal rank at k=3. This is a 20-sentence teaching corpus with a 12-query gold set -- a demonstration of method, not a clinical validation. The corpus is teaching text I wrote: no PHI, no real records, nothing to act on clinically.