[{"data":1,"prerenderedAt":938},["ShallowReactive",2],{"writing-rag-retrieval-evals":3},{"id":4,"title":5,"body":6,"category":926,"date":927,"description":928,"extension":929,"i18nKey":930,"meta":931,"navigation":932,"path":933,"readingTime":934,"seo":935,"stem":936,"__hash__":937},"writing\u002Fwriting\u002Frag-retrieval-evals.md","Evaluating RAG retrieval quality",{"type":7,"value":8,"toc":917},"minimark",[9,13,21,26,29,45,53,57,64,85,88,244,251,255,258,268,274,710,725,729,732,746,749,753,756,847,854,858,861,887,894,898,905,913],[10,11,12],"p",{},"Every RAG pipeline demos well. You ask it three questions you already know it can answer, the answers look great, everyone ships. Then a user asks something phrased slightly differently, retrieval returns the wrong chunks, and the model — confident as ever — writes a beautiful answer grounded in the wrong context.",[10,14,15,16,20],{},"The failure wasn't generation. In my experience, ",[17,18,19],"strong",{},"when a RAG system is wrong, retrieval is the culprit far more often than the model"," — and retrieval is also the part almost nobody measures. This is the lightweight eval setup I use on client projects: small enough to build in a day, honest enough to catch regressions.",[22,23,25],"h2",{"id":24},"separate-the-two-questions","Separate the two questions",[10,27,28],{},"A RAG answer can fail in two independent places, and mixing them up makes debugging miserable:",[30,31,32,39],"ol",{},[33,34,35,38],"li",{},[17,36,37],{},"Did we retrieve the right context?"," (retrieval quality)",[33,40,41,44],{},[17,42,43],{},"Did the model answer faithfully from that context?"," (generation quality)",[10,46,47,48,52],{},"Evaluate them separately. If retrieval recall is bad, no prompt engineering will save you. If retrieval is fine and answers are still wrong, ",[49,50,51],"em",{},"then"," it's a prompting or model problem. Most teams jump straight to end-to-end \"does the answer look good\" checks and can never tell which knob to turn.",[22,54,56],{"id":55},"build-a-gold-set-small-is-fine","Build a gold set — small is fine",[10,58,59,60,63],{},"The eval set that actually gets maintained is a small one. Mine are usually ",[17,61,62],{},"30–60 questions",", built from three sources:",[65,66,67,73,79],"ul",{},[33,68,69,72],{},[17,70,71],{},"Real user questions"," from logs, the moment you have any. These are gold — users phrase things in ways you never will.",[33,74,75,78],{},[17,76,77],{},"Questions written from the docs"," — for each important document, write 2–3 questions it should answer.",[33,80,81,84],{},[17,82,83],{},"Adversarial phrasings"," — the same question asked colloquially, with typos, or with vocabulary that doesn't appear in the source (\"¿cuánto sale?\" vs \"pricing tiers\").",[10,86,87],{},"For each question, record which chunks (or at least which documents) contain the answer:",[89,90,95],"pre",{"className":91,"code":92,"language":93,"meta":94,"style":94},"language-ts shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","\u002F\u002F evals\u002Fgold.ts\nexport const goldSet: GoldItem[] = [\n  {\n    id: 'refund-window',\n    question: 'How long do I have to request a refund?',\n    relevantDocs: ['policies\u002Frefunds.md'],\n    answerContains: ['30 days']\n  }\n  \u002F\u002F ...40 more\n]\n","ts","",[96,97,98,107,139,145,167,184,207,227,233,239],"code",{"__ignoreMap":94},[99,100,103],"span",{"class":101,"line":102},"line",1,[99,104,106],{"class":105},"sHwdD","\u002F\u002F evals\u002Fgold.ts\n",[99,108,110,114,118,122,126,130,133,136],{"class":101,"line":109},2,[99,111,113],{"class":112},"s7zQu","export",[99,115,117],{"class":116},"spNyl"," const",[99,119,121],{"class":120},"sTEyZ"," goldSet",[99,123,125],{"class":124},"sMK4o",":",[99,127,129],{"class":128},"sBMFI"," GoldItem",[99,131,132],{"class":120},"[] ",[99,134,135],{"class":124},"=",[99,137,138],{"class":120}," [\n",[99,140,142],{"class":101,"line":141},3,[99,143,144],{"class":124},"  {\n",[99,146,148,152,154,157,161,164],{"class":101,"line":147},4,[99,149,151],{"class":150},"swJcz","    id",[99,153,125],{"class":124},[99,155,156],{"class":124}," '",[99,158,160],{"class":159},"sfazB","refund-window",[99,162,163],{"class":124},"'",[99,165,166],{"class":124},",\n",[99,168,170,173,175,177,180,182],{"class":101,"line":169},5,[99,171,172],{"class":150},"    question",[99,174,125],{"class":124},[99,176,156],{"class":124},[99,178,179],{"class":159},"How long do I have to request a refund?",[99,181,163],{"class":124},[99,183,166],{"class":124},[99,185,187,190,192,195,197,200,202,205],{"class":101,"line":186},6,[99,188,189],{"class":150},"    relevantDocs",[99,191,125],{"class":124},[99,193,194],{"class":120}," [",[99,196,163],{"class":124},[99,198,199],{"class":159},"policies\u002Frefunds.md",[99,201,163],{"class":124},[99,203,204],{"class":120},"]",[99,206,166],{"class":124},[99,208,210,213,215,217,219,222,224],{"class":101,"line":209},7,[99,211,212],{"class":150},"    answerContains",[99,214,125],{"class":124},[99,216,194],{"class":120},[99,218,163],{"class":124},[99,220,221],{"class":159},"30 days",[99,223,163],{"class":124},[99,225,226],{"class":120},"]\n",[99,228,230],{"class":101,"line":229},8,[99,231,232],{"class":124},"  }\n",[99,234,236],{"class":101,"line":235},9,[99,237,238],{"class":105},"  \u002F\u002F ...40 more\n",[99,240,242],{"class":101,"line":241},10,[99,243,226],{"class":120},[10,245,246,247,250],{},"That ",[96,248,249],{},"answerContains"," field is deliberately dumb — a substring the correct answer must include. Dumb checks are underrated: they're free, deterministic, and catch a surprising share of regressions.",[22,252,254],{"id":253},"the-two-retrieval-metrics-worth-computing","The two retrieval metrics worth computing",[10,256,257],{},"You don't need an IR degree. Two numbers tell you nearly everything:",[10,259,260,263,264,267],{},[17,261,262],{},"Recall@k"," — for what fraction of questions does the top-k contain at least one relevant chunk? This is your ceiling: if recall@5 is 70%, then 30% of questions ",[49,265,266],{},"cannot"," be answered correctly no matter how good the model is.",[10,269,270,273],{},[17,271,272],{},"MRR (mean reciprocal rank)"," — how high does the first relevant chunk rank? Two systems with identical recall@5 feel very different when one puts the answer at rank 1 and the other at rank 5, because rank affects what the model attends to — and what you can afford to include in a tight context.",[89,275,277],{"className":91,"code":276,"language":93,"meta":94,"style":94},"const evalRetrieval = async (gold: GoldItem[], k = 5) => {\n  const rows = []\n  for (const item of gold) {\n    const hits = await retrieve(item.question, k)\n    const rank = hits.findIndex(h => item.relevantDocs.includes(h.doc)) + 1\n    rows.push({ id: item.id, hit: rank > 0, rr: rank > 0 ? 1 \u002F rank : 0 })\n  }\n  return {\n    recallAtK: rows.filter(r => r.hit).length \u002F rows.length,\n    mrr: rows.reduce((s, r) => s + r.rr, 0) \u002F rows.length,\n    misses: rows.filter(r => !r.hit).map(r => r.id) \u002F\u002F ← the actionable part\n  }\n}\n",[96,278,279,328,341,365,401,454,530,534,541,587,646,699,704],{"__ignoreMap":94},[99,280,281,284,287,289,292,295,299,301,303,306,309,312,315,319,322,325],{"class":101,"line":102},[99,282,283],{"class":116},"const",[99,285,286],{"class":120}," evalRetrieval ",[99,288,135],{"class":124},[99,290,291],{"class":116}," async",[99,293,294],{"class":124}," (",[99,296,298],{"class":297},"sHdIc","gold",[99,300,125],{"class":124},[99,302,129],{"class":128},[99,304,305],{"class":120},"[]",[99,307,308],{"class":124},",",[99,310,311],{"class":297}," k",[99,313,314],{"class":124}," =",[99,316,318],{"class":317},"sbssI"," 5",[99,320,321],{"class":124},")",[99,323,324],{"class":116}," =>",[99,326,327],{"class":124}," {\n",[99,329,330,333,336,338],{"class":101,"line":109},[99,331,332],{"class":116},"  const",[99,334,335],{"class":120}," rows",[99,337,314],{"class":124},[99,339,340],{"class":150}," []\n",[99,342,343,346,348,350,353,356,359,362],{"class":101,"line":141},[99,344,345],{"class":112},"  for",[99,347,294],{"class":150},[99,349,283],{"class":116},[99,351,352],{"class":120}," item",[99,354,355],{"class":124}," of",[99,357,358],{"class":120}," gold",[99,360,361],{"class":150},") ",[99,363,364],{"class":124},"{\n",[99,366,367,370,373,375,378,382,385,388,391,394,396,398],{"class":101,"line":147},[99,368,369],{"class":116},"    const",[99,371,372],{"class":120}," hits",[99,374,314],{"class":124},[99,376,377],{"class":112}," await",[99,379,381],{"class":380},"s2Zo4"," retrieve",[99,383,384],{"class":150},"(",[99,386,387],{"class":120},"item",[99,389,390],{"class":124},".",[99,392,393],{"class":120},"question",[99,395,308],{"class":124},[99,397,311],{"class":120},[99,399,400],{"class":150},")\n",[99,402,403,405,408,410,412,414,417,419,422,424,426,428,431,433,436,438,440,442,445,448,451],{"class":101,"line":169},[99,404,369],{"class":116},[99,406,407],{"class":120}," rank",[99,409,314],{"class":124},[99,411,372],{"class":120},[99,413,390],{"class":124},[99,415,416],{"class":380},"findIndex",[99,418,384],{"class":150},[99,420,421],{"class":297},"h",[99,423,324],{"class":116},[99,425,352],{"class":120},[99,427,390],{"class":124},[99,429,430],{"class":120},"relevantDocs",[99,432,390],{"class":124},[99,434,435],{"class":380},"includes",[99,437,384],{"class":150},[99,439,421],{"class":120},[99,441,390],{"class":124},[99,443,444],{"class":120},"doc",[99,446,447],{"class":150},")) ",[99,449,450],{"class":124},"+",[99,452,453],{"class":317}," 1\n",[99,455,456,459,461,464,466,469,472,474,476,478,481,483,486,488,490,493,496,498,501,503,505,507,509,512,515,518,520,523,525,528],{"class":101,"line":186},[99,457,458],{"class":120},"    rows",[99,460,390],{"class":124},[99,462,463],{"class":380},"push",[99,465,384],{"class":150},[99,467,468],{"class":124},"{",[99,470,471],{"class":150}," id",[99,473,125],{"class":124},[99,475,352],{"class":120},[99,477,390],{"class":124},[99,479,480],{"class":120},"id",[99,482,308],{"class":124},[99,484,485],{"class":150}," hit",[99,487,125],{"class":124},[99,489,407],{"class":120},[99,491,492],{"class":124}," >",[99,494,495],{"class":317}," 0",[99,497,308],{"class":124},[99,499,500],{"class":150}," rr",[99,502,125],{"class":124},[99,504,407],{"class":120},[99,506,492],{"class":124},[99,508,495],{"class":317},[99,510,511],{"class":124}," ?",[99,513,514],{"class":317}," 1",[99,516,517],{"class":124}," \u002F",[99,519,407],{"class":120},[99,521,522],{"class":124}," :",[99,524,495],{"class":317},[99,526,527],{"class":124}," }",[99,529,400],{"class":150},[99,531,532],{"class":101,"line":209},[99,533,232],{"class":124},[99,535,536,539],{"class":101,"line":229},[99,537,538],{"class":112},"  return",[99,540,327],{"class":124},[99,542,543,546,548,550,552,555,557,560,562,565,567,570,572,574,577,579,581,583,585],{"class":101,"line":235},[99,544,545],{"class":150},"    recallAtK",[99,547,125],{"class":124},[99,549,335],{"class":120},[99,551,390],{"class":124},[99,553,554],{"class":380},"filter",[99,556,384],{"class":150},[99,558,559],{"class":297},"r",[99,561,324],{"class":116},[99,563,564],{"class":120}," r",[99,566,390],{"class":124},[99,568,569],{"class":120},"hit",[99,571,321],{"class":150},[99,573,390],{"class":124},[99,575,576],{"class":120},"length",[99,578,517],{"class":124},[99,580,335],{"class":120},[99,582,390],{"class":124},[99,584,576],{"class":120},[99,586,166],{"class":124},[99,588,589,592,594,596,598,601,603,605,608,610,612,614,616,619,622,624,626,629,631,633,635,638,640,642,644],{"class":101,"line":241},[99,590,591],{"class":150},"    mrr",[99,593,125],{"class":124},[99,595,335],{"class":120},[99,597,390],{"class":124},[99,599,600],{"class":380},"reduce",[99,602,384],{"class":150},[99,604,384],{"class":124},[99,606,607],{"class":297},"s",[99,609,308],{"class":124},[99,611,564],{"class":297},[99,613,321],{"class":124},[99,615,324],{"class":116},[99,617,618],{"class":120}," s",[99,620,621],{"class":124}," +",[99,623,564],{"class":120},[99,625,390],{"class":124},[99,627,628],{"class":120},"rr",[99,630,308],{"class":124},[99,632,495],{"class":317},[99,634,361],{"class":150},[99,636,637],{"class":124},"\u002F",[99,639,335],{"class":120},[99,641,390],{"class":124},[99,643,576],{"class":120},[99,645,166],{"class":124},[99,647,649,652,654,656,658,660,662,664,666,669,671,673,675,677,679,682,684,686,688,690,692,694,696],{"class":101,"line":648},11,[99,650,651],{"class":150},"    misses",[99,653,125],{"class":124},[99,655,335],{"class":120},[99,657,390],{"class":124},[99,659,554],{"class":380},[99,661,384],{"class":150},[99,663,559],{"class":297},[99,665,324],{"class":116},[99,667,668],{"class":124}," !",[99,670,559],{"class":120},[99,672,390],{"class":124},[99,674,569],{"class":120},[99,676,321],{"class":150},[99,678,390],{"class":124},[99,680,681],{"class":380},"map",[99,683,384],{"class":150},[99,685,559],{"class":297},[99,687,324],{"class":116},[99,689,564],{"class":120},[99,691,390],{"class":124},[99,693,480],{"class":120},[99,695,361],{"class":150},[99,697,698],{"class":105},"\u002F\u002F ← the actionable part\n",[99,700,702],{"class":101,"line":701},12,[99,703,232],{"class":124},[99,705,707],{"class":101,"line":706},13,[99,708,709],{"class":124},"}\n",[10,711,712,713,716,717,720,721,724],{},"The ",[96,714,715],{},"misses"," list is the real product of the eval. A recall number tells you ",[49,718,719],{},"that"," something is wrong; the list of missed questions tells you ",[49,722,723],{},"what"," — and the misses almost always cluster (all the colloquial phrasings, all the questions about one badly-chunked doc).",[22,726,728],{"id":727},"judging-generation-without-fooling-yourself","Judging generation without fooling yourself",[10,730,731],{},"For the generation side I use an LLM judge, with two rules that keep it honest:",[65,733,734,740],{},[33,735,736,739],{},[17,737,738],{},"Judge faithfulness, not quality."," The question for the judge is narrow: \"Is every claim in this answer supported by the provided context — yes or no, and which claim isn't?\" Narrow questions get reliable judgments; \"rate this answer 1–10\" gets vibes.",[33,741,742,745],{},[17,743,744],{},"Calibrate it once."," Hand-label 20 answers yourself, run the judge on the same 20, and check agreement. If the judge disagrees with you more than ~15% of the time, tighten its prompt before trusting it at scale.",[10,747,748],{},"The judge runs with a stronger model than the pipeline uses, and its verdict lands next to the retrieval metrics — one row per gold question.",[22,750,752],{"id":751},"the-dashboard-is-a-table","The dashboard is a table",[10,754,755],{},"My \"eval dashboard\" is deliberately unglamorous: a run per row, a metric per column, stored as JSON and rendered in a simple Nuxt page.",[757,758,759,784],"table",{},[760,761,762],"thead",{},[763,764,765,769,772,775,778,781],"tr",{},[766,767,768],"th",{},"run",[766,770,771],{},"change",[766,773,774],{},"recall@5",[766,776,777],{},"MRR",[766,779,780],{},"faithful",[766,782,783],{},"dumb-checks",[785,786,787,808,828],"tbody",{},[763,788,789,793,796,799,802,805],{},[790,791,792],"td",{},"#14",[790,794,795],{},"baseline",[790,797,798],{},"0.72",[790,800,801],{},"0.58",[790,803,804],{},"0.88",[790,806,807],{},"34\u002F41",[763,809,810,813,816,819,822,825],{},[790,811,812],{},"#15",[790,814,815],{},"smaller chunks (400 tok)",[790,817,818],{},"0.83",[790,820,821],{},"0.66",[790,823,824],{},"0.90",[790,826,827],{},"37\u002F41",[763,829,830,833,836,838,841,844],{},[790,831,832],{},"#16",[790,834,835],{},"+ hybrid search",[790,837,824],{},[790,839,840],{},"0.74",[790,842,843],{},"0.91",[790,845,846],{},"39\u002F41",[10,848,849,850,853],{},"What matters is that ",[17,851,852],{},"every pipeline change gets a row before it merges",". Chunking tweak? Row. New embedding model? Row. Re-ranker? Row. The table turns \"I think this made it better\" into a diff you can read — and it catches the sneaky cases where a change helps one cluster of questions while quietly breaking another.",[22,855,857],{"id":856},"where-the-wins-actually-come-from","Where the wins actually come from",[10,859,860],{},"Across projects, the improvements that move these numbers, in rough order of return on effort:",[30,862,863,869,875,881],{},[33,864,865,868],{},[17,866,867],{},"Chunking."," Oversized chunks are the most common silent killer. Respect document structure (headings, sections) instead of fixed windows, and keep a small overlap.",[33,870,871,874],{},[17,872,873],{},"Hybrid search."," Vector search misses exact terms — SKUs, error codes, product names. Adding BM25 alongside embeddings and merging results is a reliable recall boost.",[33,876,877,880],{},[17,878,879],{},"Re-ranking."," A cross-encoder re-ranker over the top-20 fixes ordering, which shows up directly in MRR.",[33,882,883,886],{},[17,884,885],{},"Query rewriting."," Expanding the user's question with the model before retrieval helps colloquial phrasings find formal docs.",[10,888,889,890,893],{},"None of these are exotic. The eval is what tells you which one ",[49,891,892],{},"your"," system needs — instead of implementing all four and hoping.",[22,895,897],{"id":896},"takeaways","Takeaways",[10,899,900,901,904],{},"Split retrieval from generation and measure them separately. Keep the gold set small enough to maintain and seeded with real user phrasings. Track recall@k and MRR per change, read the misses instead of the averages, and keep the LLM judge on a narrow, calibrated question. A day of setup buys you the thing most RAG systems never have: the ability to know you broke something ",[49,902,903],{},"before"," your users find out.",[10,906,907,908,390],{},"If your team is shipping a RAG feature and flying blind on retrieval, ",[909,910,912],"a",{"href":911},"\u002F#contact","this is exactly the kind of setup I build",[914,915,916],"style",{},"html pre.shiki code .sHwdD, html code.shiki .sHwdD{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic}html pre.shiki code .s7zQu, html code.shiki .s7zQu{--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#89DDFF;--shiki-default-font-style:italic;--shiki-dark:#89DDFF;--shiki-dark-font-style:italic}html pre.shiki code .spNyl, html code.shiki .spNyl{--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA}html pre.shiki code .sTEyZ, html code.shiki .sTEyZ{--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8}html pre.shiki code .sMK4o, html code.shiki .sMK4o{--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF}html pre.shiki code .sBMFI, html code.shiki .sBMFI{--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B}html pre.shiki code .swJcz, html code.shiki .swJcz{--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178}html pre.shiki code .sfazB, html code.shiki .sfazB{--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sHdIc, html code.shiki .sHdIc{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#EEFFFF;--shiki-default-font-style:italic;--shiki-dark:#BABED8;--shiki-dark-font-style:italic}html pre.shiki code .sbssI, html code.shiki .sbssI{--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C}html pre.shiki code .s2Zo4, html code.shiki .s2Zo4{--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF}",{"title":94,"searchDepth":109,"depth":109,"links":918},[919,920,921,922,923,924,925],{"id":24,"depth":109,"text":25},{"id":55,"depth":109,"text":56},{"id":253,"depth":109,"text":254},{"id":727,"depth":109,"text":728},{"id":751,"depth":109,"text":752},{"id":856,"depth":109,"text":857},{"id":896,"depth":109,"text":897},"LLM Integration","2026-08-03","A practical RAG eval setup — a small gold set, recall@k and MRR, an LLM judge you can actually trust, and the dashboard that catches regressions before users do.","md","ragEvals",{},true,"\u002Fwriting\u002Frag-retrieval-evals","8 min read",{"title":5,"description":928},"writing\u002Frag-retrieval-evals","OAGCuh06m-oICzVYz7yMaw8ltBf8N-LQ1eDoF6HjEv8",1785786127680]