[{"data":1,"prerenderedAt":3048},["ShallowReactive",2],{"navigation":3,"page-\u002Fchapters\u002Fretrieval":102,"surround-\u002Fchapters\u002Fretrieval":3043},[4,8],{"title":5,"path":6,"stem":7},"Home","\u002F","index",{"title":9,"path":10,"stem":11,"children":12,"page":101},"Chapters","\u002Fchapters","2.chapters",[13,17,21,25,29,33,37,41,45,49,53,57,61,65,69,73,77,81,85,89,93,97],{"title":14,"path":15,"stem":16},"Chapter 1. What an Agent Actually Is","\u002Fchapters\u002Fwhat-an-agent-actually-is","2.chapters\u002F01.what-an-agent-actually-is",{"title":18,"path":19,"stem":20},"Chapter 2. The Minimum Viable Loop","\u002Fchapters\u002Fminimum-viable-loop","2.chapters\u002F02.minimum-viable-loop",{"title":22,"path":23,"stem":24},"Chapter 3. Messages, Turns, and the Transcript","\u002Fchapters\u002Fmessages-turns-transcript","2.chapters\u002F03.messages-turns-transcript",{"title":26,"path":27,"stem":28},"Chapter 4. The Tool Protocol","\u002Fchapters\u002Ftool-protocol","2.chapters\u002F04.tool-protocol",{"title":30,"path":31,"stem":32},"Chapter 5. Streaming, Interruption, and Error Handling","\u002Fchapters\u002Fstreaming-interruption-errors","2.chapters\u002F05.streaming-interruption-errors",{"title":34,"path":35,"stem":36},"Chapter 6. Safe Tool Execution","\u002Fchapters\u002Fsafe-tool-execution","2.chapters\u002F06.safe-tool-execution",{"title":38,"path":39,"stem":40},"Chapter 7. The Context Window Is a Resource","\u002Fchapters\u002Fcontext-window-as-resource","2.chapters\u002F07.context-window-as-resource",{"title":42,"path":43,"stem":44},"Chapter 8. Compaction","\u002Fchapters\u002Fcompaction","2.chapters\u002F08.compaction",{"title":46,"path":47,"stem":48},"Chapter 9. External State: The Scratchpad","\u002Fchapters\u002Fscratchpad","2.chapters\u002F09.scratchpad",{"title":50,"path":51,"stem":52},"Chapter 10. Retrieval","\u002Fchapters\u002Fretrieval","2.chapters\u002F10.retrieval",{"title":54,"path":55,"stem":56},"Chapter 11. Designing Tools Models Can Actually Use","\u002Fchapters\u002Fdesigning-tools","2.chapters\u002F11.designing-tools",{"title":58,"path":59,"stem":60},"Chapter 12. The Tool Cliff and Dynamic Loading","\u002Fchapters\u002Ftool-cliff","2.chapters\u002F12.tool-cliff",{"title":62,"path":63,"stem":64},"Chapter 13. MCP: Tools From the Outside World","\u002Fchapters\u002Fmcp","2.chapters\u002F13.mcp",{"title":66,"path":67,"stem":68},"Chapter 14. Sandboxing and Permissions","\u002Fchapters\u002Fsandboxing-permissions","2.chapters\u002F14.sandboxing-permissions",{"title":70,"path":71,"stem":72},"Chapter 15. Sub-agents","\u002Fchapters\u002Fsub-agents","2.chapters\u002F15.sub-agents",{"title":74,"path":75,"stem":76},"Chapter 16. Structured Plans and Verified Completion","\u002Fchapters\u002Fplans-verified-completion","2.chapters\u002F16.plans-verified-completion",{"title":78,"path":79,"stem":80},"Chapter 17. Parallelism and Shared State","\u002Fchapters\u002Fparallelism-shared-state","2.chapters\u002F17.parallelism-shared-state",{"title":82,"path":83,"stem":84},"Chapter 18. Observability","\u002Fchapters\u002Fobservability","2.chapters\u002F18.observability",{"title":86,"path":87,"stem":88},"Chapter 19. Evals","\u002Fchapters\u002Fevals","2.chapters\u002F19.evals",{"title":90,"path":91,"stem":92},"Chapter 20. Cost Control","\u002Fchapters\u002Fcost-control","2.chapters\u002F20.cost-control",{"title":94,"path":95,"stem":96},"Chapter 21. Resumability and Durable State","\u002Fchapters\u002Fresumability","2.chapters\u002F21.resumability",{"title":98,"path":99,"stem":100},"Chapter 22. What Transfers, Where to Go","\u002Fchapters\u002Fwhat-transfers","2.chapters\u002F22.what-transfers",false,{"id":103,"title":50,"body":104,"description":117,"extension":3038,"meta":3039,"navigation":3040,"path":51,"seo":3041,"stem":52,"__hash__":3042},"content\u002F2.chapters\u002F10.retrieval.md",{"type":105,"value":106,"toc":3027},"minimark",[107,111,118,125,128,151,154,233,236,241,244,247,253,259,265,268,270,274,277,309,1490,1493,1499,1505,1518,1535,1537,1541,2206,2212,2219,2221,2225,2232,2239,2250,2256,2259,2261,2265,2836,2843,2845,2849,2852,2861,2867,2876,2878,2882,2892,2895,2916,2919,2921,2925,2969,2973,3000,3002,3023],[108,109,50],"h1",{"id":110},"chapter-10-retrieval",[112,113,114],"p",{},[115,116,117],"em",{},"Previously: the scratchpad gave the agent durable state for what it produces. What it doesn't cover is what the agent needs to read from but didn't write — a codebase it's exploring, documentation, a knowledge base that's larger than the context window could hold even empty.",[112,119,120,121,124],{},"Retrieval is how an agent works over a corpus too large to fit in context. The idea is not new: Lewis et al.'s 2020 NeurIPS paper \"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks\" established the RAG pattern — retrieve relevant passages, inject them into the prompt, generate an answer conditioned on both — and every production retrieval system in LLM-land is a descendant of that work. What most implementations miss is a subtle point that post-2020 research made inescapable: retrieval is not just about getting the right content, it's about getting the right content ",[115,122,123],{},"in the right place",". The lost-in-the-middle effect Liu et al. documented in 2023 is real and quantified. A relevant document shoved into the middle of a 100K-token context gets less attention than a less-relevant one placed at the end. You can have perfect recall and terrible answers.",[112,126,127],{},"This chapter builds a small retrieval system for the harness with three specific disciplines:",[129,130,131,139,145],"ol",{},[132,133,134,138],"li",{},[135,136,137],"strong",{},"Agent-driven, not passive."," The agent chooses when to retrieve, via a tool, rather than retrieval happening every turn.",[132,140,141,144],{},[135,142,143],{},"Edge-placed."," Retrieved content goes at the end of the context, right before the user's current turn — the position with the highest attention weight.",[132,146,147,150],{},[135,148,149],{},"Explicit cost."," Every retrieval declares what it will add to the context so the agent can make informed choices.",[112,152,153],{},"By the end, the agent can search a directory of documents, get relevant chunks with scores, and be trusted not to drown itself.",[155,156,160,161,160,226],"figure",{"className":157},[158,159],"not-prose","my-8","\n  ",[162,163,168,169,168,200,168,215,160],"div",{"className":164},[165,166,167],"flex","flex-col","gap-3","\n    ",[162,170,178,179,178,190,178,196,168],{"className":171,"style":177},[165,172,173,174,175,176],"items-stretch","border","border-default","rounded-md","overflow-hidden","height:56px;","\n      ",[162,180,189],{"className":181,"style":188},[182,165,183,184,185,186,187],"flex-1","items-center","justify-center","text-xs","font-semibold","text-default","background:color-mix(in oklab, currentColor 32%, transparent);","start · ~90%",[162,191,195],{"className":192,"style":194},[182,165,183,184,185,193],"text-muted","background:color-mix(in oklab, currentColor 12%, transparent);","middle · ~55%",[162,197,199],{"className":198,"style":188},[182,165,183,184,185,186,187],"end · ~90%",[162,201,178,205,178,209,178,212,168],{"className":202},[165,203,185,193,204],"justify-between","px-1",[206,207,208],"span",{},"system prompt",[206,210,211],{},"history",[206,213,214],{},"current turn",[162,216,225],{"className":217},[218,219,220,221,222,185,223,186,224],"border-2","border-primary","rounded-lg","py-2","px-4","text-primary","text-center","place critical retrieval results at the edges — end preferred",[227,228,232],"figcaption",{"className":229},[185,193,230,224,231],"mt-3","italic","Lost-in-the-middle: attention retention dips hardest in the centre of long contexts.",[234,235],"hr",{},[237,238,240],"h2",{"id":239},"_101-naive-rag-and-whats-wrong-with-it","10.1 Naive RAG and What's Wrong With It",[112,242,243],{},"The classic pattern: on every user turn, embed the user's message, search a vector store, take top-K results, prepend them to the prompt. Many tutorials stop there.",[112,245,246],{},"Three problems with the naive version.",[112,248,249,252],{},[135,250,251],{},"It retrieves whether or not retrieval is needed."," A simple arithmetic prompt triggers a vector search; the top-K results are irrelevant; the model now has irrelevant content in its context, which — per context rot — degrades rather than improves its output.",[112,254,255,258],{},[135,256,257],{},"Placement is wrong."," Prepending to the system prompt is the worst spot: middle of the context as soon as history accumulates. The U-curve bites.",[112,260,261,264],{},[135,262,263],{},"The agent can't see the retrieval."," If the search was bad, the model doesn't know; it just knows its context contains weird stuff. An agent-driven retrieval tool means the agent decides, sees the results, and can re-query with a better term.",[112,266,267],{},"We'll do agent-driven retrieval with edge placement, backed by the cheapest index that can possibly work.",[234,269],{},[237,271,273],{"id":272},"_102-the-index","10.2 The Index",[112,275,276],{},"For the book's scenarios, we don't need a vector database. A BM25 index over a directory of text documents is accurate enough, fast enough, and — importantly — runs without a network call or an embedding model. The BM25 scoring function itself dates back to Robertson and Zaragoza's 2009 survey \"The Probabilistic Relevance Framework: BM25 and Beyond\" and the decades of information-retrieval work it consolidated; it is not a stopgap or a simplification, it is the algorithm classical IR converged on for keyword relevance and the one against which every embedding-based retriever is still benchmarked. Chapter 22 discusses when you'd upgrade to embeddings or hybrid retrieval; most harnesses under 10K documents are fine without.",[278,279,284],"pre",{"className":280,"code":281,"language":282,"meta":283,"style":283},"language-bash shiki shiki-themes material-theme-lighter github-light github-dark","uv add 'rank-bm25>=0.2.2'\n","bash","",[285,286,287],"code",{"__ignoreMap":283},[206,288,291,295,299,303,306],{"class":289,"line":290},"line",1,[206,292,294],{"class":293},"sbgvK","uv",[206,296,298],{"class":297},"s_sjI"," add",[206,300,302],{"class":301},"sjJ54"," '",[206,304,305],{"class":297},"rank-bm25>=0.2.2",[206,307,308],{"class":301},"'\n",[278,310,314],{"className":311,"code":312,"language":313,"meta":283,"style":283},"language-python shiki shiki-themes material-theme-lighter github-light github-dark","# src\u002Fharness\u002Fretrieval\u002Findex.py\nfrom __future__ import annotations\n\nimport re\nfrom dataclasses import dataclass\nfrom pathlib import Path\n\nfrom rank_bm25 import BM25Okapi\n\n\ndef _tokenize(text: str) -> list[str]:\n    return re.findall(r\"\\w+\", text.lower())\n\n\n@dataclass\nclass Chunk:\n    doc_id: str\n    chunk_id: int\n    text: str\n\n\n@dataclass\nclass SearchHit:\n    chunk: Chunk\n    score: float\n\n\nclass DocumentIndex:\n    \"\"\"A BM25 index over text files in a directory.\n\n    Chunks files into ~500-token pieces with 50-token overlap.\n    \"\"\"\n\n    def __init__(self, root: Path | str, chunk_tokens: int = 500,\n                 overlap: int = 50) -> None:\n        self.root = Path(root)\n        self.chunks: list[Chunk] = []\n        self._build(chunk_tokens, overlap)\n        tokenized = [_tokenize(c.text) for c in self.chunks]\n        self._bm25 = BM25Okapi(tokenized)\n\n    def _build(self, chunk_tokens: int, overlap: int) -> None:\n        for path in sorted(self.root.rglob(\"*\")):\n            if not path.is_file():\n                continue\n            try:\n                text = path.read_text(encoding=\"utf-8\")\n            except (UnicodeDecodeError, PermissionError):\n                continue\n            words = text.split()\n            for i, start in enumerate(range(0, len(words),\n                                             chunk_tokens - overlap)):\n                chunk_text = \" \".join(words[start:start + chunk_tokens])\n                if chunk_text.strip():\n                    self.chunks.append(Chunk(\n                        doc_id=str(path.relative_to(self.root)),\n                        chunk_id=i,\n                        text=chunk_text,\n                    ))\n\n    def search(self, query: str, k: int = 5) -> list[SearchHit]:\n        tokenized_query = _tokenize(query)\n        scores = self._bm25.get_scores(tokenized_query)\n        indexed = sorted(enumerate(scores), key=lambda x: -x[1])[:k]\n        return [SearchHit(chunk=self.chunks[i], score=s)\n                for i, s in indexed if s > 0]\n","python",[285,315,316,322,340,347,356,369,382,387,400,405,410,454,502,507,512,522,534,545,556,566,571,576,583,593,604,615,620,625,635,646,651,657,663,668,719,744,768,795,817,862,884,889,925,964,984,990,998,1030,1050,1055,1073,1115,1128,1167,1183,1205,1236,1249,1262,1268,1273,1321,1338,1364,1419,1459],{"__ignoreMap":283},[206,317,318],{"class":289,"line":290},[206,319,321],{"class":320},"sutJx","# src\u002Fharness\u002Fretrieval\u002Findex.py\n",[206,323,325,329,333,336],{"class":289,"line":324},2,[206,326,328],{"class":327},"sVHd0","from",[206,330,332],{"class":331},"s_hVV"," __future__",[206,334,335],{"class":327}," import",[206,337,339],{"class":338},"su5hD"," annotations\n",[206,341,343],{"class":289,"line":342},3,[206,344,346],{"emptyLinePlaceholder":345},true,"\n",[206,348,350,353],{"class":289,"line":349},4,[206,351,352],{"class":327},"import",[206,354,355],{"class":338}," re\n",[206,357,359,361,364,366],{"class":289,"line":358},5,[206,360,328],{"class":327},[206,362,363],{"class":338}," dataclasses ",[206,365,352],{"class":327},[206,367,368],{"class":338}," dataclass\n",[206,370,372,374,377,379],{"class":289,"line":371},6,[206,373,328],{"class":327},[206,375,376],{"class":338}," pathlib ",[206,378,352],{"class":327},[206,380,381],{"class":338}," Path\n",[206,383,385],{"class":289,"line":384},7,[206,386,346],{"emptyLinePlaceholder":345},[206,388,390,392,395,397],{"class":289,"line":389},8,[206,391,328],{"class":327},[206,393,394],{"class":338}," rank_bm25 ",[206,396,352],{"class":327},[206,398,399],{"class":338}," BM25Okapi\n",[206,401,403],{"class":289,"line":402},9,[206,404,346],{"emptyLinePlaceholder":345},[206,406,408],{"class":289,"line":407},10,[206,409,346],{"emptyLinePlaceholder":345},[206,411,413,417,421,425,429,432,436,439,442,445,448,451],{"class":289,"line":412},11,[206,414,416],{"class":415},"sbsja","def",[206,418,420],{"class":419},"sGLFI"," _tokenize",[206,422,424],{"class":423},"sP7_E","(",[206,426,428],{"class":427},"sFwrP","text",[206,430,431],{"class":423},":",[206,433,435],{"class":434},"sZMiF"," str",[206,437,438],{"class":423},")",[206,440,441],{"class":423}," ->",[206,443,444],{"class":338}," list",[206,446,447],{"class":423},"[",[206,449,450],{"class":434},"str",[206,452,453],{"class":423},"]:\n",[206,455,457,460,463,466,470,472,475,478,482,486,488,491,494,496,499],{"class":289,"line":456},12,[206,458,459],{"class":327},"    return",[206,461,462],{"class":338}," re",[206,464,465],{"class":423},".",[206,467,469],{"class":468},"slqww","findall",[206,471,424],{"class":423},[206,473,474],{"class":415},"r",[206,476,477],{"class":301},"\"",[206,479,481],{"class":480},"stzsN","\\w",[206,483,485],{"class":484},"smGrS","+",[206,487,477],{"class":301},[206,489,490],{"class":423},",",[206,492,493],{"class":468}," text",[206,495,465],{"class":423},[206,497,498],{"class":468},"lower",[206,500,501],{"class":423},"())\n",[206,503,505],{"class":289,"line":504},13,[206,506,346],{"emptyLinePlaceholder":345},[206,508,510],{"class":289,"line":509},14,[206,511,346],{"emptyLinePlaceholder":345},[206,513,515,519],{"class":289,"line":514},15,[206,516,518],{"class":517},"stp6e","@",[206,520,521],{"class":419},"dataclass\n",[206,523,525,528,531],{"class":289,"line":524},16,[206,526,527],{"class":415},"class",[206,529,530],{"class":293}," Chunk",[206,532,533],{"class":423},":\n",[206,535,537,540,542],{"class":289,"line":536},17,[206,538,539],{"class":338},"    doc_id",[206,541,431],{"class":423},[206,543,544],{"class":434}," str\n",[206,546,548,551,553],{"class":289,"line":547},18,[206,549,550],{"class":338},"    chunk_id",[206,552,431],{"class":423},[206,554,555],{"class":434}," int\n",[206,557,559,562,564],{"class":289,"line":558},19,[206,560,561],{"class":338},"    text",[206,563,431],{"class":423},[206,565,544],{"class":434},[206,567,569],{"class":289,"line":568},20,[206,570,346],{"emptyLinePlaceholder":345},[206,572,574],{"class":289,"line":573},21,[206,575,346],{"emptyLinePlaceholder":345},[206,577,579,581],{"class":289,"line":578},22,[206,580,518],{"class":517},[206,582,521],{"class":419},[206,584,586,588,591],{"class":289,"line":585},23,[206,587,527],{"class":415},[206,589,590],{"class":293}," SearchHit",[206,592,533],{"class":423},[206,594,596,599,601],{"class":289,"line":595},24,[206,597,598],{"class":338},"    chunk",[206,600,431],{"class":423},[206,602,603],{"class":338}," Chunk\n",[206,605,607,610,612],{"class":289,"line":606},25,[206,608,609],{"class":338},"    score",[206,611,431],{"class":423},[206,613,614],{"class":434}," float\n",[206,616,618],{"class":289,"line":617},26,[206,619,346],{"emptyLinePlaceholder":345},[206,621,623],{"class":289,"line":622},27,[206,624,346],{"emptyLinePlaceholder":345},[206,626,628,630,633],{"class":289,"line":627},28,[206,629,527],{"class":415},[206,631,632],{"class":293}," DocumentIndex",[206,634,533],{"class":423},[206,636,638,642],{"class":289,"line":637},29,[206,639,641],{"class":640},"s2W-s","    \"\"\"",[206,643,645],{"class":644},"sithA","A BM25 index over text files in a directory.\n",[206,647,649],{"class":289,"line":648},30,[206,650,346],{"emptyLinePlaceholder":345},[206,652,654],{"class":289,"line":653},31,[206,655,656],{"class":644},"    Chunks files into ~500-token pieces with 50-token overlap.\n",[206,658,660],{"class":289,"line":659},32,[206,661,662],{"class":640},"    \"\"\"\n",[206,664,666],{"class":289,"line":665},33,[206,667,346],{"emptyLinePlaceholder":345},[206,669,671,674,678,680,684,686,689,691,694,697,699,701,704,706,709,712,716],{"class":289,"line":670},34,[206,672,673],{"class":415},"    def",[206,675,677],{"class":676},"sptTA"," __init__",[206,679,424],{"class":423},[206,681,683],{"class":682},"smCYv","self",[206,685,490],{"class":423},[206,687,688],{"class":427}," root",[206,690,431],{"class":423},[206,692,693],{"class":338}," Path ",[206,695,696],{"class":484},"|",[206,698,435],{"class":434},[206,700,490],{"class":423},[206,702,703],{"class":427}," chunk_tokens",[206,705,431],{"class":423},[206,707,708],{"class":434}," int",[206,710,711],{"class":484}," =",[206,713,715],{"class":714},"srdBf"," 500",[206,717,718],{"class":423},",\n",[206,720,722,725,727,729,731,734,736,738,742],{"class":289,"line":721},35,[206,723,724],{"class":427},"                 overlap",[206,726,431],{"class":423},[206,728,708],{"class":434},[206,730,711],{"class":484},[206,732,733],{"class":714}," 50",[206,735,438],{"class":423},[206,737,441],{"class":423},[206,739,741],{"class":740},"s39Yj"," None",[206,743,533],{"class":423},[206,745,747,750,752,756,758,761,763,765],{"class":289,"line":746},36,[206,748,749],{"class":331},"        self",[206,751,465],{"class":423},[206,753,755],{"class":754},"skxfh","root",[206,757,711],{"class":484},[206,759,760],{"class":468}," Path",[206,762,424],{"class":423},[206,764,755],{"class":468},[206,766,767],{"class":423},")\n",[206,769,771,773,775,778,780,782,784,787,790,792],{"class":289,"line":770},37,[206,772,749],{"class":331},[206,774,465],{"class":423},[206,776,777],{"class":754},"chunks",[206,779,431],{"class":423},[206,781,444],{"class":338},[206,783,447],{"class":423},[206,785,786],{"class":338},"Chunk",[206,788,789],{"class":423},"]",[206,791,711],{"class":484},[206,793,794],{"class":423}," []\n",[206,796,798,800,802,805,807,810,812,815],{"class":289,"line":797},38,[206,799,749],{"class":331},[206,801,465],{"class":423},[206,803,804],{"class":468},"_build",[206,806,424],{"class":423},[206,808,809],{"class":468},"chunk_tokens",[206,811,490],{"class":423},[206,813,814],{"class":468}," overlap",[206,816,767],{"class":423},[206,818,820,823,826,829,832,834,837,839,841,843,846,849,852,855,857,859],{"class":289,"line":819},39,[206,821,822],{"class":338},"        tokenized ",[206,824,825],{"class":484},"=",[206,827,828],{"class":423}," [",[206,830,831],{"class":468},"_tokenize",[206,833,424],{"class":423},[206,835,836],{"class":468},"c",[206,838,465],{"class":423},[206,840,428],{"class":754},[206,842,438],{"class":423},[206,844,845],{"class":327}," for",[206,847,848],{"class":338}," c ",[206,850,851],{"class":327},"in",[206,853,854],{"class":331}," self",[206,856,465],{"class":423},[206,858,777],{"class":754},[206,860,861],{"class":423},"]\n",[206,863,865,867,869,872,874,877,879,882],{"class":289,"line":864},40,[206,866,749],{"class":331},[206,868,465],{"class":423},[206,870,871],{"class":754},"_bm25",[206,873,711],{"class":484},[206,875,876],{"class":468}," BM25Okapi",[206,878,424],{"class":423},[206,880,881],{"class":468},"tokenized",[206,883,767],{"class":423},[206,885,887],{"class":289,"line":886},41,[206,888,346],{"emptyLinePlaceholder":345},[206,890,892,894,897,899,901,903,905,907,909,911,913,915,917,919,921,923],{"class":289,"line":891},42,[206,893,673],{"class":415},[206,895,896],{"class":419}," _build",[206,898,424],{"class":423},[206,900,683],{"class":682},[206,902,490],{"class":423},[206,904,703],{"class":427},[206,906,431],{"class":423},[206,908,708],{"class":434},[206,910,490],{"class":423},[206,912,814],{"class":427},[206,914,431],{"class":423},[206,916,708],{"class":434},[206,918,438],{"class":423},[206,920,441],{"class":423},[206,922,741],{"class":740},[206,924,533],{"class":423},[206,926,928,931,934,936,939,941,943,945,947,949,952,954,956,959,961],{"class":289,"line":927},43,[206,929,930],{"class":327},"        for",[206,932,933],{"class":338}," path ",[206,935,851],{"class":327},[206,937,938],{"class":676}," sorted",[206,940,424],{"class":423},[206,942,683],{"class":331},[206,944,465],{"class":423},[206,946,755],{"class":754},[206,948,465],{"class":423},[206,950,951],{"class":468},"rglob",[206,953,424],{"class":423},[206,955,477],{"class":301},[206,957,958],{"class":297},"*",[206,960,477],{"class":301},[206,962,963],{"class":423},")):\n",[206,965,967,970,973,976,978,981],{"class":289,"line":966},44,[206,968,969],{"class":327},"            if",[206,971,972],{"class":484}," not",[206,974,975],{"class":338}," path",[206,977,465],{"class":423},[206,979,980],{"class":468},"is_file",[206,982,983],{"class":423},"():\n",[206,985,987],{"class":289,"line":986},45,[206,988,989],{"class":327},"                continue\n",[206,991,993,996],{"class":289,"line":992},46,[206,994,995],{"class":327},"            try",[206,997,533],{"class":423},[206,999,1001,1004,1006,1008,1010,1013,1015,1019,1021,1023,1026,1028],{"class":289,"line":1000},47,[206,1002,1003],{"class":338},"                text ",[206,1005,825],{"class":484},[206,1007,975],{"class":338},[206,1009,465],{"class":423},[206,1011,1012],{"class":468},"read_text",[206,1014,424],{"class":423},[206,1016,1018],{"class":1017},"s99_P","encoding",[206,1020,825],{"class":484},[206,1022,477],{"class":301},[206,1024,1025],{"class":297},"utf-8",[206,1027,477],{"class":301},[206,1029,767],{"class":423},[206,1031,1033,1036,1039,1042,1044,1047],{"class":289,"line":1032},48,[206,1034,1035],{"class":327},"            except",[206,1037,1038],{"class":423}," (",[206,1040,1041],{"class":434},"UnicodeDecodeError",[206,1043,490],{"class":423},[206,1045,1046],{"class":434}," PermissionError",[206,1048,1049],{"class":423},"):\n",[206,1051,1053],{"class":289,"line":1052},49,[206,1054,989],{"class":327},[206,1056,1058,1061,1063,1065,1067,1070],{"class":289,"line":1057},50,[206,1059,1060],{"class":338},"            words ",[206,1062,825],{"class":484},[206,1064,493],{"class":338},[206,1066,465],{"class":423},[206,1068,1069],{"class":468},"split",[206,1071,1072],{"class":423},"()\n",[206,1074,1076,1079,1082,1084,1087,1089,1092,1094,1097,1099,1102,1104,1107,1109,1112],{"class":289,"line":1075},51,[206,1077,1078],{"class":327},"            for",[206,1080,1081],{"class":338}," i",[206,1083,490],{"class":423},[206,1085,1086],{"class":338}," start ",[206,1088,851],{"class":327},[206,1090,1091],{"class":676}," enumerate",[206,1093,424],{"class":423},[206,1095,1096],{"class":676},"range",[206,1098,424],{"class":423},[206,1100,1101],{"class":714},"0",[206,1103,490],{"class":423},[206,1105,1106],{"class":676}," len",[206,1108,424],{"class":423},[206,1110,1111],{"class":468},"words",[206,1113,1114],{"class":423},"),\n",[206,1116,1118,1121,1124,1126],{"class":289,"line":1117},52,[206,1119,1120],{"class":468},"                                             chunk_tokens ",[206,1122,1123],{"class":484},"-",[206,1125,814],{"class":468},[206,1127,963],{"class":423},[206,1129,1131,1134,1136,1139,1141,1143,1146,1148,1150,1152,1155,1157,1160,1162,1164],{"class":289,"line":1130},53,[206,1132,1133],{"class":338},"                chunk_text ",[206,1135,825],{"class":484},[206,1137,1138],{"class":301}," \"",[206,1140,1138],{"class":301},[206,1142,465],{"class":423},[206,1144,1145],{"class":468},"join",[206,1147,424],{"class":423},[206,1149,1111],{"class":468},[206,1151,447],{"class":423},[206,1153,1154],{"class":468},"start",[206,1156,431],{"class":423},[206,1158,1159],{"class":468},"start ",[206,1161,485],{"class":484},[206,1163,703],{"class":468},[206,1165,1166],{"class":423},"])\n",[206,1168,1170,1173,1176,1178,1181],{"class":289,"line":1169},54,[206,1171,1172],{"class":327},"                if",[206,1174,1175],{"class":338}," chunk_text",[206,1177,465],{"class":423},[206,1179,1180],{"class":468},"strip",[206,1182,983],{"class":423},[206,1184,1186,1189,1191,1193,1195,1198,1200,1202],{"class":289,"line":1185},55,[206,1187,1188],{"class":331},"                    self",[206,1190,465],{"class":423},[206,1192,777],{"class":754},[206,1194,465],{"class":423},[206,1196,1197],{"class":468},"append",[206,1199,424],{"class":423},[206,1201,786],{"class":468},[206,1203,1204],{"class":423},"(\n",[206,1206,1208,1211,1213,1215,1217,1220,1222,1225,1227,1229,1231,1233],{"class":289,"line":1207},56,[206,1209,1210],{"class":1017},"                        doc_id",[206,1212,825],{"class":484},[206,1214,450],{"class":434},[206,1216,424],{"class":423},[206,1218,1219],{"class":468},"path",[206,1221,465],{"class":423},[206,1223,1224],{"class":468},"relative_to",[206,1226,424],{"class":423},[206,1228,683],{"class":331},[206,1230,465],{"class":423},[206,1232,755],{"class":754},[206,1234,1235],{"class":423},")),\n",[206,1237,1239,1242,1244,1247],{"class":289,"line":1238},57,[206,1240,1241],{"class":1017},"                        chunk_id",[206,1243,825],{"class":484},[206,1245,1246],{"class":468},"i",[206,1248,718],{"class":423},[206,1250,1252,1255,1257,1260],{"class":289,"line":1251},58,[206,1253,1254],{"class":1017},"                        text",[206,1256,825],{"class":484},[206,1258,1259],{"class":468},"chunk_text",[206,1261,718],{"class":423},[206,1263,1265],{"class":289,"line":1264},59,[206,1266,1267],{"class":423},"                    ))\n",[206,1269,1271],{"class":289,"line":1270},60,[206,1272,346],{"emptyLinePlaceholder":345},[206,1274,1276,1278,1281,1283,1285,1287,1290,1292,1294,1296,1299,1301,1303,1305,1308,1310,1312,1314,1316,1319],{"class":289,"line":1275},61,[206,1277,673],{"class":415},[206,1279,1280],{"class":419}," search",[206,1282,424],{"class":423},[206,1284,683],{"class":682},[206,1286,490],{"class":423},[206,1288,1289],{"class":427}," query",[206,1291,431],{"class":423},[206,1293,435],{"class":434},[206,1295,490],{"class":423},[206,1297,1298],{"class":427}," k",[206,1300,431],{"class":423},[206,1302,708],{"class":434},[206,1304,711],{"class":484},[206,1306,1307],{"class":714}," 5",[206,1309,438],{"class":423},[206,1311,441],{"class":423},[206,1313,444],{"class":338},[206,1315,447],{"class":423},[206,1317,1318],{"class":338},"SearchHit",[206,1320,453],{"class":423},[206,1322,1324,1327,1329,1331,1333,1336],{"class":289,"line":1323},62,[206,1325,1326],{"class":338},"        tokenized_query ",[206,1328,825],{"class":484},[206,1330,420],{"class":468},[206,1332,424],{"class":423},[206,1334,1335],{"class":468},"query",[206,1337,767],{"class":423},[206,1339,1341,1344,1346,1348,1350,1352,1354,1357,1359,1362],{"class":289,"line":1340},63,[206,1342,1343],{"class":338},"        scores ",[206,1345,825],{"class":484},[206,1347,854],{"class":331},[206,1349,465],{"class":423},[206,1351,871],{"class":754},[206,1353,465],{"class":423},[206,1355,1356],{"class":468},"get_scores",[206,1358,424],{"class":423},[206,1360,1361],{"class":468},"tokenized_query",[206,1363,767],{"class":423},[206,1365,1367,1370,1372,1374,1376,1379,1381,1384,1387,1390,1392,1395,1398,1400,1403,1406,1408,1411,1414,1417],{"class":289,"line":1366},64,[206,1368,1369],{"class":338},"        indexed ",[206,1371,825],{"class":484},[206,1373,938],{"class":676},[206,1375,424],{"class":423},[206,1377,1378],{"class":676},"enumerate",[206,1380,424],{"class":423},[206,1382,1383],{"class":468},"scores",[206,1385,1386],{"class":423},"),",[206,1388,1389],{"class":1017}," key",[206,1391,825],{"class":484},[206,1393,1394],{"class":415},"lambda",[206,1396,1397],{"class":427}," x",[206,1399,431],{"class":423},[206,1401,1402],{"class":484}," -",[206,1404,1405],{"class":468},"x",[206,1407,447],{"class":423},[206,1409,1410],{"class":714},"1",[206,1412,1413],{"class":423},"])[:",[206,1415,1416],{"class":338},"k",[206,1418,861],{"class":423},[206,1420,1422,1425,1427,1429,1431,1434,1436,1438,1440,1442,1444,1446,1449,1452,1454,1457],{"class":289,"line":1421},65,[206,1423,1424],{"class":327},"        return",[206,1426,828],{"class":423},[206,1428,1318],{"class":468},[206,1430,424],{"class":423},[206,1432,1433],{"class":1017},"chunk",[206,1435,825],{"class":484},[206,1437,683],{"class":331},[206,1439,465],{"class":423},[206,1441,777],{"class":754},[206,1443,447],{"class":423},[206,1445,1246],{"class":754},[206,1447,1448],{"class":423},"],",[206,1450,1451],{"class":1017}," score",[206,1453,825],{"class":484},[206,1455,1456],{"class":468},"s",[206,1458,767],{"class":423},[206,1460,1462,1465,1467,1469,1472,1474,1477,1480,1482,1485,1488],{"class":289,"line":1461},66,[206,1463,1464],{"class":327},"                for",[206,1466,1081],{"class":338},[206,1468,490],{"class":423},[206,1470,1471],{"class":338}," s ",[206,1473,851],{"class":327},[206,1475,1476],{"class":338}," indexed ",[206,1478,1479],{"class":327},"if",[206,1481,1471],{"class":338},[206,1483,1484],{"class":484},">",[206,1486,1487],{"class":714}," 0",[206,1489,861],{"class":423},[112,1491,1492],{},"Four design choices worth noting.",[112,1494,1495,1498],{},[135,1496,1497],{},"Word-based chunking, ~500 tokens, 50-token overlap."," Good enough for the book's scenarios; production systems use semantic chunking, sentence-aware splitters, or recursive structure-aware approaches. We optimize for readability, not SOTA retrieval quality. The overlap prevents information loss at chunk boundaries.",[112,1500,1501,1504],{},[135,1502,1503],{},"BM25, not embeddings."," BM25 is a bag-of-words score: TF-IDF on steroids. It works shockingly well on technical documentation, code, and any corpus with meaningful keywords. Embeddings are better for semantic similarity (paraphrase queries) but require an embedding model, a vector store, and a network hop. The book's harness can index 5,000 documents in seconds and search them in milliseconds; that's the right engineering budget here.",[112,1506,1507,1510,1511,1513,1514,1517],{},[135,1508,1509],{},"Filter zero-score hits."," BM25 returns a score for every chunk, many near zero. Returning them would pollute the agent's context with pretend-relevant noise. We cap at ",[285,1512,1416],{}," ",[115,1515,1516],{},"and"," require positive score; if the query matches nothing, we return empty.",[112,1519,1520,1530,1531,1534],{},[135,1521,1522,1523,1526,1527,465],{},"Chunks carry ",[285,1524,1525],{},"doc_id"," and ",[285,1528,1529],{},"chunk_id"," The agent sees where each hit came from. It can refer back to \"the third chunk of ",[285,1532,1533],{},"config.yaml","\" in its reasoning; Chapter 13's viewport reader can render the full chunk if needed.",[234,1536],{},[237,1538,1540],{"id":1539},"_103-the-retrieve-tool","10.3 The Retrieve Tool",[278,1542,1544],{"className":311,"code":1543,"language":313,"meta":283,"style":283},"# src\u002Fharness\u002Ftools\u002Fretrieval.py\nfrom __future__ import annotations\n\nfrom ..retrieval.index import DocumentIndex\nfrom .base import Tool\nfrom .decorator import tool\n\n\nclass RetrievalInterface:\n    def __init__(self, index: DocumentIndex) -> None:\n        self.index = index\n\n    def as_tools(self) -> list[Tool]:\n        idx = self.index\n\n        @tool(side_effects={\"read\"})\n        def search_docs(query: str, k: int = 5) -> str:\n            \"\"\"Search the document corpus for chunks matching a query.\n\n            query: keywords or a short sentence describing what you're\n                   looking for.\n            k: number of hits to return (default 5, max 10).\n\n            Returns up to k hits, each with: doc_id, chunk_id, score,\n            and the chunk text. Chunks are ~500 tokens each; plan your\n            context budget before calling with k > 3.\n\n            Side effects: reads the in-memory index.\n            \"\"\"\n            k = min(max(1, k), 10)\n            hits = idx.search(query, k=k)\n            if not hits:\n                return \"(no results)\"\n\n            lines: list[str] = []\n            total_chars = 0\n            for hit in hits:\n                c = hit.chunk\n                lines.append(f\"\\n--- {c.doc_id}#{c.chunk_id} \"\n                             f\"(score={hit.score:.2f}) ---\")\n                lines.append(c.text)\n                total_chars += len(c.text)\n            lines.append(f\"\\n[{len(hits)} hits, ~{total_chars} chars \"\n                         f\"(~{total_chars \u002F\u002F 4} tokens)]\")\n            return \"\\n\".join(lines)\n\n        return [search_docs]\n",[285,1545,1546,1551,1561,1565,1585,1600,1614,1618,1622,1631,1658,1671,1675,1699,1713,1717,1745,1781,1789,1793,1798,1803,1808,1812,1817,1822,1827,1831,1836,1841,1871,1900,1911,1924,1928,1947,1957,1970,1985,2034,2062,2080,2100,2145,2169,2191,2195],{"__ignoreMap":283},[206,1547,1548],{"class":289,"line":290},[206,1549,1550],{"class":320},"# src\u002Fharness\u002Ftools\u002Fretrieval.py\n",[206,1552,1553,1555,1557,1559],{"class":289,"line":324},[206,1554,328],{"class":327},[206,1556,332],{"class":331},[206,1558,335],{"class":327},[206,1560,339],{"class":338},[206,1562,1563],{"class":289,"line":342},[206,1564,346],{"emptyLinePlaceholder":345},[206,1566,1567,1569,1572,1575,1577,1580,1582],{"class":289,"line":349},[206,1568,328],{"class":327},[206,1570,1571],{"class":423}," ..",[206,1573,1574],{"class":338},"retrieval",[206,1576,465],{"class":423},[206,1578,1579],{"class":338},"index ",[206,1581,352],{"class":327},[206,1583,1584],{"class":338}," DocumentIndex\n",[206,1586,1587,1589,1592,1595,1597],{"class":289,"line":358},[206,1588,328],{"class":327},[206,1590,1591],{"class":423}," .",[206,1593,1594],{"class":338},"base ",[206,1596,352],{"class":327},[206,1598,1599],{"class":338}," Tool\n",[206,1601,1602,1604,1606,1609,1611],{"class":289,"line":371},[206,1603,328],{"class":327},[206,1605,1591],{"class":423},[206,1607,1608],{"class":338},"decorator ",[206,1610,352],{"class":327},[206,1612,1613],{"class":338}," tool\n",[206,1615,1616],{"class":289,"line":384},[206,1617,346],{"emptyLinePlaceholder":345},[206,1619,1620],{"class":289,"line":389},[206,1621,346],{"emptyLinePlaceholder":345},[206,1623,1624,1626,1629],{"class":289,"line":402},[206,1625,527],{"class":415},[206,1627,1628],{"class":293}," RetrievalInterface",[206,1630,533],{"class":423},[206,1632,1633,1635,1637,1639,1641,1643,1646,1648,1650,1652,1654,1656],{"class":289,"line":407},[206,1634,673],{"class":415},[206,1636,677],{"class":676},[206,1638,424],{"class":423},[206,1640,683],{"class":682},[206,1642,490],{"class":423},[206,1644,1645],{"class":427}," index",[206,1647,431],{"class":423},[206,1649,632],{"class":338},[206,1651,438],{"class":423},[206,1653,441],{"class":423},[206,1655,741],{"class":740},[206,1657,533],{"class":423},[206,1659,1660,1662,1664,1666,1668],{"class":289,"line":412},[206,1661,749],{"class":331},[206,1663,465],{"class":423},[206,1665,7],{"class":754},[206,1667,711],{"class":484},[206,1669,1670],{"class":338}," index\n",[206,1672,1673],{"class":289,"line":456},[206,1674,346],{"emptyLinePlaceholder":345},[206,1676,1677,1679,1682,1684,1686,1688,1690,1692,1694,1697],{"class":289,"line":504},[206,1678,673],{"class":415},[206,1680,1681],{"class":419}," as_tools",[206,1683,424],{"class":423},[206,1685,683],{"class":682},[206,1687,438],{"class":423},[206,1689,441],{"class":423},[206,1691,444],{"class":338},[206,1693,447],{"class":423},[206,1695,1696],{"class":338},"Tool",[206,1698,453],{"class":423},[206,1700,1701,1704,1706,1708,1710],{"class":289,"line":509},[206,1702,1703],{"class":338},"        idx ",[206,1705,825],{"class":484},[206,1707,854],{"class":331},[206,1709,465],{"class":423},[206,1711,1712],{"class":754},"index\n",[206,1714,1715],{"class":289,"line":514},[206,1716,346],{"emptyLinePlaceholder":345},[206,1718,1719,1722,1725,1727,1730,1732,1735,1737,1740,1742],{"class":289,"line":524},[206,1720,1721],{"class":517},"        @",[206,1723,1724],{"class":419},"tool",[206,1726,424],{"class":423},[206,1728,1729],{"class":1017},"side_effects",[206,1731,825],{"class":484},[206,1733,1734],{"class":423},"{",[206,1736,477],{"class":301},[206,1738,1739],{"class":297},"read",[206,1741,477],{"class":301},[206,1743,1744],{"class":423},"})\n",[206,1746,1747,1750,1753,1755,1757,1759,1761,1763,1765,1767,1769,1771,1773,1775,1777,1779],{"class":289,"line":536},[206,1748,1749],{"class":415},"        def",[206,1751,1752],{"class":419}," search_docs",[206,1754,424],{"class":423},[206,1756,1335],{"class":427},[206,1758,431],{"class":423},[206,1760,435],{"class":434},[206,1762,490],{"class":423},[206,1764,1298],{"class":427},[206,1766,431],{"class":423},[206,1768,708],{"class":434},[206,1770,711],{"class":484},[206,1772,1307],{"class":714},[206,1774,438],{"class":423},[206,1776,441],{"class":423},[206,1778,435],{"class":434},[206,1780,533],{"class":423},[206,1782,1783,1786],{"class":289,"line":547},[206,1784,1785],{"class":640},"            \"\"\"",[206,1787,1788],{"class":644},"Search the document corpus for chunks matching a query.\n",[206,1790,1791],{"class":289,"line":558},[206,1792,346],{"emptyLinePlaceholder":345},[206,1794,1795],{"class":289,"line":568},[206,1796,1797],{"class":644},"            query: keywords or a short sentence describing what you're\n",[206,1799,1800],{"class":289,"line":573},[206,1801,1802],{"class":644},"                   looking for.\n",[206,1804,1805],{"class":289,"line":578},[206,1806,1807],{"class":644},"            k: number of hits to return (default 5, max 10).\n",[206,1809,1810],{"class":289,"line":585},[206,1811,346],{"emptyLinePlaceholder":345},[206,1813,1814],{"class":289,"line":595},[206,1815,1816],{"class":644},"            Returns up to k hits, each with: doc_id, chunk_id, score,\n",[206,1818,1819],{"class":289,"line":606},[206,1820,1821],{"class":644},"            and the chunk text. Chunks are ~500 tokens each; plan your\n",[206,1823,1824],{"class":289,"line":617},[206,1825,1826],{"class":644},"            context budget before calling with k > 3.\n",[206,1828,1829],{"class":289,"line":622},[206,1830,346],{"emptyLinePlaceholder":345},[206,1832,1833],{"class":289,"line":627},[206,1834,1835],{"class":644},"            Side effects: reads the in-memory index.\n",[206,1837,1838],{"class":289,"line":637},[206,1839,1840],{"class":640},"            \"\"\"\n",[206,1842,1843,1846,1848,1851,1853,1856,1858,1860,1862,1864,1866,1869],{"class":289,"line":648},[206,1844,1845],{"class":338},"            k ",[206,1847,825],{"class":484},[206,1849,1850],{"class":676}," min",[206,1852,424],{"class":423},[206,1854,1855],{"class":676},"max",[206,1857,424],{"class":423},[206,1859,1410],{"class":714},[206,1861,490],{"class":423},[206,1863,1298],{"class":468},[206,1865,1386],{"class":423},[206,1867,1868],{"class":714}," 10",[206,1870,767],{"class":423},[206,1872,1873,1876,1878,1881,1883,1886,1888,1890,1892,1894,1896,1898],{"class":289,"line":653},[206,1874,1875],{"class":338},"            hits ",[206,1877,825],{"class":484},[206,1879,1880],{"class":338}," idx",[206,1882,465],{"class":423},[206,1884,1885],{"class":468},"search",[206,1887,424],{"class":423},[206,1889,1335],{"class":468},[206,1891,490],{"class":423},[206,1893,1298],{"class":1017},[206,1895,825],{"class":484},[206,1897,1416],{"class":468},[206,1899,767],{"class":423},[206,1901,1902,1904,1906,1909],{"class":289,"line":659},[206,1903,969],{"class":327},[206,1905,972],{"class":484},[206,1907,1908],{"class":338}," hits",[206,1910,533],{"class":423},[206,1912,1913,1916,1918,1921],{"class":289,"line":665},[206,1914,1915],{"class":327},"                return",[206,1917,1138],{"class":301},[206,1919,1920],{"class":297},"(no results)",[206,1922,1923],{"class":301},"\"\n",[206,1925,1926],{"class":289,"line":670},[206,1927,346],{"emptyLinePlaceholder":345},[206,1929,1930,1933,1935,1937,1939,1941,1943,1945],{"class":289,"line":721},[206,1931,1932],{"class":338},"            lines",[206,1934,431],{"class":423},[206,1936,444],{"class":338},[206,1938,447],{"class":423},[206,1940,450],{"class":434},[206,1942,789],{"class":423},[206,1944,711],{"class":484},[206,1946,794],{"class":423},[206,1948,1949,1952,1954],{"class":289,"line":746},[206,1950,1951],{"class":338},"            total_chars ",[206,1953,825],{"class":484},[206,1955,1956],{"class":714}," 0\n",[206,1958,1959,1961,1964,1966,1968],{"class":289,"line":770},[206,1960,1078],{"class":327},[206,1962,1963],{"class":338}," hit ",[206,1965,851],{"class":327},[206,1967,1908],{"class":338},[206,1969,533],{"class":423},[206,1971,1972,1975,1977,1980,1982],{"class":289,"line":797},[206,1973,1974],{"class":338},"                c ",[206,1976,825],{"class":484},[206,1978,1979],{"class":338}," hit",[206,1981,465],{"class":423},[206,1983,1984],{"class":754},"chunk\n",[206,1986,1987,1990,1992,1994,1996,1999,2001,2004,2007,2009,2011,2013,2015,2018,2021,2023,2025,2027,2029,2031],{"class":289,"line":819},[206,1988,1989],{"class":338},"                lines",[206,1991,465],{"class":423},[206,1993,1197],{"class":468},[206,1995,424],{"class":423},[206,1997,1998],{"class":415},"f",[206,2000,477],{"class":297},[206,2002,2003],{"class":331},"\\n",[206,2005,2006],{"class":297},"--- ",[206,2008,1734],{"class":714},[206,2010,836],{"class":468},[206,2012,465],{"class":423},[206,2014,1525],{"class":754},[206,2016,2017],{"class":714},"}",[206,2019,2020],{"class":297},"#",[206,2022,1734],{"class":714},[206,2024,836],{"class":468},[206,2026,465],{"class":423},[206,2028,1529],{"class":754},[206,2030,2017],{"class":714},[206,2032,2033],{"class":297}," \"\n",[206,2035,2036,2039,2042,2044,2047,2049,2052,2055,2057,2060],{"class":289,"line":864},[206,2037,2038],{"class":415},"                             f",[206,2040,2041],{"class":297},"\"(score=",[206,2043,1734],{"class":714},[206,2045,2046],{"class":468},"hit",[206,2048,465],{"class":423},[206,2050,2051],{"class":754},"score",[206,2053,2054],{"class":415},":.2f",[206,2056,2017],{"class":714},[206,2058,2059],{"class":297},") ---\"",[206,2061,767],{"class":423},[206,2063,2064,2066,2068,2070,2072,2074,2076,2078],{"class":289,"line":886},[206,2065,1989],{"class":338},[206,2067,465],{"class":423},[206,2069,1197],{"class":468},[206,2071,424],{"class":423},[206,2073,836],{"class":468},[206,2075,465],{"class":423},[206,2077,428],{"class":754},[206,2079,767],{"class":423},[206,2081,2082,2085,2088,2090,2092,2094,2096,2098],{"class":289,"line":891},[206,2083,2084],{"class":338},"                total_chars ",[206,2086,2087],{"class":484},"+=",[206,2089,1106],{"class":676},[206,2091,424],{"class":423},[206,2093,836],{"class":468},[206,2095,465],{"class":423},[206,2097,428],{"class":754},[206,2099,767],{"class":423},[206,2101,2102,2104,2106,2108,2110,2112,2114,2116,2118,2120,2123,2125,2128,2130,2132,2135,2137,2140,2142],{"class":289,"line":927},[206,2103,1932],{"class":338},[206,2105,465],{"class":423},[206,2107,1197],{"class":468},[206,2109,424],{"class":423},[206,2111,1998],{"class":415},[206,2113,477],{"class":297},[206,2115,2003],{"class":331},[206,2117,447],{"class":297},[206,2119,1734],{"class":714},[206,2121,2122],{"class":676},"len",[206,2124,424],{"class":423},[206,2126,2127],{"class":468},"hits",[206,2129,438],{"class":423},[206,2131,2017],{"class":714},[206,2133,2134],{"class":297}," hits, ~",[206,2136,1734],{"class":714},[206,2138,2139],{"class":468},"total_chars",[206,2141,2017],{"class":714},[206,2143,2144],{"class":297}," chars \"\n",[206,2146,2147,2150,2153,2155,2158,2161,2164,2167],{"class":289,"line":966},[206,2148,2149],{"class":415},"                         f",[206,2151,2152],{"class":297},"\"(~",[206,2154,1734],{"class":714},[206,2156,2157],{"class":468},"total_chars ",[206,2159,2160],{"class":484},"\u002F\u002F",[206,2162,2163],{"class":714}," 4}",[206,2165,2166],{"class":297}," tokens)]\"",[206,2168,767],{"class":423},[206,2170,2171,2174,2176,2178,2180,2182,2184,2186,2189],{"class":289,"line":986},[206,2172,2173],{"class":327},"            return",[206,2175,1138],{"class":301},[206,2177,2003],{"class":331},[206,2179,477],{"class":301},[206,2181,465],{"class":423},[206,2183,1145],{"class":468},[206,2185,424],{"class":423},[206,2187,2188],{"class":468},"lines",[206,2190,767],{"class":423},[206,2192,2193],{"class":289,"line":992},[206,2194,346],{"emptyLinePlaceholder":345},[206,2196,2197,2199,2201,2204],{"class":289,"line":1000},[206,2198,1424],{"class":327},[206,2200,828],{"class":423},[206,2202,2203],{"class":338},"search_docs",[206,2205,861],{"class":423},[112,2207,2208,2209,2211],{},"The tool description carries three specific instructions. It names the cost (chunks are ~500 tokens). It caps ",[285,2210,1416],{}," at 10. It includes the total token estimate in the result text, so the agent knows what it just paid for.",[112,2213,2214,2215,2218],{},"The last line of the result — ",[285,2216,2217],{},"[5 hits, ~12500 chars (~3125 tokens)]"," — is a deliberate choice. Without it, the agent has no way to feel the cost of retrieval. With it, the agent learns: \"this query cost me 3K tokens; I should synthesize rather than retrieve again.\"",[234,2220],{},[237,2222,2224],{"id":2223},"_104-edge-placement","10.4 Edge Placement",[112,2226,2227,2228,2231],{},"Retrieval hits come back as a ",[285,2229,2230],{},"ToolResult",", which ends up in the transcript like any other tool result. By the time the next turn runs, the hit is somewhere in the history. If the session is long, the hit is in the middle — the worst position.",[112,2233,2234,2235,2238],{},"The fix: we want retrieved content to be freshly placed at the ",[115,2236,2237],{},"end"," of the context on the turn the agent wants to act on it. Two ways to do this.",[112,2240,2241,2244,2245,2249],{},[135,2242,2243],{},"The agent chooses placement."," The agent reads the hit from the tool result and rewrites it into its own reasoning on the next turn. \"I found: ",[2246,2247,2248],"quote",{},"...",". Based on this, I will...\" The retrieved content now occupies the fresh assistant-message position. This is how most agents work naturally, and it works as long as the agent has the discipline.",[112,2251,2252,2255],{},[135,2253,2254],{},"The harness places it."," The harness intercepts search results and re-inserts them as a synthesized recent message, right before the next user turn. This is more invasive — and can confuse the model about what happened — but it guarantees placement regardless of agent discipline.",[112,2257,2258],{},"We do the first, with a small assist: the retrieval tool's result is structured so the agent can easily lift it verbatim. Chapter 16's structured plans build on this pattern — the plan is the thing the agent reads every turn, and it sits at the end of context by construction.",[234,2260],{},[237,2262,2264],{"id":2263},"_105-the-scenario","10.5 The Scenario",[278,2266,2268],{"className":311,"code":2267,"language":313,"meta":283,"style":283},"# examples\u002Fch10_corpus.py\nimport asyncio\nfrom pathlib import Path\n\nfrom harness.agent import arun\nfrom harness.context.accountant import ContextAccountant\nfrom harness.context.compactor import Compactor\nfrom harness.providers.anthropic import AnthropicProvider\nfrom harness.retrieval.index import DocumentIndex\nfrom harness.tools.registry import ToolRegistry\nfrom harness.tools.retrieval import RetrievalInterface\nfrom harness.tools.std import calc, read_file, write_file\n\n\nSYSTEM = \"\"\"\\\nYou have a tool `search_docs(query, k)` that searches a corpus of\ndocumentation. Use it when the user asks questions that likely have answers\nin the docs, rather than guessing. Each result is ~500 tokens; prefer k=3\nor k=5 over k=10 unless you need breadth. After getting results, quote the\nrelevant passages in your reasoning — do not rely on memory of them across\nmany turns. If the first query is not useful, refine the query; do not\ngive up after one search.\n\"\"\"\n\n\nasync def main() -> None:\n    provider = AnthropicProvider()\n    index = DocumentIndex(root=Path(\".\u002Fdocs-corpus\"))\n    retriever = RetrievalInterface(index)\n    registry = ToolRegistry(tools=[calc, read_file, write_file,\n                                    *retriever.as_tools()])\n    accountant = ContextAccountant()\n    compactor = Compactor(accountant, provider)\n\n    await arun(\n        provider=provider,\n        registry=registry,\n        system=SYSTEM,\n        accountant=accountant,\n        compactor=compactor,\n        user_message=(\n            \"Look through the docs and explain how retry budgets are \"\n            \"configured. Quote the relevant passage. If retry budgets \"\n            \"aren't documented, say so explicitly.\"\n        ),\n    )\n\n\nasyncio.run(main())\n",[285,2269,2270,2275,2282,2292,2296,2313,2334,2354,2375,2393,2414,2434,2464,2468,2472,2485,2490,2495,2500,2505,2510,2515,2520,2525,2529,2533,2553,2565,2595,2610,2642,2658,2670,2692,2696,2706,2718,2730,2741,2752,2764,2773,2783,2792,2801,2806,2811,2815,2819],{"__ignoreMap":283},[206,2271,2272],{"class":289,"line":290},[206,2273,2274],{"class":320},"# examples\u002Fch10_corpus.py\n",[206,2276,2277,2279],{"class":289,"line":324},[206,2278,352],{"class":327},[206,2280,2281],{"class":338}," asyncio\n",[206,2283,2284,2286,2288,2290],{"class":289,"line":342},[206,2285,328],{"class":327},[206,2287,376],{"class":338},[206,2289,352],{"class":327},[206,2291,381],{"class":338},[206,2293,2294],{"class":289,"line":349},[206,2295,346],{"emptyLinePlaceholder":345},[206,2297,2298,2300,2303,2305,2308,2310],{"class":289,"line":358},[206,2299,328],{"class":327},[206,2301,2302],{"class":338}," harness",[206,2304,465],{"class":423},[206,2306,2307],{"class":338},"agent ",[206,2309,352],{"class":327},[206,2311,2312],{"class":338}," arun\n",[206,2314,2315,2317,2319,2321,2324,2326,2329,2331],{"class":289,"line":371},[206,2316,328],{"class":327},[206,2318,2302],{"class":338},[206,2320,465],{"class":423},[206,2322,2323],{"class":338},"context",[206,2325,465],{"class":423},[206,2327,2328],{"class":338},"accountant ",[206,2330,352],{"class":327},[206,2332,2333],{"class":338}," ContextAccountant\n",[206,2335,2336,2338,2340,2342,2344,2346,2349,2351],{"class":289,"line":384},[206,2337,328],{"class":327},[206,2339,2302],{"class":338},[206,2341,465],{"class":423},[206,2343,2323],{"class":338},[206,2345,465],{"class":423},[206,2347,2348],{"class":338},"compactor ",[206,2350,352],{"class":327},[206,2352,2353],{"class":338}," Compactor\n",[206,2355,2356,2358,2360,2362,2365,2367,2370,2372],{"class":289,"line":389},[206,2357,328],{"class":327},[206,2359,2302],{"class":338},[206,2361,465],{"class":423},[206,2363,2364],{"class":338},"providers",[206,2366,465],{"class":423},[206,2368,2369],{"class":338},"anthropic ",[206,2371,352],{"class":327},[206,2373,2374],{"class":338}," AnthropicProvider\n",[206,2376,2377,2379,2381,2383,2385,2387,2389,2391],{"class":289,"line":402},[206,2378,328],{"class":327},[206,2380,2302],{"class":338},[206,2382,465],{"class":423},[206,2384,1574],{"class":338},[206,2386,465],{"class":423},[206,2388,1579],{"class":338},[206,2390,352],{"class":327},[206,2392,1584],{"class":338},[206,2394,2395,2397,2399,2401,2404,2406,2409,2411],{"class":289,"line":407},[206,2396,328],{"class":327},[206,2398,2302],{"class":338},[206,2400,465],{"class":423},[206,2402,2403],{"class":338},"tools",[206,2405,465],{"class":423},[206,2407,2408],{"class":338},"registry ",[206,2410,352],{"class":327},[206,2412,2413],{"class":338}," ToolRegistry\n",[206,2415,2416,2418,2420,2422,2424,2426,2429,2431],{"class":289,"line":412},[206,2417,328],{"class":327},[206,2419,2302],{"class":338},[206,2421,465],{"class":423},[206,2423,2403],{"class":338},[206,2425,465],{"class":423},[206,2427,2428],{"class":338},"retrieval ",[206,2430,352],{"class":327},[206,2432,2433],{"class":338}," RetrievalInterface\n",[206,2435,2436,2438,2440,2442,2444,2446,2449,2451,2454,2456,2459,2461],{"class":289,"line":456},[206,2437,328],{"class":327},[206,2439,2302],{"class":338},[206,2441,465],{"class":423},[206,2443,2403],{"class":338},[206,2445,465],{"class":423},[206,2447,2448],{"class":338},"std ",[206,2450,352],{"class":327},[206,2452,2453],{"class":338}," calc",[206,2455,490],{"class":423},[206,2457,2458],{"class":338}," read_file",[206,2460,490],{"class":423},[206,2462,2463],{"class":338}," write_file\n",[206,2465,2466],{"class":289,"line":504},[206,2467,346],{"emptyLinePlaceholder":345},[206,2469,2470],{"class":289,"line":509},[206,2471,346],{"emptyLinePlaceholder":345},[206,2473,2474,2477,2479,2482],{"class":289,"line":514},[206,2475,2476],{"class":331},"SYSTEM",[206,2478,711],{"class":484},[206,2480,2481],{"class":301}," \"\"\"",[206,2483,2484],{"class":740},"\\\n",[206,2486,2487],{"class":289,"line":524},[206,2488,2489],{"class":297},"You have a tool `search_docs(query, k)` that searches a corpus of\n",[206,2491,2492],{"class":289,"line":536},[206,2493,2494],{"class":297},"documentation. Use it when the user asks questions that likely have answers\n",[206,2496,2497],{"class":289,"line":547},[206,2498,2499],{"class":297},"in the docs, rather than guessing. Each result is ~500 tokens; prefer k=3\n",[206,2501,2502],{"class":289,"line":558},[206,2503,2504],{"class":297},"or k=5 over k=10 unless you need breadth. After getting results, quote the\n",[206,2506,2507],{"class":289,"line":568},[206,2508,2509],{"class":297},"relevant passages in your reasoning — do not rely on memory of them across\n",[206,2511,2512],{"class":289,"line":573},[206,2513,2514],{"class":297},"many turns. If the first query is not useful, refine the query; do not\n",[206,2516,2517],{"class":289,"line":578},[206,2518,2519],{"class":297},"give up after one search.\n",[206,2521,2522],{"class":289,"line":585},[206,2523,2524],{"class":301},"\"\"\"\n",[206,2526,2527],{"class":289,"line":595},[206,2528,346],{"emptyLinePlaceholder":345},[206,2530,2531],{"class":289,"line":606},[206,2532,346],{"emptyLinePlaceholder":345},[206,2534,2535,2538,2541,2544,2547,2549,2551],{"class":289,"line":617},[206,2536,2537],{"class":415},"async",[206,2539,2540],{"class":415}," def",[206,2542,2543],{"class":419}," main",[206,2545,2546],{"class":423},"()",[206,2548,441],{"class":423},[206,2550,741],{"class":740},[206,2552,533],{"class":423},[206,2554,2555,2558,2560,2563],{"class":289,"line":622},[206,2556,2557],{"class":338},"    provider ",[206,2559,825],{"class":484},[206,2561,2562],{"class":468}," AnthropicProvider",[206,2564,1072],{"class":423},[206,2566,2567,2570,2572,2574,2576,2578,2580,2583,2585,2587,2590,2592],{"class":289,"line":627},[206,2568,2569],{"class":338},"    index ",[206,2571,825],{"class":484},[206,2573,632],{"class":468},[206,2575,424],{"class":423},[206,2577,755],{"class":1017},[206,2579,825],{"class":484},[206,2581,2582],{"class":468},"Path",[206,2584,424],{"class":423},[206,2586,477],{"class":301},[206,2588,2589],{"class":297},".\u002Fdocs-corpus",[206,2591,477],{"class":301},[206,2593,2594],{"class":423},"))\n",[206,2596,2597,2600,2602,2604,2606,2608],{"class":289,"line":637},[206,2598,2599],{"class":338},"    retriever ",[206,2601,825],{"class":484},[206,2603,1628],{"class":468},[206,2605,424],{"class":423},[206,2607,7],{"class":468},[206,2609,767],{"class":423},[206,2611,2612,2615,2617,2620,2622,2624,2626,2628,2631,2633,2635,2637,2640],{"class":289,"line":648},[206,2613,2614],{"class":338},"    registry ",[206,2616,825],{"class":484},[206,2618,2619],{"class":468}," ToolRegistry",[206,2621,424],{"class":423},[206,2623,2403],{"class":1017},[206,2625,825],{"class":484},[206,2627,447],{"class":423},[206,2629,2630],{"class":468},"calc",[206,2632,490],{"class":423},[206,2634,2458],{"class":468},[206,2636,490],{"class":423},[206,2638,2639],{"class":468}," write_file",[206,2641,718],{"class":423},[206,2643,2644,2647,2650,2652,2655],{"class":289,"line":653},[206,2645,2646],{"class":484},"                                    *",[206,2648,2649],{"class":468},"retriever",[206,2651,465],{"class":423},[206,2653,2654],{"class":468},"as_tools",[206,2656,2657],{"class":423},"()])\n",[206,2659,2660,2663,2665,2668],{"class":289,"line":659},[206,2661,2662],{"class":338},"    accountant ",[206,2664,825],{"class":484},[206,2666,2667],{"class":468}," ContextAccountant",[206,2669,1072],{"class":423},[206,2671,2672,2675,2677,2680,2682,2685,2687,2690],{"class":289,"line":665},[206,2673,2674],{"class":338},"    compactor ",[206,2676,825],{"class":484},[206,2678,2679],{"class":468}," Compactor",[206,2681,424],{"class":423},[206,2683,2684],{"class":468},"accountant",[206,2686,490],{"class":423},[206,2688,2689],{"class":468}," provider",[206,2691,767],{"class":423},[206,2693,2694],{"class":289,"line":670},[206,2695,346],{"emptyLinePlaceholder":345},[206,2697,2698,2701,2704],{"class":289,"line":721},[206,2699,2700],{"class":327},"    await",[206,2702,2703],{"class":468}," arun",[206,2705,1204],{"class":423},[206,2707,2708,2711,2713,2716],{"class":289,"line":746},[206,2709,2710],{"class":1017},"        provider",[206,2712,825],{"class":484},[206,2714,2715],{"class":468},"provider",[206,2717,718],{"class":423},[206,2719,2720,2723,2725,2728],{"class":289,"line":770},[206,2721,2722],{"class":1017},"        registry",[206,2724,825],{"class":484},[206,2726,2727],{"class":468},"registry",[206,2729,718],{"class":423},[206,2731,2732,2735,2737,2739],{"class":289,"line":797},[206,2733,2734],{"class":1017},"        system",[206,2736,825],{"class":484},[206,2738,2476],{"class":676},[206,2740,718],{"class":423},[206,2742,2743,2746,2748,2750],{"class":289,"line":819},[206,2744,2745],{"class":1017},"        accountant",[206,2747,825],{"class":484},[206,2749,2684],{"class":468},[206,2751,718],{"class":423},[206,2753,2754,2757,2759,2762],{"class":289,"line":864},[206,2755,2756],{"class":1017},"        compactor",[206,2758,825],{"class":484},[206,2760,2761],{"class":468},"compactor",[206,2763,718],{"class":423},[206,2765,2766,2769,2771],{"class":289,"line":886},[206,2767,2768],{"class":1017},"        user_message",[206,2770,825],{"class":484},[206,2772,1204],{"class":423},[206,2774,2775,2778,2781],{"class":289,"line":891},[206,2776,2777],{"class":301},"            \"",[206,2779,2780],{"class":297},"Look through the docs and explain how retry budgets are ",[206,2782,1923],{"class":301},[206,2784,2785,2787,2790],{"class":289,"line":927},[206,2786,2777],{"class":301},[206,2788,2789],{"class":297},"configured. Quote the relevant passage. If retry budgets ",[206,2791,1923],{"class":301},[206,2793,2794,2796,2799],{"class":289,"line":966},[206,2795,2777],{"class":301},[206,2797,2798],{"class":297},"aren't documented, say so explicitly.",[206,2800,1923],{"class":301},[206,2802,2803],{"class":289,"line":986},[206,2804,2805],{"class":423},"        ),\n",[206,2807,2808],{"class":289,"line":992},[206,2809,2810],{"class":423},"    )\n",[206,2812,2813],{"class":289,"line":1000},[206,2814,346],{"emptyLinePlaceholder":345},[206,2816,2817],{"class":289,"line":1032},[206,2818,346],{"emptyLinePlaceholder":345},[206,2820,2821,2824,2826,2829,2831,2834],{"class":289,"line":1052},[206,2822,2823],{"class":338},"asyncio",[206,2825,465],{"class":423},[206,2827,2828],{"class":468},"run",[206,2830,424],{"class":423},[206,2832,2833],{"class":468},"main",[206,2835,501],{"class":423},[112,2837,2838,2839,2842],{},"Point this at any directory with docs — the book's own ",[285,2840,2841],{},"research\u002F"," directory works, or a cloned project's docs. The agent now queries the index instead of trying to divine the answer; when the query is weak, it retries with a better one; when the answer isn't in the corpus, it says so, because the retrieved chunks don't mention retry budgets and the agent knows not to invent.",[234,2844],{},[237,2846,2848],{"id":2847},"_106-when-retrieval-hurts","10.6 When Retrieval Hurts",[112,2850,2851],{},"Three failure modes to recognize.",[112,2853,2854,2857,2858,2860],{},[135,2855,2856],{},"Distractor interference."," The query returns chunks that look related but aren't. The model latches onto them and answers confidently wrong. Mitigation: higher score thresholds (our code filters score > 0, but you can lift the floor to 0.5 or 1.0 depending on your corpus); smaller ",[285,2859,1416],{},"; better chunk boundaries. Evals — Chapter 19 — are how you discover whether your thresholds are right for your corpus.",[112,2862,2863,2866],{},[135,2864,2865],{},"Query-document mismatch."," The user asks about \"rate limiting\"; the docs use \"throttling\"; BM25 doesn't know they're synonyms. An embedding-based index would handle this; BM25 requires the agent to re-query with broader terms. Well-written tool descriptions that tell the agent to refine queries help a lot here.",[112,2868,2869,2872,2873,2875],{},[135,2870,2871],{},"Redundancy within top-K."," Two of the five hits are the same content from overlapping chunks. The model burns tokens on a duplicate. Mitigation: de-duplicate by doc\u002Fchunk proximity in the retriever, or enlarge the chunk size and reduce K. Simple post-filtering in ",[285,2874,2203],{}," would be: after the top-K, skip any chunk that overlaps with an already-included one by more than X tokens.",[234,2877],{},[237,2879,2881],{"id":2880},"_107-hybrid-retrieval-and-why-were-not-building-it","10.7 Hybrid Retrieval and Why We're Not Building It",[112,2883,2884,2885,2888,2889,2891],{},"Production retrieval systems usually combine BM25 (keyword precision) with embeddings (semantic recall) via reciprocal rank fusion. The harness supports this straightforwardly — swap ",[285,2886,2887],{},"DocumentIndex"," for a hybrid implementation, keep the same ",[285,2890,1885],{}," method — but the book doesn't need it. The scenarios we run are keyword-rich (technical docs, code, configs), and BM25 dominates on those.",[112,2893,2894],{},"When you'd switch:",[2896,2897,2898,2904,2910],"ul",{},[132,2899,2900,2903],{},[135,2901,2902],{},"Paraphrase-heavy queries."," Users asking \"how do I make my agent remember things?\" when the docs say \"context persistence.\"",[132,2905,2906,2909],{},[135,2907,2908],{},"Cross-lingual."," Queries in one language, docs in another.",[132,2911,2912,2915],{},[135,2913,2914],{},"Very short documents."," Tweets, SMS, short FAQ entries — BM25 starves on short texts because the TF component has nothing to work with.",[112,2917,2918],{},"For everything the book builds, BM25 is sufficient. Chapter 22 lists hybrid retrieval as a first-class upgrade path.",[234,2920],{},[237,2922,2924],{"id":2923},"_108-commit","10.8 Commit",[278,2926,2928],{"className":280,"code":2927,"language":282,"meta":283,"style":283},"git add -A && git commit -m \"ch10: BM25 document index + agent-driven retrieval tool\"\ngit tag ch10-retrieval\n",[285,2929,2930,2959],{"__ignoreMap":283},[206,2931,2932,2935,2937,2940,2943,2946,2949,2952,2954,2957],{"class":289,"line":290},[206,2933,2934],{"class":293},"git",[206,2936,298],{"class":297},[206,2938,2939],{"class":480}," -A",[206,2941,2942],{"class":423}," &&",[206,2944,2945],{"class":293}," git",[206,2947,2948],{"class":297}," commit",[206,2950,2951],{"class":480}," -m",[206,2953,1138],{"class":301},[206,2955,2956],{"class":297},"ch10: BM25 document index + agent-driven retrieval tool",[206,2958,1923],{"class":301},[206,2960,2961,2963,2966],{"class":289,"line":324},[206,2962,2934],{"class":293},[206,2964,2965],{"class":297}," tag",[206,2967,2968],{"class":297}," ch10-retrieval\n",[237,2970,2972],{"id":2971},"_109-try-it-yourself","10.9 Try It Yourself",[129,2974,2975,2988,2994],{},[132,2976,2977,2980,2981,2983,2984,2987],{},[135,2978,2979],{},"Index the book itself."," Point ",[285,2982,2887],{}," at this book's ",[285,2985,2986],{},"chapters\u002F"," directory and ask the agent \"how does compaction work in this harness?\" Does the retrieval find Chapter 8? If not, what's wrong with the chunking or the query?",[132,2989,2990,2993],{},[135,2991,2992],{},"Stress the retrieval."," Index a directory with 10,000+ files (a cloned open-source project's source tree, say). Time the index build and the query. Acceptable? If not, what would you profile first?",[132,2995,2996,2999],{},[135,2997,2998],{},"Build a distractor test."," Index two directories — one with docs on a topic, one with docs on an unrelated topic. Ask a question whose answer is in the first. Measure how often the second directory's chunks appear in top-5. That's your distractor rate; it tells you whether to raise your score threshold or rewrite your chunks.",[234,3001],{},[3003,3004,3005,3008],"what-you-understand",{},[112,3006,3007],{},"Your agent can query a document corpus via a tool, choose when to retrieve, see the retrieval cost, and place results where the model will actually read them. BM25 gets you far on keyword-rich corpora; the upgrade path to embeddings or hybrid retrieval is clean. The harness now has the three context-engineering pillars — accounting, compaction, external state (scratchpad), retrieval — all interoperating through the same seams.",[112,3009,3010,3011,3014,3015,3018,3019,3022],{},"What's still missing. Every tool the agent has reads or writes in a way optimized for ",[115,3012,3013],{},"humans"," — ",[285,3016,3017],{},"read_file"," returns the whole file, ",[285,3020,3021],{},"write_file"," overwrites wholesale. When you watched the long-session scenario in Chapter 8, most of the context-filling was tool output, and most of that was tools returning more than the agent needed. Chapter 11 is the SWE-agent lesson applied: tool design for a non-human reader. Viewport reads. Line-range edits. Explicit truncation envelopes.",[3024,3025,3026],"style",{},"html pre.shiki code .sbgvK, html code.shiki .sbgvK{--shiki-light:#E2931D;--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .s_sjI, html code.shiki .s_sjI{--shiki-light:#91B859;--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sjJ54, html code.shiki .sjJ54{--shiki-light:#39ADB5;--shiki-default:#032F62;--shiki-dark:#9ECBFF}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sutJx, html code.shiki .sutJx{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#6A737D;--shiki-default-font-style:inherit;--shiki-dark:#6A737D;--shiki-dark-font-style:inherit}html pre.shiki code .sVHd0, html code.shiki .sVHd0{--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit}html pre.shiki code .s_hVV, html code.shiki .s_hVV{--shiki-light:#90A4AE;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .su5hD, html code.shiki .su5hD{--shiki-light:#90A4AE;--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sbsja, html code.shiki .sbsja{--shiki-light:#9C3EDA;--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sGLFI, html code.shiki .sGLFI{--shiki-light:#6182B8;--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sP7_E, html code.shiki .sP7_E{--shiki-light:#39ADB5;--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sFwrP, html code.shiki .sFwrP{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit}html pre.shiki code .sZMiF, html code.shiki .sZMiF{--shiki-light:#E2931D;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .slqww, html code.shiki .slqww{--shiki-light:#6182B8;--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .stzsN, html code.shiki .stzsN{--shiki-light:#91B859;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .smGrS, html code.shiki .smGrS{--shiki-light:#39ADB5;--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .stp6e, html code.shiki .stp6e{--shiki-light:#39ADB5;--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .s2W-s, html code.shiki .s2W-s{--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#032F62;--shiki-default-font-style:inherit;--shiki-dark:#9ECBFF;--shiki-dark-font-style:inherit}html pre.shiki code .sithA, html code.shiki .sithA{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#032F62;--shiki-default-font-style:inherit;--shiki-dark:#9ECBFF;--shiki-dark-font-style:inherit}html pre.shiki code .sptTA, html code.shiki .sptTA{--shiki-light:#6182B8;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .smCYv, html code.shiki .smCYv{--shiki-light:#E53935;--shiki-light-font-style:italic;--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit}html pre.shiki code .srdBf, html code.shiki .srdBf{--shiki-light:#F76D47;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .s39Yj, html code.shiki .s39Yj{--shiki-light:#39ADB5;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .skxfh, html code.shiki .skxfh{--shiki-light:#E53935;--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .s99_P, html code.shiki .s99_P{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit}",{"title":283,"searchDepth":324,"depth":324,"links":3028},[3029,3030,3031,3032,3033,3034,3035,3036,3037],{"id":239,"depth":324,"text":240},{"id":272,"depth":324,"text":273},{"id":1539,"depth":324,"text":1540},{"id":2223,"depth":324,"text":2224},{"id":2263,"depth":324,"text":2264},{"id":2847,"depth":324,"text":2848},{"id":2880,"depth":324,"text":2881},{"id":2923,"depth":324,"text":2924},{"id":2971,"depth":324,"text":2972},"md",{},null,{"title":50,"description":117},"ojcjnZXMuNdZlJltIudZJwVlxyfm3oMfRQRQEF3ZqM4",[3044,3046],{"title":46,"path":47,"stem":48,"description":3045,"children":-1},"Previously: compaction. Older tool results get masked; the prefix gets summarized when masking isn't enough. The transcript survives long sessions. But anything the agent wanted to keep verbatim is at the mercy of the compactor.",{"title":54,"path":55,"stem":56,"description":3047,"children":-1},"Previously: context-engineering pillars are in place — accounting, compaction, scratchpad, retrieval. What's left is the source of most of the context pressure we've been managing: tools that return too much because they were designed for humans, not models.",1776848984232]