[{"data":1,"prerenderedAt":3519},["ShallowReactive",2],{"navigation":3,"page-\u002Fchapters\u002Fobservability":102,"surround-\u002Fchapters\u002Fobservability":3514},[4,8],{"title":5,"path":6,"stem":7},"Home","\u002F","index",{"title":9,"path":10,"stem":11,"children":12,"page":101},"Chapters","\u002Fchapters","2.chapters",[13,17,21,25,29,33,37,41,45,49,53,57,61,65,69,73,77,81,85,89,93,97],{"title":14,"path":15,"stem":16},"Chapter 1. What an Agent Actually Is","\u002Fchapters\u002Fwhat-an-agent-actually-is","2.chapters\u002F01.what-an-agent-actually-is",{"title":18,"path":19,"stem":20},"Chapter 2. The Minimum Viable Loop","\u002Fchapters\u002Fminimum-viable-loop","2.chapters\u002F02.minimum-viable-loop",{"title":22,"path":23,"stem":24},"Chapter 3. Messages, Turns, and the Transcript","\u002Fchapters\u002Fmessages-turns-transcript","2.chapters\u002F03.messages-turns-transcript",{"title":26,"path":27,"stem":28},"Chapter 4. The Tool Protocol","\u002Fchapters\u002Ftool-protocol","2.chapters\u002F04.tool-protocol",{"title":30,"path":31,"stem":32},"Chapter 5. Streaming, Interruption, and Error Handling","\u002Fchapters\u002Fstreaming-interruption-errors","2.chapters\u002F05.streaming-interruption-errors",{"title":34,"path":35,"stem":36},"Chapter 6. Safe Tool Execution","\u002Fchapters\u002Fsafe-tool-execution","2.chapters\u002F06.safe-tool-execution",{"title":38,"path":39,"stem":40},"Chapter 7. The Context Window Is a Resource","\u002Fchapters\u002Fcontext-window-as-resource","2.chapters\u002F07.context-window-as-resource",{"title":42,"path":43,"stem":44},"Chapter 8. Compaction","\u002Fchapters\u002Fcompaction","2.chapters\u002F08.compaction",{"title":46,"path":47,"stem":48},"Chapter 9. External State: The Scratchpad","\u002Fchapters\u002Fscratchpad","2.chapters\u002F09.scratchpad",{"title":50,"path":51,"stem":52},"Chapter 10. Retrieval","\u002Fchapters\u002Fretrieval","2.chapters\u002F10.retrieval",{"title":54,"path":55,"stem":56},"Chapter 11. Designing Tools Models Can Actually Use","\u002Fchapters\u002Fdesigning-tools","2.chapters\u002F11.designing-tools",{"title":58,"path":59,"stem":60},"Chapter 12. The Tool Cliff and Dynamic Loading","\u002Fchapters\u002Ftool-cliff","2.chapters\u002F12.tool-cliff",{"title":62,"path":63,"stem":64},"Chapter 13. MCP: Tools From the Outside World","\u002Fchapters\u002Fmcp","2.chapters\u002F13.mcp",{"title":66,"path":67,"stem":68},"Chapter 14. Sandboxing and Permissions","\u002Fchapters\u002Fsandboxing-permissions","2.chapters\u002F14.sandboxing-permissions",{"title":70,"path":71,"stem":72},"Chapter 15. Sub-agents","\u002Fchapters\u002Fsub-agents","2.chapters\u002F15.sub-agents",{"title":74,"path":75,"stem":76},"Chapter 16. Structured Plans and Verified Completion","\u002Fchapters\u002Fplans-verified-completion","2.chapters\u002F16.plans-verified-completion",{"title":78,"path":79,"stem":80},"Chapter 17. Parallelism and Shared State","\u002Fchapters\u002Fparallelism-shared-state","2.chapters\u002F17.parallelism-shared-state",{"title":82,"path":83,"stem":84},"Chapter 18. Observability","\u002Fchapters\u002Fobservability","2.chapters\u002F18.observability",{"title":86,"path":87,"stem":88},"Chapter 19. Evals","\u002Fchapters\u002Fevals","2.chapters\u002F19.evals",{"title":90,"path":91,"stem":92},"Chapter 20. Cost Control","\u002Fchapters\u002Fcost-control","2.chapters\u002F20.cost-control",{"title":94,"path":95,"stem":96},"Chapter 21. Resumability and Durable State","\u002Fchapters\u002Fresumability","2.chapters\u002F21.resumability",{"title":98,"path":99,"stem":100},"Chapter 22. What Transfers, Where to Go","\u002Fchapters\u002Fwhat-transfers","2.chapters\u002F22.what-transfers",false,{"id":103,"title":82,"body":104,"description":117,"extension":3509,"meta":3510,"navigation":3511,"path":83,"seo":3512,"stem":84,"__hash__":3513},"content\u002F2.chapters\u002F18.observability.md",{"type":105,"value":106,"toc":3498},"minimark",[107,111,118,125,128,161,164,265,268,273,276,283,292,298,301,347,349,353,360,1500,1503,1525,1538,1547,1549,1553,1556,2507,2521,2523,2527,2537,2754,2772,2974,2983,2985,2989,2992,3029,3032,3039,3042,3190,3193,3195,3199,3202,3254,3257,3259,3263,3274,3283,3389,3395,3397,3401,3446,3450,3479,3481,3494],[108,109,82],"h1",{"id":110},"chapter-18-observability",[112,113,114],"p",{},[115,116,117],"em",{},"Previously: parallel sub-agents, leases, grounded verification. The harness is capable but opaque. A failed run tells you the final error but nothing about which sub-agent burned tokens, which tool call took 12 seconds, which compaction event dropped what the final agent wanted.",[112,119,120,121,124],{},"Observability for agents is not the same shape as observability for typical web services. Request\u002Fresponse latency matters, but so does the entire trajectory: which tools fired in which order, how much each cost, when compaction ran, which sub-agents spawned, what the model's output was at each turn. A failed run needs a ",[115,122,123],{},"timeline",", not a metric.",[112,126,127],{},"This chapter adds OpenTelemetry-based instrumentation to the harness. Three things by the end:",[129,130,131,135,147],"ol",{},[132,133,134],"li",{},"Every LLM call, tool call, compaction event, and sub-agent spawn is a span in a trace.",[132,136,137,138,142,143,146],{},"Spans carry the OpenTelemetry GenAI semantic conventions — ",[139,140,141],"code",{},"gen_ai.system",", ",[139,144,145],{},"gen_ai.usage.input_tokens",", etc.",[132,148,149,150,142,153,156,157,160],{},"Every span is tagged with ",[139,151,152],{},"session_id",[139,154,155],{},"task_id",", and ",[139,158,159],{},"agent_id"," so per-agent cost attribution is possible.",[112,162,163],{},"The GenAI semantic conventions are still marked experimental by OpenTelemetry as of April 2026, but they're stable enough to build on — the major observability platforms (Datadog, Langfuse, Braintrust) have adopted them.",[165,166,170,257],"figure",{"className":167},[168,169],"not-prose","my-8",[171,172,181],"div",{"className":173},[174,175,176,177,178,179,180],"rounded-lg","border","border-default","bg-elevated","p-3","font-mono","text-xs",[171,182,188,194,195,200,213,246],{"className":183},[184,185,186,187],"border-l-2","border-primary","pl-3","py-1",[189,190,193],"span",{"className":191},[192],"text-primary","agent.run"," ",[189,196,199],{"className":197},[198],"text-muted","session_id=s_42 task_id=t_07 agent_id=root",[171,201,204,194,209],{"className":202},[184,176,186,187,203],"mt-1",[189,205,208],{"className":206},[207],"text-default","agent.turn #1",[189,210,212],{"className":211},[198],"42ms",[171,214,216,194,220,224,235],{"className":215},[184,176,186,187,203],[189,217,219],{"className":218},[207],"agent.turn #2",[189,221,223],{"className":222},[198],"1.8s",[171,225,227,194,231],{"className":226},[184,176,186,187,203],[189,228,230],{"className":229},[207],"gen_ai.completion",[189,232,234],{"className":233},[198],"input=2400 output=180",[171,236,238,194,242],{"className":237},[184,176,186,187,203],[189,239,241],{"className":240},[207],"tool.call",[189,243,245],{"className":244},[198],"read_file 12ms",[171,247,249,194,253],{"className":248},[184,176,186,187,203],[189,250,252],{"className":251},[207],"agent.turn #3",[189,254,256],{"className":255},[198],"0.9s",[258,259,264],"figcaption",{"className":260},[180,198,261,262,263],"mt-3","text-center","italic","A trace tree: nested spans for run → turns → completion + tool calls. Every span carries session_id \u002F task_id \u002F agent_id.",[266,267],"hr",{},[269,270,272],"h2",{"id":271},"_181-why-opentelemetry-and-not-just-logging","18.1 Why OpenTelemetry and Not Just Logging",[112,274,275],{},"Three reasons to pick OTel over ad-hoc logs, and one piece of foundational context before the reasons: distributed tracing as a discipline grew out of Sigelman et al.'s 2010 paper \"Dapper, a Large-Scale Distributed Systems Tracing Infrastructure,\" which documented the internal Google system that correlated billions of RPCs per day across thousands of microservices by passing a trace context through every call. Every modern tracing system — Zipkin, Jaeger, OpenTelemetry itself — is a descendant of Dapper's design. The GenAI semantic conventions this chapter uses are the agent-specific attributes bolted on top of the same underlying trace-and-span model.",[112,277,278,282],{},[279,280,281],"strong",{},"Traces, not log lines."," A trace with spans preserves the parent-child relationship between operations. You see \"this LLM call happened inside this turn, which happened inside this sub-agent, which happened inside the parent task.\" With flat logs you reconstruct this by grepping; with traces it's free.",[112,284,285,288,289,291],{},[279,286,287],{},"Standardized attributes."," When you tag ",[139,290,145],{}," consistently, every observability platform knows what it means. Your dashboards, alerts, and downstream cost analysis don't have to speak your local vocabulary.",[112,293,294,297],{},[279,295,296],{},"Exporters are pluggable."," Console for development, Jaeger for local dev, Langfuse\u002FDatadog\u002FHoneycomb for production. The harness code doesn't change; the exporter does.",[112,299,300],{},"Add the dependencies:",[302,303,308],"pre",{"className":304,"code":305,"language":306,"meta":307,"style":307},"language-bash shiki shiki-themes material-theme-lighter github-light github-dark","uv add 'opentelemetry-api>=1.27' 'opentelemetry-sdk>=1.27' 'opentelemetry-exporter-otlp>=1.27'\n","bash","",[139,309,310],{"__ignoreMap":307},[189,311,314,318,322,326,329,332,334,337,339,341,344],{"class":312,"line":313},"line",1,[189,315,317],{"class":316},"sbgvK","uv",[189,319,321],{"class":320},"s_sjI"," add",[189,323,325],{"class":324},"sjJ54"," '",[189,327,328],{"class":320},"opentelemetry-api>=1.27",[189,330,331],{"class":324},"'",[189,333,325],{"class":324},[189,335,336],{"class":320},"opentelemetry-sdk>=1.27",[189,338,331],{"class":324},[189,340,325],{"class":324},[189,342,343],{"class":320},"opentelemetry-exporter-otlp>=1.27",[189,345,346],{"class":324},"'\n",[266,348],{},[269,350,352],{"id":351},"_182-the-instrumentation-layer","18.2 The Instrumentation Layer",[112,354,355,356,359],{},"A thin wrapper that gives the rest of the harness a stable interface. We don't scatter ",[139,357,358],{},"tracer.start_as_current_span(...)"," through every module; we put it behind a small API.",[302,361,365],{"className":362,"code":363,"language":364,"meta":307,"style":307},"language-python shiki shiki-themes material-theme-lighter github-light github-dark","# src\u002Fharness\u002Fobservability\u002Ftracing.py\nfrom __future__ import annotations\n\nfrom contextlib import contextmanager\nfrom dataclasses import dataclass, field\nfrom typing import Iterator\nfrom uuid import uuid4\n\nfrom opentelemetry import trace\nfrom opentelemetry.sdk.resources import Resource\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import (\n BatchSpanProcessor, ConsoleSpanExporter, SpanProcessor,\n)\nfrom opentelemetry.trace import Tracer, Status, StatusCode\n\n\n_provider: TracerProvider | None = None\n\n\ndef setup_tracing(\n service_name: str = \"agent-harness\",\n exporter: SpanProcessor | None = None,\n) -> None:\n \"\"\"Initialize OTel once per process.\"\"\"\n global _provider\n if _provider is not None:\n return\n resource = Resource.create({\"service.name\": service_name})\n _provider = TracerProvider(resource=resource)\n _provider.add_span_processor(\n exporter or BatchSpanProcessor(ConsoleSpanExporter())\n )\n trace.set_tracer_provider(_provider)\n\n\n@dataclass\nclass SessionContext:\n session_id: str = field(default_factory=lambda: str(uuid4()))\n task_id: str = field(default_factory=lambda: str(uuid4()))\n agent_id: str = \"root\"\n\n def subagent(self, agent_id: str) -> \"SessionContext\":\n return SessionContext(\n session_id=self.session_id,\n task_id=self.task_id,\n agent_id=agent_id,\n )\n\n\ndef tracer() -> Tracer:\n return trace.get_tracer(\"harness\")\n\n\n@contextmanager\ndef span(name: str, ctx: SessionContext, **attrs) -> Iterator[trace.Span]:\n t = tracer()\n with t.start_as_current_span(name) as s:\n s.set_attribute(\"harness.session_id\", ctx.session_id)\n s.set_attribute(\"harness.task_id\", ctx.task_id)\n s.set_attribute(\"harness.agent_id\", ctx.agent_id)\n for k, v in attrs.items():\n s.set_attribute(k, v)\n try:\n yield s\n except Exception as e:\n s.set_status(Status(StatusCode.ERROR, str(e)))\n s.record_exception(e)\n raise\n","python",[139,366,367,373,391,398,412,432,445,458,463,476,500,521,547,566,572,598,603,608,634,639,644,658,684,705,719,733,742,761,767,803,827,840,860,866,883,888,893,903,914,951,983,1003,1008,1045,1055,1072,1088,1100,1106,1111,1116,1133,1158,1163,1168,1176,1230,1243,1271,1301,1329,1357,1385,1407,1415,1424,1440,1478,1494],{"__ignoreMap":307},[189,368,369],{"class":312,"line":313},[189,370,372],{"class":371},"sutJx","# src\u002Fharness\u002Fobservability\u002Ftracing.py\n",[189,374,376,380,384,387],{"class":312,"line":375},2,[189,377,379],{"class":378},"sVHd0","from",[189,381,383],{"class":382},"s_hVV"," __future__",[189,385,386],{"class":378}," import",[189,388,390],{"class":389},"su5hD"," annotations\n",[189,392,394],{"class":312,"line":393},3,[189,395,397],{"emptyLinePlaceholder":396},true,"\n",[189,399,401,403,406,409],{"class":312,"line":400},4,[189,402,379],{"class":378},[189,404,405],{"class":389}," contextlib ",[189,407,408],{"class":378},"import",[189,410,411],{"class":389}," contextmanager\n",[189,413,415,417,420,422,425,429],{"class":312,"line":414},5,[189,416,379],{"class":378},[189,418,419],{"class":389}," dataclasses ",[189,421,408],{"class":378},[189,423,424],{"class":389}," dataclass",[189,426,428],{"class":427},"sP7_E",",",[189,430,431],{"class":389}," field\n",[189,433,435,437,440,442],{"class":312,"line":434},6,[189,436,379],{"class":378},[189,438,439],{"class":389}," typing ",[189,441,408],{"class":378},[189,443,444],{"class":389}," Iterator\n",[189,446,448,450,453,455],{"class":312,"line":447},7,[189,449,379],{"class":378},[189,451,452],{"class":389}," uuid ",[189,454,408],{"class":378},[189,456,457],{"class":389}," uuid4\n",[189,459,461],{"class":312,"line":460},8,[189,462,397],{"emptyLinePlaceholder":396},[189,464,466,468,471,473],{"class":312,"line":465},9,[189,467,379],{"class":378},[189,469,470],{"class":389}," opentelemetry ",[189,472,408],{"class":378},[189,474,475],{"class":389}," trace\n",[189,477,479,481,484,487,490,492,495,497],{"class":312,"line":478},10,[189,480,379],{"class":378},[189,482,483],{"class":389}," opentelemetry",[189,485,486],{"class":427},".",[189,488,489],{"class":389},"sdk",[189,491,486],{"class":427},[189,493,494],{"class":389},"resources ",[189,496,408],{"class":378},[189,498,499],{"class":389}," Resource\n",[189,501,503,505,507,509,511,513,516,518],{"class":312,"line":502},11,[189,504,379],{"class":378},[189,506,483],{"class":389},[189,508,486],{"class":427},[189,510,489],{"class":389},[189,512,486],{"class":427},[189,514,515],{"class":389},"trace ",[189,517,408],{"class":378},[189,519,520],{"class":389}," TracerProvider\n",[189,522,524,526,528,530,532,534,537,539,542,544],{"class":312,"line":523},12,[189,525,379],{"class":378},[189,527,483],{"class":389},[189,529,486],{"class":427},[189,531,489],{"class":389},[189,533,486],{"class":427},[189,535,536],{"class":389},"trace",[189,538,486],{"class":427},[189,540,541],{"class":389},"export ",[189,543,408],{"class":378},[189,545,546],{"class":427}," (\n",[189,548,550,553,555,558,560,563],{"class":312,"line":549},13,[189,551,552],{"class":389}," BatchSpanProcessor",[189,554,428],{"class":427},[189,556,557],{"class":389}," ConsoleSpanExporter",[189,559,428],{"class":427},[189,561,562],{"class":389}," SpanProcessor",[189,564,565],{"class":427},",\n",[189,567,569],{"class":312,"line":568},14,[189,570,571],{"class":427},")\n",[189,573,575,577,579,581,583,585,588,590,593,595],{"class":312,"line":574},15,[189,576,379],{"class":378},[189,578,483],{"class":389},[189,580,486],{"class":427},[189,582,515],{"class":389},[189,584,408],{"class":378},[189,586,587],{"class":389}," Tracer",[189,589,428],{"class":427},[189,591,592],{"class":389}," Status",[189,594,428],{"class":427},[189,596,597],{"class":389}," StatusCode\n",[189,599,601],{"class":312,"line":600},16,[189,602,397],{"emptyLinePlaceholder":396},[189,604,606],{"class":312,"line":605},17,[189,607,397],{"emptyLinePlaceholder":396},[189,609,611,614,617,620,624,628,631],{"class":312,"line":610},18,[189,612,613],{"class":389},"_provider",[189,615,616],{"class":427},":",[189,618,619],{"class":389}," TracerProvider ",[189,621,623],{"class":622},"smGrS","|",[189,625,627],{"class":626},"s39Yj"," None",[189,629,630],{"class":622}," =",[189,632,633],{"class":626}," None\n",[189,635,637],{"class":312,"line":636},19,[189,638,397],{"emptyLinePlaceholder":396},[189,640,642],{"class":312,"line":641},20,[189,643,397],{"emptyLinePlaceholder":396},[189,645,647,651,655],{"class":312,"line":646},21,[189,648,650],{"class":649},"sbsja","def",[189,652,654],{"class":653},"sGLFI"," setup_tracing",[189,656,657],{"class":427},"(\n",[189,659,661,665,667,671,673,676,679,682],{"class":312,"line":660},22,[189,662,664],{"class":663},"sFwrP"," service_name",[189,666,616],{"class":427},[189,668,670],{"class":669},"sZMiF"," str",[189,672,630],{"class":622},[189,674,675],{"class":324}," \"",[189,677,678],{"class":320},"agent-harness",[189,680,681],{"class":324},"\"",[189,683,565],{"class":427},[189,685,687,690,692,695,697,699,701,703],{"class":312,"line":686},23,[189,688,689],{"class":663}," exporter",[189,691,616],{"class":427},[189,693,694],{"class":389}," SpanProcessor ",[189,696,623],{"class":622},[189,698,627],{"class":626},[189,700,630],{"class":622},[189,702,627],{"class":626},[189,704,565],{"class":427},[189,706,708,711,714,716],{"class":312,"line":707},24,[189,709,710],{"class":427},")",[189,712,713],{"class":427}," ->",[189,715,627],{"class":626},[189,717,718],{"class":427},":\n",[189,720,722,726,730],{"class":312,"line":721},25,[189,723,725],{"class":724},"s2W-s"," \"\"\"",[189,727,729],{"class":728},"sithA","Initialize OTel once per process.",[189,731,732],{"class":724},"\"\"\"\n",[189,734,736,739],{"class":312,"line":735},26,[189,737,738],{"class":649}," global",[189,740,741],{"class":389}," _provider\n",[189,743,745,748,751,754,757,759],{"class":312,"line":744},27,[189,746,747],{"class":378}," if",[189,749,750],{"class":389}," _provider ",[189,752,753],{"class":622},"is",[189,755,756],{"class":622}," not",[189,758,627],{"class":626},[189,760,718],{"class":427},[189,762,764],{"class":312,"line":763},28,[189,765,766],{"class":378}," return\n",[189,768,770,773,776,779,781,785,788,790,793,795,797,800],{"class":312,"line":769},29,[189,771,772],{"class":389}," resource ",[189,774,775],{"class":622},"=",[189,777,778],{"class":389}," Resource",[189,780,486],{"class":427},[189,782,784],{"class":783},"slqww","create",[189,786,787],{"class":427},"({",[189,789,681],{"class":324},[189,791,792],{"class":320},"service.name",[189,794,681],{"class":324},[189,796,616],{"class":427},[189,798,799],{"class":783}," service_name",[189,801,802],{"class":427},"})\n",[189,804,806,809,811,814,817,821,823,825],{"class":312,"line":805},30,[189,807,808],{"class":389}," _provider ",[189,810,775],{"class":622},[189,812,813],{"class":783}," TracerProvider",[189,815,816],{"class":427},"(",[189,818,820],{"class":819},"s99_P","resource",[189,822,775],{"class":622},[189,824,820],{"class":783},[189,826,571],{"class":427},[189,828,830,833,835,838],{"class":312,"line":829},31,[189,831,832],{"class":389}," _provider",[189,834,486],{"class":427},[189,836,837],{"class":783},"add_span_processor",[189,839,657],{"class":427},[189,841,843,846,849,852,854,857],{"class":312,"line":842},32,[189,844,845],{"class":783}," exporter ",[189,847,848],{"class":378},"or",[189,850,851],{"class":783}," BatchSpanProcessor",[189,853,816],{"class":427},[189,855,856],{"class":783},"ConsoleSpanExporter",[189,858,859],{"class":427},"())\n",[189,861,863],{"class":312,"line":862},33,[189,864,865],{"class":427}," )\n",[189,867,869,872,874,877,879,881],{"class":312,"line":868},34,[189,870,871],{"class":389}," trace",[189,873,486],{"class":427},[189,875,876],{"class":783},"set_tracer_provider",[189,878,816],{"class":427},[189,880,613],{"class":783},[189,882,571],{"class":427},[189,884,886],{"class":312,"line":885},35,[189,887,397],{"emptyLinePlaceholder":396},[189,889,891],{"class":312,"line":890},36,[189,892,397],{"emptyLinePlaceholder":396},[189,894,896,900],{"class":312,"line":895},37,[189,897,899],{"class":898},"stp6e","@",[189,901,902],{"class":653},"dataclass\n",[189,904,906,909,912],{"class":312,"line":905},38,[189,907,908],{"class":649},"class",[189,910,911],{"class":316}," SessionContext",[189,913,718],{"class":427},[189,915,917,920,922,924,926,929,931,934,936,939,941,943,945,948],{"class":312,"line":916},39,[189,918,919],{"class":389}," session_id",[189,921,616],{"class":427},[189,923,670],{"class":669},[189,925,630],{"class":622},[189,927,928],{"class":783}," field",[189,930,816],{"class":427},[189,932,933],{"class":819},"default_factory",[189,935,775],{"class":622},[189,937,938],{"class":649},"lambda",[189,940,616],{"class":427},[189,942,670],{"class":669},[189,944,816],{"class":427},[189,946,947],{"class":783},"uuid4",[189,949,950],{"class":427},"()))\n",[189,952,954,957,959,961,963,965,967,969,971,973,975,977,979,981],{"class":312,"line":953},40,[189,955,956],{"class":389}," task_id",[189,958,616],{"class":427},[189,960,670],{"class":669},[189,962,630],{"class":622},[189,964,928],{"class":783},[189,966,816],{"class":427},[189,968,933],{"class":819},[189,970,775],{"class":622},[189,972,938],{"class":649},[189,974,616],{"class":427},[189,976,670],{"class":669},[189,978,816],{"class":427},[189,980,947],{"class":783},[189,982,950],{"class":427},[189,984,986,989,991,993,995,997,1000],{"class":312,"line":985},41,[189,987,988],{"class":389}," agent_id",[189,990,616],{"class":427},[189,992,670],{"class":669},[189,994,630],{"class":622},[189,996,675],{"class":324},[189,998,999],{"class":320},"root",[189,1001,1002],{"class":324},"\"\n",[189,1004,1006],{"class":312,"line":1005},42,[189,1007,397],{"emptyLinePlaceholder":396},[189,1009,1011,1014,1017,1019,1023,1025,1028,1030,1032,1034,1036,1038,1041,1043],{"class":312,"line":1010},43,[189,1012,1013],{"class":649}," def",[189,1015,1016],{"class":653}," subagent",[189,1018,816],{"class":427},[189,1020,1022],{"class":1021},"smCYv","self",[189,1024,428],{"class":427},[189,1026,1027],{"class":663}," agent_id",[189,1029,616],{"class":427},[189,1031,670],{"class":669},[189,1033,710],{"class":427},[189,1035,713],{"class":427},[189,1037,675],{"class":324},[189,1039,1040],{"class":320},"SessionContext",[189,1042,681],{"class":324},[189,1044,718],{"class":427},[189,1046,1048,1051,1053],{"class":312,"line":1047},44,[189,1049,1050],{"class":378}," return",[189,1052,911],{"class":783},[189,1054,657],{"class":427},[189,1056,1058,1061,1063,1065,1067,1070],{"class":312,"line":1057},45,[189,1059,1060],{"class":819}," session_id",[189,1062,775],{"class":622},[189,1064,1022],{"class":382},[189,1066,486],{"class":427},[189,1068,152],{"class":1069},"skxfh",[189,1071,565],{"class":427},[189,1073,1075,1078,1080,1082,1084,1086],{"class":312,"line":1074},46,[189,1076,1077],{"class":819}," task_id",[189,1079,775],{"class":622},[189,1081,1022],{"class":382},[189,1083,486],{"class":427},[189,1085,155],{"class":1069},[189,1087,565],{"class":427},[189,1089,1091,1094,1096,1098],{"class":312,"line":1090},47,[189,1092,1093],{"class":819}," agent_id",[189,1095,775],{"class":622},[189,1097,159],{"class":783},[189,1099,565],{"class":427},[189,1101,1103],{"class":312,"line":1102},48,[189,1104,1105],{"class":427}," )\n",[189,1107,1109],{"class":312,"line":1108},49,[189,1110,397],{"emptyLinePlaceholder":396},[189,1112,1114],{"class":312,"line":1113},50,[189,1115,397],{"emptyLinePlaceholder":396},[189,1117,1119,1121,1124,1127,1129,1131],{"class":312,"line":1118},51,[189,1120,650],{"class":649},[189,1122,1123],{"class":653}," tracer",[189,1125,1126],{"class":427},"()",[189,1128,713],{"class":427},[189,1130,587],{"class":389},[189,1132,718],{"class":427},[189,1134,1136,1139,1142,1144,1147,1149,1151,1154,1156],{"class":312,"line":1135},52,[189,1137,1138],{"class":378}," return",[189,1140,1141],{"class":389}," trace",[189,1143,486],{"class":427},[189,1145,1146],{"class":783},"get_tracer",[189,1148,816],{"class":427},[189,1150,681],{"class":324},[189,1152,1153],{"class":320},"harness",[189,1155,681],{"class":324},[189,1157,571],{"class":427},[189,1159,1161],{"class":312,"line":1160},53,[189,1162,397],{"emptyLinePlaceholder":396},[189,1164,1166],{"class":312,"line":1165},54,[189,1167,397],{"emptyLinePlaceholder":396},[189,1169,1171,1173],{"class":312,"line":1170},55,[189,1172,899],{"class":898},[189,1174,1175],{"class":653},"contextmanager\n",[189,1177,1179,1181,1184,1186,1189,1191,1193,1195,1198,1200,1202,1204,1207,1210,1212,1214,1217,1220,1222,1224,1227],{"class":312,"line":1178},56,[189,1180,650],{"class":649},[189,1182,1183],{"class":653}," span",[189,1185,816],{"class":427},[189,1187,1188],{"class":663},"name",[189,1190,616],{"class":427},[189,1192,670],{"class":669},[189,1194,428],{"class":427},[189,1196,1197],{"class":663}," ctx",[189,1199,616],{"class":427},[189,1201,911],{"class":389},[189,1203,428],{"class":427},[189,1205,1206],{"class":622}," **",[189,1208,1209],{"class":663},"attrs",[189,1211,710],{"class":427},[189,1213,713],{"class":427},[189,1215,1216],{"class":389}," Iterator",[189,1218,1219],{"class":427},"[",[189,1221,536],{"class":389},[189,1223,486],{"class":427},[189,1225,1226],{"class":1069},"Span",[189,1228,1229],{"class":427},"]:\n",[189,1231,1233,1236,1238,1240],{"class":312,"line":1232},57,[189,1234,1235],{"class":389}," t ",[189,1237,775],{"class":622},[189,1239,1123],{"class":783},[189,1241,1242],{"class":427},"()\n",[189,1244,1246,1249,1252,1254,1257,1259,1261,1263,1266,1269],{"class":312,"line":1245},58,[189,1247,1248],{"class":378}," with",[189,1250,1251],{"class":389}," t",[189,1253,486],{"class":427},[189,1255,1256],{"class":783},"start_as_current_span",[189,1258,816],{"class":427},[189,1260,1188],{"class":783},[189,1262,710],{"class":427},[189,1264,1265],{"class":378}," as",[189,1267,1268],{"class":389}," s",[189,1270,718],{"class":427},[189,1272,1274,1277,1279,1282,1284,1286,1289,1291,1293,1295,1297,1299],{"class":312,"line":1273},59,[189,1275,1276],{"class":389}," s",[189,1278,486],{"class":427},[189,1280,1281],{"class":783},"set_attribute",[189,1283,816],{"class":427},[189,1285,681],{"class":324},[189,1287,1288],{"class":320},"harness.session_id",[189,1290,681],{"class":324},[189,1292,428],{"class":427},[189,1294,1197],{"class":783},[189,1296,486],{"class":427},[189,1298,152],{"class":1069},[189,1300,571],{"class":427},[189,1302,1304,1306,1308,1310,1312,1314,1317,1319,1321,1323,1325,1327],{"class":312,"line":1303},60,[189,1305,1276],{"class":389},[189,1307,486],{"class":427},[189,1309,1281],{"class":783},[189,1311,816],{"class":427},[189,1313,681],{"class":324},[189,1315,1316],{"class":320},"harness.task_id",[189,1318,681],{"class":324},[189,1320,428],{"class":427},[189,1322,1197],{"class":783},[189,1324,486],{"class":427},[189,1326,155],{"class":1069},[189,1328,571],{"class":427},[189,1330,1332,1334,1336,1338,1340,1342,1345,1347,1349,1351,1353,1355],{"class":312,"line":1331},61,[189,1333,1276],{"class":389},[189,1335,486],{"class":427},[189,1337,1281],{"class":783},[189,1339,816],{"class":427},[189,1341,681],{"class":324},[189,1343,1344],{"class":320},"harness.agent_id",[189,1346,681],{"class":324},[189,1348,428],{"class":427},[189,1350,1197],{"class":783},[189,1352,486],{"class":427},[189,1354,159],{"class":1069},[189,1356,571],{"class":427},[189,1358,1360,1363,1366,1368,1371,1374,1377,1379,1382],{"class":312,"line":1359},62,[189,1361,1362],{"class":378}," for",[189,1364,1365],{"class":389}," k",[189,1367,428],{"class":427},[189,1369,1370],{"class":389}," v ",[189,1372,1373],{"class":378},"in",[189,1375,1376],{"class":389}," attrs",[189,1378,486],{"class":427},[189,1380,1381],{"class":783},"items",[189,1383,1384],{"class":427},"():\n",[189,1386,1388,1391,1393,1395,1397,1400,1402,1405],{"class":312,"line":1387},63,[189,1389,1390],{"class":389}," s",[189,1392,486],{"class":427},[189,1394,1281],{"class":783},[189,1396,816],{"class":427},[189,1398,1399],{"class":783},"k",[189,1401,428],{"class":427},[189,1403,1404],{"class":783}," v",[189,1406,571],{"class":427},[189,1408,1410,1413],{"class":312,"line":1409},64,[189,1411,1412],{"class":378}," try",[189,1414,718],{"class":427},[189,1416,1418,1421],{"class":312,"line":1417},65,[189,1419,1420],{"class":378}," yield",[189,1422,1423],{"class":389}," s\n",[189,1425,1427,1430,1433,1435,1438],{"class":312,"line":1426},66,[189,1428,1429],{"class":378}," except",[189,1431,1432],{"class":669}," Exception",[189,1434,1265],{"class":378},[189,1436,1437],{"class":389}," e",[189,1439,718],{"class":427},[189,1441,1443,1445,1447,1450,1452,1455,1457,1460,1462,1466,1468,1470,1472,1475],{"class":312,"line":1442},67,[189,1444,1390],{"class":389},[189,1446,486],{"class":427},[189,1448,1449],{"class":783},"set_status",[189,1451,816],{"class":427},[189,1453,1454],{"class":783},"Status",[189,1456,816],{"class":427},[189,1458,1459],{"class":783},"StatusCode",[189,1461,486],{"class":427},[189,1463,1465],{"class":1464},"swQdS","ERROR",[189,1467,428],{"class":427},[189,1469,670],{"class":669},[189,1471,816],{"class":427},[189,1473,1474],{"class":783},"e",[189,1476,1477],{"class":427},")))\n",[189,1479,1481,1483,1485,1488,1490,1492],{"class":312,"line":1480},68,[189,1482,1390],{"class":389},[189,1484,486],{"class":427},[189,1486,1487],{"class":783},"record_exception",[189,1489,816],{"class":427},[189,1491,1474],{"class":783},[189,1493,571],{"class":427},[189,1495,1497],{"class":312,"line":1496},69,[189,1498,1499],{"class":378}," raise\n",[112,1501,1502],{},"Three design points.",[112,1504,1505,1510,1511,1513,1514,1516,1517,1519,1520,1522,1523,486],{},[279,1506,1507,1509],{},[139,1508,1040],{}," as the correlation anchor."," Every span in a session shares its ",[139,1512,152],{}," and ",[139,1515,155],{},". Sub-agents have a different ",[139,1518,159],{}," but inherit session and task. This is how you produce \"per-agent cost attribution\" — downstream, you group by ",[139,1521,159],{}," over fixed ",[139,1524,152],{},[112,1526,1527,1533,1534,1537],{},[279,1528,1529,1532],{},[139,1530,1531],{},"span()"," is a context manager."," The pattern every instrumented operation follows: ",[139,1535,1536],{},"with span(\"name\", ctx): ...",". It handles error propagation, attribute setting, and OTel lifecycle consistently.",[112,1539,1540,194,1543,1546],{},[279,1541,1542],{},"One-time setup.",[139,1544,1545],{},"setup_tracing()"," is idempotent; repeated calls are no-ops. This matters when a harness runs inside an already-instrumented process (a CI runner, a web service).",[266,1548],{},[269,1550,1552],{"id":1551},"_183-instrumenting-the-loop","18.3 Instrumenting the Loop",[112,1554,1555],{},"The loop gets three span types, roughly corresponding to the three things happening: the overall run, each turn, and each tool call.",[302,1557,1559],{"className":362,"code":1558,"language":364,"meta":307,"style":307},"# src\u002Fharness\u002Fagent.py (observability-aware version, sketch)\nfrom .observability.tracing import SessionContext, span\n\n\nasync def arun(\n # ... existing parameters (including `transcript: Transcript | None = None`\n # from Chapter 5's chat-continuity upgrade)\n session_context: SessionContext | None = None,\n) -> str:\n ctx = session_context or SessionContext()\n\n with span(\"agent.run\", ctx,\n **{\"harness.initial_user_message_len\": len(user_message)}) as s:\n if transcript is None:\n transcript = Transcript(system=system)\n transcript.append(Message.user_text(user_message))\n # ... existing setup\n\n for iteration in range(MAX_ITERATIONS):\n with span(\"agent.turn\", ctx,\n **{\"harness.iteration\": iteration}) as turn_span:\n # compaction span\n snapshot = accountant.snapshot(transcript, tools=registry.schemas())\n turn_span.set_attribute(\"harness.context_utilization\",\n snapshot.utilization)\n if snapshot.state == \"red\":\n with span(\"agent.compact\", ctx):\n await compactor.compact_if_needed(transcript,\n registry.schemas())\n\n # LLM call span\n with span(\"gen_ai.completion\", ctx,\n **{\"gen_ai.system\": provider.name}) as llm_span:\n response = await _one_turn(provider, registry,\n transcript, on_event)\n llm_span.set_attribute(\n \"gen_ai.usage.input_tokens\", response.input_tokens)\n llm_span.set_attribute(\n \"gen_ai.usage.output_tokens\", response.output_tokens)\n\n if response.is_final:\n s.set_attribute(\"harness.final_iteration\", iteration)\n transcript.append(Message.from_assistant_response(response))\n return response.text or \"\"\n\n # Commit one assistant message with every ToolCall block.\n transcript.append(Message.from_assistant_response(response))\n # One tool.call span per dispatched call (batched turns get N spans).\n for ref in response.tool_calls:\n with span(\"tool.call\", ctx,\n **{\"tool.name\": ref.name}) as tool_span:\n result = await registry.adispatch(ref.name, ref.args, ref.id)\n tool_span.set_attribute(\"tool.is_error\", result.is_error)\n tool_span.set_attribute(\"tool.result_chars\", len(result.content))\n transcript.append(Message.tool_result(result))\n # ...\n",[139,1560,1561,1566,1590,1594,1598,1611,1616,1621,1641,1651,1667,1671,1691,1726,1740,1761,1788,1793,1797,1817,1839,1868,1873,1910,1930,1942,1967,1989,2008,2019,2023,2028,2049,2080,2105,2117,2128,2149,2159,2179,2183,2196,2220,2245,2263,2267,2272,2295,2300,2319,2339,2371,2416,2446,2479,2502],{"__ignoreMap":307},[189,1562,1563],{"class":312,"line":313},[189,1564,1565],{"class":371},"# src\u002Fharness\u002Fagent.py (observability-aware version, sketch)\n",[189,1567,1568,1570,1573,1576,1578,1581,1583,1585,1587],{"class":312,"line":375},[189,1569,379],{"class":378},[189,1571,1572],{"class":427}," .",[189,1574,1575],{"class":389},"observability",[189,1577,486],{"class":427},[189,1579,1580],{"class":389},"tracing ",[189,1582,408],{"class":378},[189,1584,911],{"class":389},[189,1586,428],{"class":427},[189,1588,1589],{"class":389}," span\n",[189,1591,1592],{"class":312,"line":393},[189,1593,397],{"emptyLinePlaceholder":396},[189,1595,1596],{"class":312,"line":400},[189,1597,397],{"emptyLinePlaceholder":396},[189,1599,1600,1603,1606,1609],{"class":312,"line":414},[189,1601,1602],{"class":649},"async",[189,1604,1605],{"class":649}," def",[189,1607,1608],{"class":653}," arun",[189,1610,657],{"class":427},[189,1612,1613],{"class":312,"line":434},[189,1614,1615],{"class":371}," # ... existing parameters (including `transcript: Transcript | None = None`\n",[189,1617,1618],{"class":312,"line":447},[189,1619,1620],{"class":371}," # from Chapter 5's chat-continuity upgrade)\n",[189,1622,1623,1626,1628,1631,1633,1635,1637,1639],{"class":312,"line":460},[189,1624,1625],{"class":663}," session_context",[189,1627,616],{"class":427},[189,1629,1630],{"class":389}," SessionContext ",[189,1632,623],{"class":622},[189,1634,627],{"class":626},[189,1636,630],{"class":622},[189,1638,627],{"class":626},[189,1640,565],{"class":427},[189,1642,1643,1645,1647,1649],{"class":312,"line":465},[189,1644,710],{"class":427},[189,1646,713],{"class":427},[189,1648,670],{"class":669},[189,1650,718],{"class":427},[189,1652,1653,1656,1658,1661,1663,1665],{"class":312,"line":478},[189,1654,1655],{"class":389}," ctx ",[189,1657,775],{"class":622},[189,1659,1660],{"class":389}," session_context ",[189,1662,848],{"class":622},[189,1664,911],{"class":783},[189,1666,1242],{"class":427},[189,1668,1669],{"class":312,"line":502},[189,1670,397],{"emptyLinePlaceholder":396},[189,1672,1673,1675,1677,1679,1681,1683,1685,1687,1689],{"class":312,"line":523},[189,1674,1248],{"class":378},[189,1676,1183],{"class":783},[189,1678,816],{"class":427},[189,1680,681],{"class":324},[189,1682,193],{"class":320},[189,1684,681],{"class":324},[189,1686,428],{"class":427},[189,1688,1197],{"class":783},[189,1690,565],{"class":427},[189,1692,1693,1696,1699,1701,1704,1706,1708,1712,1714,1717,1720,1722,1724],{"class":312,"line":549},[189,1694,1695],{"class":622}," **",[189,1697,1698],{"class":427},"{",[189,1700,681],{"class":324},[189,1702,1703],{"class":320},"harness.initial_user_message_len",[189,1705,681],{"class":324},[189,1707,616],{"class":427},[189,1709,1711],{"class":1710},"sptTA"," len",[189,1713,816],{"class":427},[189,1715,1716],{"class":783},"user_message",[189,1718,1719],{"class":427},")})",[189,1721,1265],{"class":378},[189,1723,1268],{"class":389},[189,1725,718],{"class":427},[189,1727,1728,1731,1734,1736,1738],{"class":312,"line":568},[189,1729,1730],{"class":378}," if",[189,1732,1733],{"class":389}," transcript ",[189,1735,753],{"class":622},[189,1737,627],{"class":626},[189,1739,718],{"class":427},[189,1741,1742,1745,1747,1750,1752,1755,1757,1759],{"class":312,"line":574},[189,1743,1744],{"class":389}," transcript ",[189,1746,775],{"class":622},[189,1748,1749],{"class":783}," Transcript",[189,1751,816],{"class":427},[189,1753,1754],{"class":819},"system",[189,1756,775],{"class":622},[189,1758,1754],{"class":783},[189,1760,571],{"class":427},[189,1762,1763,1766,1768,1771,1773,1776,1778,1781,1783,1785],{"class":312,"line":600},[189,1764,1765],{"class":389}," transcript",[189,1767,486],{"class":427},[189,1769,1770],{"class":783},"append",[189,1772,816],{"class":427},[189,1774,1775],{"class":783},"Message",[189,1777,486],{"class":427},[189,1779,1780],{"class":783},"user_text",[189,1782,816],{"class":427},[189,1784,1716],{"class":783},[189,1786,1787],{"class":427},"))\n",[189,1789,1790],{"class":312,"line":605},[189,1791,1792],{"class":371}," # ... existing setup\n",[189,1794,1795],{"class":312,"line":610},[189,1796,397],{"emptyLinePlaceholder":396},[189,1798,1799,1801,1804,1806,1809,1811,1814],{"class":312,"line":636},[189,1800,1362],{"class":378},[189,1802,1803],{"class":389}," iteration ",[189,1805,1373],{"class":378},[189,1807,1808],{"class":1710}," range",[189,1810,816],{"class":427},[189,1812,1813],{"class":1710},"MAX_ITERATIONS",[189,1815,1816],{"class":427},"):\n",[189,1818,1819,1822,1824,1826,1828,1831,1833,1835,1837],{"class":312,"line":641},[189,1820,1821],{"class":378}," with",[189,1823,1183],{"class":783},[189,1825,816],{"class":427},[189,1827,681],{"class":324},[189,1829,1830],{"class":320},"agent.turn",[189,1832,681],{"class":324},[189,1834,428],{"class":427},[189,1836,1197],{"class":783},[189,1838,565],{"class":427},[189,1840,1841,1844,1846,1848,1851,1853,1855,1858,1861,1863,1866],{"class":312,"line":646},[189,1842,1843],{"class":622}," **",[189,1845,1698],{"class":427},[189,1847,681],{"class":324},[189,1849,1850],{"class":320},"harness.iteration",[189,1852,681],{"class":324},[189,1854,616],{"class":427},[189,1856,1857],{"class":783}," iteration",[189,1859,1860],{"class":427},"})",[189,1862,1265],{"class":378},[189,1864,1865],{"class":389}," turn_span",[189,1867,718],{"class":427},[189,1869,1870],{"class":312,"line":660},[189,1871,1872],{"class":371}," # compaction span\n",[189,1874,1875,1878,1880,1883,1885,1888,1890,1893,1895,1898,1900,1903,1905,1908],{"class":312,"line":686},[189,1876,1877],{"class":389}," snapshot ",[189,1879,775],{"class":622},[189,1881,1882],{"class":389}," accountant",[189,1884,486],{"class":427},[189,1886,1887],{"class":783},"snapshot",[189,1889,816],{"class":427},[189,1891,1892],{"class":783},"transcript",[189,1894,428],{"class":427},[189,1896,1897],{"class":819}," tools",[189,1899,775],{"class":622},[189,1901,1902],{"class":783},"registry",[189,1904,486],{"class":427},[189,1906,1907],{"class":783},"schemas",[189,1909,859],{"class":427},[189,1911,1912,1915,1917,1919,1921,1923,1926,1928],{"class":312,"line":707},[189,1913,1914],{"class":389}," turn_span",[189,1916,486],{"class":427},[189,1918,1281],{"class":783},[189,1920,816],{"class":427},[189,1922,681],{"class":324},[189,1924,1925],{"class":320},"harness.context_utilization",[189,1927,681],{"class":324},[189,1929,565],{"class":427},[189,1931,1932,1935,1937,1940],{"class":312,"line":721},[189,1933,1934],{"class":783}," snapshot",[189,1936,486],{"class":427},[189,1938,1939],{"class":1069},"utilization",[189,1941,571],{"class":427},[189,1943,1944,1947,1950,1952,1955,1958,1960,1963,1965],{"class":312,"line":735},[189,1945,1946],{"class":378}," if",[189,1948,1949],{"class":389}," snapshot",[189,1951,486],{"class":427},[189,1953,1954],{"class":1069},"state",[189,1956,1957],{"class":622}," ==",[189,1959,675],{"class":324},[189,1961,1962],{"class":320},"red",[189,1964,681],{"class":324},[189,1966,718],{"class":427},[189,1968,1969,1972,1974,1976,1978,1981,1983,1985,1987],{"class":312,"line":744},[189,1970,1971],{"class":378}," with",[189,1973,1183],{"class":783},[189,1975,816],{"class":427},[189,1977,681],{"class":324},[189,1979,1980],{"class":320},"agent.compact",[189,1982,681],{"class":324},[189,1984,428],{"class":427},[189,1986,1197],{"class":783},[189,1988,1816],{"class":427},[189,1990,1991,1994,1997,1999,2002,2004,2006],{"class":312,"line":763},[189,1992,1993],{"class":378}," await",[189,1995,1996],{"class":389}," compactor",[189,1998,486],{"class":427},[189,2000,2001],{"class":783},"compact_if_needed",[189,2003,816],{"class":427},[189,2005,1892],{"class":783},[189,2007,565],{"class":427},[189,2009,2010,2013,2015,2017],{"class":312,"line":769},[189,2011,2012],{"class":783}," registry",[189,2014,486],{"class":427},[189,2016,1907],{"class":783},[189,2018,859],{"class":427},[189,2020,2021],{"class":312,"line":805},[189,2022,397],{"emptyLinePlaceholder":396},[189,2024,2025],{"class":312,"line":829},[189,2026,2027],{"class":371}," # LLM call span\n",[189,2029,2030,2033,2035,2037,2039,2041,2043,2045,2047],{"class":312,"line":842},[189,2031,2032],{"class":378}," with",[189,2034,1183],{"class":783},[189,2036,816],{"class":427},[189,2038,681],{"class":324},[189,2040,230],{"class":320},[189,2042,681],{"class":324},[189,2044,428],{"class":427},[189,2046,1197],{"class":783},[189,2048,565],{"class":427},[189,2050,2051,2054,2056,2058,2060,2062,2064,2067,2069,2071,2073,2075,2078],{"class":312,"line":862},[189,2052,2053],{"class":622}," **",[189,2055,1698],{"class":427},[189,2057,681],{"class":324},[189,2059,141],{"class":320},[189,2061,681],{"class":324},[189,2063,616],{"class":427},[189,2065,2066],{"class":783}," provider",[189,2068,486],{"class":427},[189,2070,1188],{"class":1069},[189,2072,1860],{"class":427},[189,2074,1265],{"class":378},[189,2076,2077],{"class":389}," llm_span",[189,2079,718],{"class":427},[189,2081,2082,2085,2087,2090,2093,2095,2098,2100,2103],{"class":312,"line":868},[189,2083,2084],{"class":389}," response ",[189,2086,775],{"class":622},[189,2088,2089],{"class":378}," await",[189,2091,2092],{"class":783}," _one_turn",[189,2094,816],{"class":427},[189,2096,2097],{"class":783},"provider",[189,2099,428],{"class":427},[189,2101,2102],{"class":783}," registry",[189,2104,565],{"class":427},[189,2106,2107,2110,2112,2115],{"class":312,"line":885},[189,2108,2109],{"class":783}," transcript",[189,2111,428],{"class":427},[189,2113,2114],{"class":783}," on_event",[189,2116,571],{"class":427},[189,2118,2119,2122,2124,2126],{"class":312,"line":890},[189,2120,2121],{"class":389}," llm_span",[189,2123,486],{"class":427},[189,2125,1281],{"class":783},[189,2127,657],{"class":427},[189,2129,2130,2133,2135,2137,2139,2142,2144,2147],{"class":312,"line":895},[189,2131,2132],{"class":324}," \"",[189,2134,145],{"class":320},[189,2136,681],{"class":324},[189,2138,428],{"class":427},[189,2140,2141],{"class":783}," response",[189,2143,486],{"class":427},[189,2145,2146],{"class":1069},"input_tokens",[189,2148,571],{"class":427},[189,2150,2151,2153,2155,2157],{"class":312,"line":905},[189,2152,2121],{"class":389},[189,2154,486],{"class":427},[189,2156,1281],{"class":783},[189,2158,657],{"class":427},[189,2160,2161,2163,2166,2168,2170,2172,2174,2177],{"class":312,"line":916},[189,2162,2132],{"class":324},[189,2164,2165],{"class":320},"gen_ai.usage.output_tokens",[189,2167,681],{"class":324},[189,2169,428],{"class":427},[189,2171,2141],{"class":783},[189,2173,486],{"class":427},[189,2175,2176],{"class":1069},"output_tokens",[189,2178,571],{"class":427},[189,2180,2181],{"class":312,"line":953},[189,2182,397],{"emptyLinePlaceholder":396},[189,2184,2185,2187,2189,2191,2194],{"class":312,"line":985},[189,2186,1946],{"class":378},[189,2188,2141],{"class":389},[189,2190,486],{"class":427},[189,2192,2193],{"class":1069},"is_final",[189,2195,718],{"class":427},[189,2197,2198,2201,2203,2205,2207,2209,2212,2214,2216,2218],{"class":312,"line":1005},[189,2199,2200],{"class":389}," s",[189,2202,486],{"class":427},[189,2204,1281],{"class":783},[189,2206,816],{"class":427},[189,2208,681],{"class":324},[189,2210,2211],{"class":320},"harness.final_iteration",[189,2213,681],{"class":324},[189,2215,428],{"class":427},[189,2217,1857],{"class":783},[189,2219,571],{"class":427},[189,2221,2222,2225,2227,2229,2231,2233,2235,2238,2240,2243],{"class":312,"line":1010},[189,2223,2224],{"class":389}," transcript",[189,2226,486],{"class":427},[189,2228,1770],{"class":783},[189,2230,816],{"class":427},[189,2232,1775],{"class":783},[189,2234,486],{"class":427},[189,2236,2237],{"class":783},"from_assistant_response",[189,2239,816],{"class":427},[189,2241,2242],{"class":783},"response",[189,2244,1787],{"class":427},[189,2246,2247,2250,2252,2254,2257,2260],{"class":312,"line":1047},[189,2248,2249],{"class":378}," return",[189,2251,2141],{"class":389},[189,2253,486],{"class":427},[189,2255,2256],{"class":1069},"text",[189,2258,2259],{"class":622}," or",[189,2261,2262],{"class":324}," \"\"\n",[189,2264,2265],{"class":312,"line":1057},[189,2266,397],{"emptyLinePlaceholder":396},[189,2268,2269],{"class":312,"line":1074},[189,2270,2271],{"class":371}," # Commit one assistant message with every ToolCall block.\n",[189,2273,2274,2277,2279,2281,2283,2285,2287,2289,2291,2293],{"class":312,"line":1090},[189,2275,2276],{"class":389}," transcript",[189,2278,486],{"class":427},[189,2280,1770],{"class":783},[189,2282,816],{"class":427},[189,2284,1775],{"class":783},[189,2286,486],{"class":427},[189,2288,2237],{"class":783},[189,2290,816],{"class":427},[189,2292,2242],{"class":783},[189,2294,1787],{"class":427},[189,2296,2297],{"class":312,"line":1102},[189,2298,2299],{"class":371}," # One tool.call span per dispatched call (batched turns get N spans).\n",[189,2301,2302,2305,2308,2310,2312,2314,2317],{"class":312,"line":1108},[189,2303,2304],{"class":378}," for",[189,2306,2307],{"class":389}," ref ",[189,2309,1373],{"class":378},[189,2311,2141],{"class":389},[189,2313,486],{"class":427},[189,2315,2316],{"class":1069},"tool_calls",[189,2318,718],{"class":427},[189,2320,2321,2323,2325,2327,2329,2331,2333,2335,2337],{"class":312,"line":1113},[189,2322,1971],{"class":378},[189,2324,1183],{"class":783},[189,2326,816],{"class":427},[189,2328,681],{"class":324},[189,2330,241],{"class":320},[189,2332,681],{"class":324},[189,2334,428],{"class":427},[189,2336,1197],{"class":783},[189,2338,565],{"class":427},[189,2340,2341,2344,2346,2348,2351,2353,2355,2358,2360,2362,2364,2366,2369],{"class":312,"line":1118},[189,2342,2343],{"class":622}," **",[189,2345,1698],{"class":427},[189,2347,681],{"class":324},[189,2349,2350],{"class":320},"tool.name",[189,2352,681],{"class":324},[189,2354,616],{"class":427},[189,2356,2357],{"class":783}," ref",[189,2359,486],{"class":427},[189,2361,1188],{"class":1069},[189,2363,1860],{"class":427},[189,2365,1265],{"class":378},[189,2367,2368],{"class":389}," tool_span",[189,2370,718],{"class":427},[189,2372,2373,2376,2378,2380,2382,2384,2387,2389,2392,2394,2396,2398,2400,2402,2405,2407,2409,2411,2414],{"class":312,"line":1135},[189,2374,2375],{"class":389}," result ",[189,2377,775],{"class":622},[189,2379,2089],{"class":378},[189,2381,2102],{"class":389},[189,2383,486],{"class":427},[189,2385,2386],{"class":783},"adispatch",[189,2388,816],{"class":427},[189,2390,2391],{"class":783},"ref",[189,2393,486],{"class":427},[189,2395,1188],{"class":1069},[189,2397,428],{"class":427},[189,2399,2357],{"class":783},[189,2401,486],{"class":427},[189,2403,2404],{"class":1069},"args",[189,2406,428],{"class":427},[189,2408,2357],{"class":783},[189,2410,486],{"class":427},[189,2412,2413],{"class":1069},"id",[189,2415,571],{"class":427},[189,2417,2418,2421,2423,2425,2427,2429,2432,2434,2436,2439,2441,2444],{"class":312,"line":1160},[189,2419,2420],{"class":389}," tool_span",[189,2422,486],{"class":427},[189,2424,1281],{"class":783},[189,2426,816],{"class":427},[189,2428,681],{"class":324},[189,2430,2431],{"class":320},"tool.is_error",[189,2433,681],{"class":324},[189,2435,428],{"class":427},[189,2437,2438],{"class":783}," result",[189,2440,486],{"class":427},[189,2442,2443],{"class":1069},"is_error",[189,2445,571],{"class":427},[189,2447,2448,2450,2452,2454,2456,2458,2461,2463,2465,2467,2469,2472,2474,2477],{"class":312,"line":1165},[189,2449,2420],{"class":389},[189,2451,486],{"class":427},[189,2453,1281],{"class":783},[189,2455,816],{"class":427},[189,2457,681],{"class":324},[189,2459,2460],{"class":320},"tool.result_chars",[189,2462,681],{"class":324},[189,2464,428],{"class":427},[189,2466,1711],{"class":1710},[189,2468,816],{"class":427},[189,2470,2471],{"class":783},"result",[189,2473,486],{"class":427},[189,2475,2476],{"class":1069},"content",[189,2478,1787],{"class":427},[189,2480,2481,2483,2485,2487,2489,2491,2493,2496,2498,2500],{"class":312,"line":1170},[189,2482,2224],{"class":389},[189,2484,486],{"class":427},[189,2486,1770],{"class":783},[189,2488,816],{"class":427},[189,2490,1775],{"class":783},[189,2492,486],{"class":427},[189,2494,2495],{"class":783},"tool_result",[189,2497,816],{"class":427},[189,2499,2471],{"class":783},[189,2501,1787],{"class":427},[189,2503,2504],{"class":312,"line":1178},[189,2505,2506],{"class":371}," # ...\n",[112,2508,2509,2510,2512,2513,2515,2516,1513,2518,2520],{},"The spans nest naturally: ",[139,2511,193],{}," contains ",[139,2514,1830],{},"s which contain ",[139,2517,230],{},[139,2519,241],{}," spans. A trace visualizer shows this as a flame chart — you see at a glance which turn took the time, which tool call was slow, which iteration hit compaction.",[266,2522],{},[269,2524,2526],{"id":2525},"_184-instrumenting-sub-agents","18.4 Instrumenting Sub-agents",[112,2528,2529,2530,2532,2533,1513,2535,616],{},"Sub-agents get their own ",[139,2531,1040],{}," but share the parent's ",[139,2534,152],{},[139,2536,155],{},[302,2538,2540],{"className":362,"code":2539,"language":364,"meta":307,"style":307},"# src\u002Fharness\u002Fsubagents\u002Fspawner.py (observability-aware)\n\nasync def spawn(self, spec: SubagentSpec, ...) -> SubagentResult:\n parent_ctx = get_current_session_context() # from a context var\n sub_ctx = parent_ctx.subagent(agent_id=f\"sub-{uuid4().hex[:8]}\")\n\n with span(\"subagent.spawn\", sub_ctx,\n **{\"subagent.objective_preview\": spec.objective[:200],\n \"subagent.tools_allowed\": \",\".join(spec.tools_allowed)}):\n # ... run sub-agent with sub_ctx propagated into arun\n",[139,2541,2542,2547,2551,2588,2603,2657,2661,2683,2713,2749],{"__ignoreMap":307},[189,2543,2544],{"class":312,"line":313},[189,2545,2546],{"class":371},"# src\u002Fharness\u002Fsubagents\u002Fspawner.py (observability-aware)\n",[189,2548,2549],{"class":312,"line":375},[189,2550,397],{"emptyLinePlaceholder":396},[189,2552,2553,2555,2557,2560,2562,2564,2566,2569,2571,2574,2576,2579,2581,2583,2586],{"class":312,"line":393},[189,2554,1602],{"class":649},[189,2556,1605],{"class":649},[189,2558,2559],{"class":653}," spawn",[189,2561,816],{"class":427},[189,2563,1022],{"class":1021},[189,2565,428],{"class":427},[189,2567,2568],{"class":663}," spec",[189,2570,616],{"class":427},[189,2572,2573],{"class":389}," SubagentSpec",[189,2575,428],{"class":427},[189,2577,2578],{"class":389}," ...",[189,2580,710],{"class":427},[189,2582,713],{"class":427},[189,2584,2585],{"class":389}," SubagentResult",[189,2587,718],{"class":427},[189,2589,2590,2593,2595,2598,2600],{"class":312,"line":400},[189,2591,2592],{"class":389}," parent_ctx ",[189,2594,775],{"class":622},[189,2596,2597],{"class":783}," get_current_session_context",[189,2599,1126],{"class":427},[189,2601,2602],{"class":371}," # from a context var\n",[189,2604,2605,2608,2610,2613,2615,2618,2620,2622,2624,2627,2630,2633,2635,2638,2641,2644,2647,2650,2653,2655],{"class":312,"line":414},[189,2606,2607],{"class":389}," sub_ctx ",[189,2609,775],{"class":622},[189,2611,2612],{"class":389}," parent_ctx",[189,2614,486],{"class":427},[189,2616,2617],{"class":783},"subagent",[189,2619,816],{"class":427},[189,2621,159],{"class":819},[189,2623,775],{"class":622},[189,2625,2626],{"class":649},"f",[189,2628,2629],{"class":320},"\"sub-",[189,2631,1698],{"class":2632},"srdBf",[189,2634,947],{"class":783},[189,2636,2637],{"class":427},"().",[189,2639,2640],{"class":1069},"hex",[189,2642,2643],{"class":427},"[:",[189,2645,2646],{"class":2632},"8",[189,2648,2649],{"class":427},"]",[189,2651,2652],{"class":2632},"}",[189,2654,681],{"class":320},[189,2656,571],{"class":427},[189,2658,2659],{"class":312,"line":434},[189,2660,397],{"emptyLinePlaceholder":396},[189,2662,2663,2665,2667,2669,2671,2674,2676,2678,2681],{"class":312,"line":447},[189,2664,1248],{"class":378},[189,2666,1183],{"class":783},[189,2668,816],{"class":427},[189,2670,681],{"class":324},[189,2672,2673],{"class":320},"subagent.spawn",[189,2675,681],{"class":324},[189,2677,428],{"class":427},[189,2679,2680],{"class":783}," sub_ctx",[189,2682,565],{"class":427},[189,2684,2685,2687,2689,2691,2694,2696,2698,2700,2702,2705,2707,2710],{"class":312,"line":460},[189,2686,1695],{"class":622},[189,2688,1698],{"class":427},[189,2690,681],{"class":324},[189,2692,2693],{"class":320},"subagent.objective_preview",[189,2695,681],{"class":324},[189,2697,616],{"class":427},[189,2699,2568],{"class":783},[189,2701,486],{"class":427},[189,2703,2704],{"class":1069},"objective",[189,2706,2643],{"class":427},[189,2708,2709],{"class":2632},"200",[189,2711,2712],{"class":427},"],\n",[189,2714,2715,2718,2721,2723,2725,2727,2729,2731,2733,2736,2738,2741,2743,2746],{"class":312,"line":465},[189,2716,2717],{"class":324}," \"",[189,2719,2720],{"class":320},"subagent.tools_allowed",[189,2722,681],{"class":324},[189,2724,616],{"class":427},[189,2726,675],{"class":324},[189,2728,428],{"class":320},[189,2730,681],{"class":324},[189,2732,486],{"class":427},[189,2734,2735],{"class":783},"join",[189,2737,816],{"class":427},[189,2739,2740],{"class":783},"spec",[189,2742,486],{"class":427},[189,2744,2745],{"class":1069},"tools_allowed",[189,2747,2748],{"class":427},")}):\n",[189,2750,2751],{"class":312,"line":478},[189,2752,2753],{"class":371}," # ... run sub-agent with sub_ctx propagated into arun\n",[112,2755,2756,2757,2760,2761,2764,2765,2767,2768,2771],{},"Under the hood, this uses Python's ",[139,2758,2759],{},"contextvars"," to propagate the context through the call stack — ",[139,2762,2763],{},"trace.get_current_span()"," would give us the OTel parent, but we want the harness-specific ",[139,2766,1040],{}," too. We add a ",[139,2769,2770],{},"contextvar"," for it:",[302,2773,2775],{"className":362,"code":2774,"language":364,"meta":307,"style":307},"# src\u002Fharness\u002Fobservability\u002Fcontext.py\nfrom contextvars import ContextVar\nfrom .tracing import SessionContext\n\n_current: ContextVar[SessionContext | None] = ContextVar(\"session_ctx\", default=None)\n\ndef set_current(ctx: SessionContext) -> None:\n _current.set(ctx)\n\ndef get_current_session_context() -> SessionContext:\n ctx = _current.get()\n if ctx is None:\n raise RuntimeError(\"no session context; call arun with one\")\n return ctx\n",[139,2776,2777,2782,2794,2807,2811,2857,2861,2885,2901,2905,2919,2935,2948,2967],{"__ignoreMap":307},[189,2778,2779],{"class":312,"line":313},[189,2780,2781],{"class":371},"# src\u002Fharness\u002Fobservability\u002Fcontext.py\n",[189,2783,2784,2786,2789,2791],{"class":312,"line":375},[189,2785,379],{"class":378},[189,2787,2788],{"class":389}," contextvars ",[189,2790,408],{"class":378},[189,2792,2793],{"class":389}," ContextVar\n",[189,2795,2796,2798,2800,2802,2804],{"class":312,"line":393},[189,2797,379],{"class":378},[189,2799,1572],{"class":427},[189,2801,1580],{"class":389},[189,2803,408],{"class":378},[189,2805,2806],{"class":389}," SessionContext\n",[189,2808,2809],{"class":312,"line":400},[189,2810,397],{"emptyLinePlaceholder":396},[189,2812,2813,2816,2818,2821,2823,2826,2828,2830,2832,2834,2836,2838,2840,2843,2845,2847,2850,2852,2855],{"class":312,"line":414},[189,2814,2815],{"class":389},"_current",[189,2817,616],{"class":427},[189,2819,2820],{"class":389}," ContextVar",[189,2822,1219],{"class":427},[189,2824,2825],{"class":389},"SessionContext ",[189,2827,623],{"class":622},[189,2829,627],{"class":626},[189,2831,2649],{"class":427},[189,2833,630],{"class":622},[189,2835,2820],{"class":783},[189,2837,816],{"class":427},[189,2839,681],{"class":324},[189,2841,2842],{"class":320},"session_ctx",[189,2844,681],{"class":324},[189,2846,428],{"class":427},[189,2848,2849],{"class":819}," default",[189,2851,775],{"class":622},[189,2853,2854],{"class":626},"None",[189,2856,571],{"class":427},[189,2858,2859],{"class":312,"line":434},[189,2860,397],{"emptyLinePlaceholder":396},[189,2862,2863,2865,2868,2870,2873,2875,2877,2879,2881,2883],{"class":312,"line":447},[189,2864,650],{"class":649},[189,2866,2867],{"class":653}," set_current",[189,2869,816],{"class":427},[189,2871,2872],{"class":663},"ctx",[189,2874,616],{"class":427},[189,2876,911],{"class":389},[189,2878,710],{"class":427},[189,2880,713],{"class":427},[189,2882,627],{"class":626},[189,2884,718],{"class":427},[189,2886,2887,2890,2892,2895,2897,2899],{"class":312,"line":460},[189,2888,2889],{"class":389}," _current",[189,2891,486],{"class":427},[189,2893,2894],{"class":783},"set",[189,2896,816],{"class":427},[189,2898,2872],{"class":783},[189,2900,571],{"class":427},[189,2902,2903],{"class":312,"line":465},[189,2904,397],{"emptyLinePlaceholder":396},[189,2906,2907,2909,2911,2913,2915,2917],{"class":312,"line":478},[189,2908,650],{"class":649},[189,2910,2597],{"class":653},[189,2912,1126],{"class":427},[189,2914,713],{"class":427},[189,2916,911],{"class":389},[189,2918,718],{"class":427},[189,2920,2921,2923,2925,2928,2930,2933],{"class":312,"line":502},[189,2922,1655],{"class":389},[189,2924,775],{"class":622},[189,2926,2927],{"class":389}," _current",[189,2929,486],{"class":427},[189,2931,2932],{"class":783},"get",[189,2934,1242],{"class":427},[189,2936,2937,2939,2942,2944,2946],{"class":312,"line":523},[189,2938,747],{"class":378},[189,2940,2941],{"class":389}," ctx ",[189,2943,753],{"class":622},[189,2945,627],{"class":626},[189,2947,718],{"class":427},[189,2949,2950,2953,2956,2958,2960,2963,2965],{"class":312,"line":549},[189,2951,2952],{"class":378}," raise",[189,2954,2955],{"class":669}," RuntimeError",[189,2957,816],{"class":427},[189,2959,681],{"class":324},[189,2961,2962],{"class":320},"no session context; call arun with one",[189,2964,681],{"class":324},[189,2966,571],{"class":427},[189,2968,2969,2971],{"class":312,"line":568},[189,2970,1138],{"class":378},[189,2972,2973],{"class":389}," ctx\n",[112,2975,2976,2977,2979,2980,2982],{},"The loop sets ",[139,2978,2815],{}," at the start; sub-agent runs re-set it when they begin; OTel spans carry the context via the ",[139,2981,1531],{}," helper.",[266,2984],{},[269,2986,2988],{"id":2987},"_185-what-the-traces-look-like","18.5 What the Traces Look Like",[112,2990,2991],{},"Run any example in the book with the console exporter enabled:",[302,2993,2995],{"className":362,"code":2994,"language":364,"meta":307,"style":307},"# prepend to any main() function\nfrom harness.observability.tracing import setup_tracing\nsetup_tracing()\n",[139,2996,2997,3002,3022],{"__ignoreMap":307},[189,2998,2999],{"class":312,"line":313},[189,3000,3001],{"class":371},"# prepend to any main() function\n",[189,3003,3004,3006,3009,3011,3013,3015,3017,3019],{"class":312,"line":375},[189,3005,379],{"class":378},[189,3007,3008],{"class":389}," harness",[189,3010,486],{"class":427},[189,3012,1575],{"class":389},[189,3014,486],{"class":427},[189,3016,1580],{"class":389},[189,3018,408],{"class":378},[189,3020,3021],{"class":389}," setup_tracing\n",[189,3023,3024,3027],{"class":312,"line":393},[189,3025,3026],{"class":783},"setup_tracing",[189,3028,1242],{"class":427},[112,3030,3031],{},"You get output like:",[302,3033,3037],{"className":3034,"code":3036,"language":2256},[3035],"language-text","{\n \"name\": \"agent.run\",\n \"trace_id\": \"...\",\n \"span_id\": \"a1b2c3d4\",\n \"attributes\": {\n \"service.name\": \"agent-harness\",\n \"harness.session_id\": \"s-xyz\",\n \"harness.task_id\": \"t-abc\",\n \"harness.agent_id\": \"root\",\n \"harness.final_iteration\": 7\n },\n \"duration_ms\": 12340\n}\n{\n \"name\": \"agent.turn\",\n \"parent_id\": \"a1b2c3d4\",\n ...\n}\n{\n \"name\": \"gen_ai.completion\",\n \"parent_id\": \"...\",\n \"attributes\": {\n \"gen_ai.system\": \"anthropic\",\n \"gen_ai.usage.input_tokens\": 3421,\n \"gen_ai.usage.output_tokens\": 188\n },\n \"duration_ms\": 1230\n}\n...\n",[139,3038,3036],{"__ignoreMap":307},[112,3040,3041],{},"Now point the exporter at a real backend:",[302,3043,3045],{"className":362,"code":3044,"language":364,"meta":307,"style":307},"# Langfuse, Braintrust, Datadog, Honeycomb, Jaeger all accept OTLP\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\nfrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter\n\nsetup_tracing(\n exporter=BatchSpanProcessor(OTLPSpanExporter(\n endpoint=\"https:\u002F\u002Fyour-backend\u002Fv1\u002Ftraces\",\n headers={\"Authorization\": \"Bearer ...\"},\n ))\n)\n",[139,3046,3047,3052,3075,3111,3115,3121,3137,3153,3181,3186],{"__ignoreMap":307},[189,3048,3049],{"class":312,"line":313},[189,3050,3051],{"class":371},"# Langfuse, Braintrust, Datadog, Honeycomb, Jaeger all accept OTLP\n",[189,3053,3054,3056,3058,3060,3062,3064,3066,3068,3070,3072],{"class":312,"line":375},[189,3055,379],{"class":378},[189,3057,483],{"class":389},[189,3059,486],{"class":427},[189,3061,489],{"class":389},[189,3063,486],{"class":427},[189,3065,536],{"class":389},[189,3067,486],{"class":427},[189,3069,541],{"class":389},[189,3071,408],{"class":378},[189,3073,3074],{"class":389}," BatchSpanProcessor\n",[189,3076,3077,3079,3081,3083,3086,3088,3091,3093,3096,3098,3101,3103,3106,3108],{"class":312,"line":393},[189,3078,379],{"class":378},[189,3080,483],{"class":389},[189,3082,486],{"class":427},[189,3084,3085],{"class":389},"exporter",[189,3087,486],{"class":427},[189,3089,3090],{"class":389},"otlp",[189,3092,486],{"class":427},[189,3094,3095],{"class":389},"proto",[189,3097,486],{"class":427},[189,3099,3100],{"class":389},"http",[189,3102,486],{"class":427},[189,3104,3105],{"class":389},"trace_exporter ",[189,3107,408],{"class":378},[189,3109,3110],{"class":389}," OTLPSpanExporter\n",[189,3112,3113],{"class":312,"line":400},[189,3114,397],{"emptyLinePlaceholder":396},[189,3116,3117,3119],{"class":312,"line":414},[189,3118,3026],{"class":783},[189,3120,657],{"class":427},[189,3122,3123,3125,3127,3130,3132,3135],{"class":312,"line":434},[189,3124,689],{"class":819},[189,3126,775],{"class":622},[189,3128,3129],{"class":783},"BatchSpanProcessor",[189,3131,816],{"class":427},[189,3133,3134],{"class":783},"OTLPSpanExporter",[189,3136,657],{"class":427},[189,3138,3139,3142,3144,3146,3149,3151],{"class":312,"line":447},[189,3140,3141],{"class":819}," endpoint",[189,3143,775],{"class":622},[189,3145,681],{"class":324},[189,3147,3148],{"class":320},"https:\u002F\u002Fyour-backend\u002Fv1\u002Ftraces",[189,3150,681],{"class":324},[189,3152,565],{"class":427},[189,3154,3155,3158,3160,3162,3164,3167,3169,3171,3173,3176,3178],{"class":312,"line":460},[189,3156,3157],{"class":819}," headers",[189,3159,775],{"class":622},[189,3161,1698],{"class":427},[189,3163,681],{"class":324},[189,3165,3166],{"class":320},"Authorization",[189,3168,681],{"class":324},[189,3170,616],{"class":427},[189,3172,675],{"class":324},[189,3174,3175],{"class":320},"Bearer ...",[189,3177,681],{"class":324},[189,3179,3180],{"class":427},"},\n",[189,3182,3183],{"class":312,"line":465},[189,3184,3185],{"class":427}," ))\n",[189,3187,3188],{"class":312,"line":478},[189,3189,571],{"class":427},[112,3191,3192],{},"Every run of the harness now produces structured traces you can visualize, filter, and compare. A regression investigation looks like: find the slow trace, expand the span tree, see the tool call that took 12 seconds, look at its arguments.",[266,3194],{},[269,3196,3198],{"id":3197},"_186-metrics-that-matter","18.6 Metrics That Matter",[112,3200,3201],{},"From the span attributes above, production dashboards usually show:",[3203,3204,3205,3217,3230,3239,3248],"ul",{},[132,3206,3207,194,3210,3213,3214,3216],{},[279,3208,3209],{},"Per-agent token cost over time.",[139,3211,3212],{},"gen_ai.usage.input_tokens + gen_ai.usage.output_tokens"," grouped by ",[139,3215,1344],{},". A spike in one sub-agent is visible.",[132,3218,3219,3222,3223,3225,3226,3229],{},[279,3220,3221],{},"Tool call error rate."," Percent of ",[139,3224,241],{}," spans with ",[139,3227,3228],{},"tool.is_error = true",". A sustained rise is a regression signal.",[132,3231,3232,3235,3236,3238],{},[279,3233,3234],{},"Compaction frequency."," Count of ",[139,3237,1980],{}," spans per session. If compaction fires every turn, something is wrong with either your tool output sizes or your budget thresholds.",[132,3240,3241,3244,3245,3247],{},[279,3242,3243],{},"Sub-agent outcome mix."," Count ",[139,3246,2673],{}," spans with error vs. success. If a specific sub-agent objective starts failing more, you're seeing prompt or model drift.",[132,3249,3250,3253],{},[279,3251,3252],{},"Trace duration p50\u002Fp99."," Standard latency SLOs. Agent workloads are slow on average; what you care about is the tail and the trend.",[112,3255,3256],{},"These are queries over your tracing backend, not custom code in the harness. The harness emits; the platform aggregates.",[266,3258],{},[269,3260,3262],{"id":3261},"_187-cost-attribution-specifically","18.7 Cost Attribution, Specifically",[112,3264,3265,3266,3273],{},"The DEV Community 2025 post ",[3267,3268,3272],"a",{"href":3269,"rel":3270},"https:\u002F\u002Fdev.to\u002Fgeorge_belsky_a513cfbf3df\u002Fyour-ai-agent-spent-500-overnight-and-nobody-noticed-4chj",[3271],"nofollow","\"Your AI Agent Spent $500 Overnight and Nobody Noticed\""," was about a team that got a billing alert with no diagnostic signal. No per-agent attribution; they couldn't tell which agent ran away.",[112,3275,3276,3277,3279,3280,3282],{},"Our instrumentation answers that. Every ",[139,3278,230],{}," span carries ",[139,3281,1344],{},". A production dashboard:",[302,3284,3288],{"className":3285,"code":3286,"language":3287,"meta":307,"style":307},"language-sql shiki shiki-themes material-theme-lighter github-light github-dark","SELECT harness_agent_id, SUM(input_tokens + output_tokens) AS total_tokens\nFROM spans\nWHERE span_name = 'gen_ai.completion'\n AND timestamp > NOW() - INTERVAL '1 hour'\nGROUP BY harness_agent_id\nORDER BY total_tokens DESC\n","sql",[139,3289,3290,3317,3325,3341,3370,3378],{"__ignoreMap":307},[189,3291,3292,3296,3299,3302,3305,3308,3311,3314],{"class":312,"line":313},[189,3293,3295],{"class":3294},"sw1J6","SELECT",[189,3297,3298],{"class":389}," harness_agent_id, ",[189,3300,3301],{"class":1710},"SUM",[189,3303,3304],{"class":389},"(input_tokens ",[189,3306,3307],{"class":622},"+",[189,3309,3310],{"class":389}," output_tokens) ",[189,3312,3313],{"class":3294},"AS",[189,3315,3316],{"class":389}," total_tokens\n",[189,3318,3319,3322],{"class":312,"line":375},[189,3320,3321],{"class":3294},"FROM",[189,3323,3324],{"class":389}," spans\n",[189,3326,3327,3330,3333,3335,3337,3339],{"class":312,"line":393},[189,3328,3329],{"class":3294},"WHERE",[189,3331,3332],{"class":389}," span_name ",[189,3334,775],{"class":622},[189,3336,325],{"class":324},[189,3338,230],{"class":320},[189,3340,346],{"class":324},[189,3342,3343,3346,3349,3352,3355,3357,3360,3363,3365,3368],{"class":312,"line":400},[189,3344,3345],{"class":3294}," AND",[189,3347,3348],{"class":649}," timestamp",[189,3350,3351],{"class":622}," >",[189,3353,3354],{"class":3294}," NOW",[189,3356,1126],{"class":427},[189,3358,3359],{"class":622}," -",[189,3361,3362],{"class":389}," INTERVAL ",[189,3364,331],{"class":324},[189,3366,3367],{"class":320},"1 hour",[189,3369,346],{"class":324},[189,3371,3372,3375],{"class":312,"line":414},[189,3373,3374],{"class":3294},"GROUP BY",[189,3376,3377],{"class":389}," harness_agent_id\n",[189,3379,3380,3383,3386],{"class":312,"line":434},[189,3381,3382],{"class":3294},"ORDER BY",[189,3384,3385],{"class":389}," total_tokens ",[189,3387,3388],{"class":3294},"DESC\n",[112,3390,3391,3392,3394],{},"A runaway agent shows up immediately — one ",[139,3393,159],{}," with 10× the tokens of any other. Alert on it; kill the session manually. Chapter 20 automates the kill.",[266,3396],{},[269,3398,3400],{"id":3399},"_188-commit","18.8 Commit",[302,3402,3404],{"className":304,"code":3403,"language":306,"meta":307,"style":307},"git add -A && git commit -m \"ch18: OpenTelemetry instrumentation with GenAI semantic conventions\"\ngit tag ch18-observability\n",[139,3405,3406,3436],{"__ignoreMap":307},[189,3407,3408,3411,3413,3417,3420,3423,3426,3429,3431,3434],{"class":312,"line":313},[189,3409,3410],{"class":316},"git",[189,3412,321],{"class":320},[189,3414,3416],{"class":3415},"stzsN"," -A",[189,3418,3419],{"class":427}," &&",[189,3421,3422],{"class":316}," git",[189,3424,3425],{"class":320}," commit",[189,3427,3428],{"class":3415}," -m",[189,3430,675],{"class":324},[189,3432,3433],{"class":320},"ch18: OpenTelemetry instrumentation with GenAI semantic conventions",[189,3435,1002],{"class":324},[189,3437,3438,3440,3443],{"class":312,"line":375},[189,3439,3410],{"class":316},[189,3441,3442],{"class":320}," tag",[189,3444,3445],{"class":320}," ch18-observability\n",[269,3447,3449],{"id":3448},"_189-try-it-yourself","18.9 Try It Yourself",[129,3451,3452,3467,3473],{},[132,3453,3454,3457,3458,3461,3462,3466],{},[279,3455,3456],{},"Run Jaeger locally and visualize."," Stand up Jaeger in a container (",[139,3459,3460],{},"docker run -p 16686:16686 jaegertracing\u002Fall-in-one","). Point the harness OTLP exporter at it. Run a multi-agent scenario. Open ",[3267,3463,3464],{"href":3464,"rel":3465},"http:\u002F\u002Flocalhost:16686",[3271]," and explore the trace tree.",[132,3468,3469,3472],{},[279,3470,3471],{},"Find the slow operation."," Instrument a realistic task. Look at the trace. What's the longest span? Is it what you expected? If not, what did that tell you?",[132,3474,3475,3478],{},[279,3476,3477],{},"Add one custom attribute that matters."," Pick a metric that's specific to your use case — retrieval hit count, plan steps completed, retry count, whatever. Attach it to the right spans. Build a dashboard query that uses it.",[266,3480],{},[3482,3483,3484,3487],"what-you-understand",{},[112,3485,3486],{},"The harness emits structured, correlated, OTel-compliant traces. Every LLM call, tool call, compaction event, and sub-agent spawn is a span with session, task, and agent IDs. Exporters plug in without code changes. Per-agent cost attribution — the thing the $500 post-mortem lacked — is a single query on your tracing backend. Dashboards track regression indicators (error rate, compaction frequency, token cost by agent) that let you catch drift before it costs you.",[112,3488,3489,3490,3493],{},"What's still missing. Observability tells you what happened; it doesn't tell you whether what happened was ",[115,3491,3492],{},"correct",". A run with zero errors can still be wrong. Chapter 19 is evals: how you define \"correct\" for an agent, how you run regression tests against golden trajectories, and how you feed production failures back into your eval set so tomorrow's agent doesn't repeat today's mistakes.",[3495,3496,3497],"style",{},"html pre.shiki code .sbgvK, html code.shiki .sbgvK{--shiki-light:#E2931D;--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .s_sjI, html code.shiki .s_sjI{--shiki-light:#91B859;--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sjJ54, html code.shiki .sjJ54{--shiki-light:#39ADB5;--shiki-default:#032F62;--shiki-dark:#9ECBFF}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sutJx, html code.shiki .sutJx{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#6A737D;--shiki-default-font-style:inherit;--shiki-dark:#6A737D;--shiki-dark-font-style:inherit}html pre.shiki code .sVHd0, html code.shiki .sVHd0{--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#D73A49;--shiki-default-font-style:inherit;--shiki-dark:#F97583;--shiki-dark-font-style:inherit}html pre.shiki code .s_hVV, html code.shiki .s_hVV{--shiki-light:#90A4AE;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .su5hD, html code.shiki .su5hD{--shiki-light:#90A4AE;--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sP7_E, html code.shiki .sP7_E{--shiki-light:#39ADB5;--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .smGrS, html code.shiki .smGrS{--shiki-light:#39ADB5;--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .s39Yj, html code.shiki .s39Yj{--shiki-light:#39ADB5;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sbsja, html code.shiki .sbsja{--shiki-light:#9C3EDA;--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sGLFI, html code.shiki .sGLFI{--shiki-light:#6182B8;--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sFwrP, html code.shiki .sFwrP{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit}html pre.shiki code .sZMiF, html code.shiki .sZMiF{--shiki-light:#E2931D;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .s2W-s, html code.shiki .s2W-s{--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#032F62;--shiki-default-font-style:inherit;--shiki-dark:#9ECBFF;--shiki-dark-font-style:inherit}html pre.shiki code .sithA, html code.shiki .sithA{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#032F62;--shiki-default-font-style:inherit;--shiki-dark:#9ECBFF;--shiki-dark-font-style:inherit}html pre.shiki code .slqww, html code.shiki .slqww{--shiki-light:#6182B8;--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .s99_P, html code.shiki .s99_P{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#E36209;--shiki-default-font-style:inherit;--shiki-dark:#FFAB70;--shiki-dark-font-style:inherit}html pre.shiki code .stp6e, html code.shiki .stp6e{--shiki-light:#39ADB5;--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .smCYv, html code.shiki .smCYv{--shiki-light:#E53935;--shiki-light-font-style:italic;--shiki-default:#24292E;--shiki-default-font-style:inherit;--shiki-dark:#E1E4E8;--shiki-dark-font-style:inherit}html pre.shiki code .skxfh, html code.shiki .skxfh{--shiki-light:#E53935;--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .swQdS, html code.shiki .swQdS{--shiki-light:#E53935;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sptTA, html code.shiki .sptTA{--shiki-light:#6182B8;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .srdBf, html code.shiki .srdBf{--shiki-light:#F76D47;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .stzsN, html code.shiki .stzsN{--shiki-light:#91B859;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sw1J6, html code.shiki .sw1J6{--shiki-light:#F76D47;--shiki-default:#D73A49;--shiki-dark:#F97583}",{"title":307,"searchDepth":375,"depth":375,"links":3499},[3500,3501,3502,3503,3504,3505,3506,3507,3508],{"id":271,"depth":375,"text":272},{"id":351,"depth":375,"text":352},{"id":1551,"depth":375,"text":1552},{"id":2525,"depth":375,"text":2526},{"id":2987,"depth":375,"text":2988},{"id":3197,"depth":375,"text":3198},{"id":3261,"depth":375,"text":3262},{"id":3399,"depth":375,"text":3400},{"id":3448,"depth":375,"text":3449},"md",{},null,{"title":82,"description":117},"FZPFUx-ui7hh-QUSpKGC-0Z8hnBszSTuCMcYWio2qH0",[3515,3517],{"title":78,"path":79,"stem":80,"description":3516,"children":-1},"Previously: structured plans with evidence-backed completion. Sub-agents still run sequentially. The payoff for sub-agents comes from running them in parallel — the Anthropic multi-agent finding of 90%+ improvement over single-agent baselines rests on parallelism plus independent context windows, not sub-agents in series.",{"title":86,"path":87,"stem":88,"description":3518,"children":-1},"Previously: observability — every operation in the harness emits a structured span, per-agent cost attribution works, dashboards show drift. Observability says what happened; it doesn't say whether what happened was right.",1776848986841]