pydantic-evals 2.6.0
“Framework for evaluating stochastic code execution, especially code making use of LLMs”
setup() call followed by other codeBundled Python dependency library path
SHA-25615f2dd7b17543f383f4f2c44b4f1bb41705a9a9e3697b82db976de8e433a56a4
MaleculeO(As)H(Db)
Evidence
1"""Span-based evaluators for agentic workflows.
2
3These evaluators compare the tool-call trajectory captured in OpenTelemetry
4spans against expectations. They are deterministic and require no LLM calls,
5so they are cheap to run and produce the same score for the same trace.
6
7All of the evaluators in this module ([`ToolCorrectness`][pydantic_evals.evaluators.ToolCorrectness],
8[`TrajectoryMatch`][pyd …
18:65… rver-side
19 builtin tools — such as OpenAI's file search or Anthropic's web search —
20 do not create local spans and will not be counted.
21
22!!! note "What counts as a tool call"
23 Every execut …
452:10… ),
453 )
454
455 try:
456 actual_args = json.loads(selected.arguments)
457 except json.JSONDecodeError as e:
458 return EvaluationReason(
459 value=False, …
492:20… ] = []
493 for key, expected_value in expected.items():
494 if key not in actual:
495 diffs.append(f'missing key {key!r}')
496 elif actual[key] != expected_value:
497 diffs.append(f'key {ke …
115:16… erence is None:
116 return None
117 try:
118 trace_id = int(span_reference.trace_id, 16)
119 span_id = int(span_reference.span_id, 16)
120 except ValueError: # pragma: no cover — d …
No evidence locations were recorded for this file. Raw result