← Archive
DemosBlogTry PViz
Cross-language evaluationLLM toolingRepository analysisAudit-ready evidence

Repository Context Strategies for Codebase Analysis Agents

A controlled cross-language evaluation of raw source exploration, PViz-guided source reads, and bundle-only analysis.

Evaluated sessions

84

Tasks Per Repository

4

Repositories

7

Context strategies

3

The question

What changes when an agent receives different kinds of repository context?

This evaluation compares normal source exploration, PViz-guided source verification, and bundle-only analysis. The goal is not to crown a universal winner; it is to identify which context strategy is appropriate for which class of repository-analysis task.

Raw source exploration

Repository source access with normal file discovery and inspection.

Intended use

Baseline investigation when source bodies and tests are the only available context.

PViz-assisted investigation

PViz structural bundle plus targeted source reads for verification.

Intended use

Source-grounded investigation with structural navigation, graph metrics, and scoped verification.

PViz Solo

PViz bundle only; no repository source reads.

Intended use

Architectural orientation, structural risk discovery, and explicitly bounded first-pass analysis.

Methodology

A controlled cross-language evaluation

Assessment sequence

  1. 1

    Define the task record

    Write the prompt, task-family classification, and source-grounded ground-truth card before reviewing any condition answer.

  2. 2

    Run three context conditions

    Evaluate Raw source exploration, PViz-assisted investigation, and PViz Solo against the same task.

  3. 3

    Assess independently

    Review each condition against the shared ground-truth record rather than against the other condition outputs.

  4. 4

    Score and preserve evidence

    Score D1–D5, retain transcripts and metadata, then synthesize results by task family rather than as a universal winner ranking.

Scoring dimensions

  • D1 — Core correctness: Accuracy and completeness of the task's required factual conclusions.
  • D2 — Critical depth: Whether the answer reached the task's decisive technical nuance rather than only its surface facts.
  • D3 — Confidence calibration: Clear separation of established evidence, inference, uncertainty, and source-only limits.
  • D4 — Evidence efficiency: Whether the investigation footprint was proportionate to the evidence gathered and the task's demands.
  • D5 — Answer discipline: Organization, relevance, and clarity of the final answer.

Model Used

All 84 sessions were run using Claude Sonnet 4.6 via Claude Code. Conditions differed only in the context supplied to the agent — Raw, PViz-assisted, or bundle-only — with the model and harness held constant across all sessions.

Central finding

Task family predicted the useful context strategy more reliably than language or repository.

Behavioral-semantic questions require source-level verification. Mixed tasks benefit most from PViz-assisted work because structural metrics and source semantics are both material. Structural-native questions can often be oriented from the bundle alone, with uncertainty explicitly bounded.

Task familyWhat it asksBest-supported modeWhy
Behavioral / semanticWhat happens, in what order, under which conditions?Source-backed analysisMethod bodies, branches, assertions, and runtime semantics are decisive.
MixedWhat is structurally affected, and what behavior makes that matter?PViz-assisted investigationThe bundle supplies structural metrics and orientation; source verifies behavioral detail.
Structural-nativeWhat is connected, risky, cyclic, or widely depended on?PViz Solo or PViz-assistedThe decisive facts are often directly represented in the structural bundle.

Aggregate results

Answer quality and operating score are reported separately

ConditionD1 + D2Total D1–D5Interpretation
Raw source exploration192/196 (98.0%)407/420 (96.9%)Near-ceiling source baseline across the benchmark.
PViz-assisted investigation196/196 (100.0%)400/420 (95.2%)Matched or exceeded Raw on substantive scoring; total-score deductions were concentrated in evidence-efficiency overhead and known harness artifacts.
PViz Solo131/196 (66.8%)326/420 (77.6%)Strong structural orientation with a consistent boundary on behavior-sensitive claims.

Interpretation guardrail: D1 and D2 describe substantive answer quality. Total-score differences also include D3–D5, including evidence-efficiency deductions and documented harness or query-execution effects. The results should not be collapsed into a single winner label.

Next evaluation: A follow-up showcase will apply the targeted harness and workflow fixes identified here, then rerun a controlled subset to test whether PViz-assisted and PViz Solo improve confidence calibration, evidence efficiency, and answer discipline without sacrificing substantive answer quality.

Practical guidance

What each mode is good for

Raw source exploration

Baseline investigation when source bodies and tests are the only available context.

PViz-assisted investigation

Source-grounded investigation with structural navigation, graph metrics, and scoped verification.

PViz Solo

Architectural orientation, structural risk discovery, and explicitly bounded first-pass analysis.

Representative evidence

Task-level records make the aggregate findings inspectable

Example task packet

Each task packet contains everything needed to verify a result independently — the original prompt, a source-grounded ground-truth card, per-condition transcripts and metadata, and the final assessment with dimension scores.

  • →Task packet — prompt, ground truth, assessment, scores, and anomaly notes where applicable.
  • →Condition records — full transcripts and run metadata for Raw, PViz-assisted, and PViz Solo.
  • →28 task packets — one for each of the 7 repositories × 4 tasks in the benchmark.
Browse the full evidence explorerFilter and inspect all 28 task packets, including scores, transcripts, metadata, ground truth, and assessment records.

Example

Go · Caddy

Mixed

Task D — Blast Radius and Coupling Analysis for caddy/v2

PViz supplied risk, SCC, importer, and cycle context while using fewer tool calls than the Raw condition.

Task packet

Prompt ↗Ground truth ↗Assessment ↗Scores ↗Anomalies ↗Task methodology ↗

Condition records

pviz

Transcript ↗Metadata ↗

solo

Transcript ↗Metadata ↗

Limits and interpretation

What this benchmark does and does not establish

Scope set before evaluation

These boundaries were part of the evaluation design and define the claims the benchmark was intended to support.

  • One run per condition: this is an auditable evaluation set, not a variance study across repeated stochastic trials.
  • The 28-task mix was heavily weighted toward behavioral-semantic work, where source access is structurally decisive.
  • Static structure cannot substitute for runtime behavior, implementation ordering, or test assertions.
  • Results should be interpreted by task family rather than as a universal ranking of context strategies.

Identified during evaluation

These findings emerged while running or scoring the benchmark and affect how specific results should be interpreted.

  • D4 included known non-product harness effects, including Ruby PATH failure and TypeScript JSON re-parsing overhead.
  • Task-family fit mattered more than language or repository when determining which context strategy was useful.
  • The bundle's contribution varied by task: it could improve navigation, expose structural metrics, or remain insufficient without source verification.

Audit trail

Evidence and downloads

Evaluation evidence manifest

Index of the 28 task packets, 84 condition transcripts, metadata, and associated archive artifacts.

JSON

Cross-language summary

Aggregate scoring, task-family analysis, contribution patterns, anomalies, and methodology notes.

PDF

Go summary

Per-language findings, task-family breakdown, and anomaly notes for Caddy.

PDF

Java summary

Per-language findings, task-family breakdown, and anomaly notes for Micrometer.

PDF

Kotlin summary

Per-language findings, task-family breakdown, and anomaly notes for kotlinx.coroutines.

PDF

Python summary

Per-language findings, task-family breakdown, and anomaly notes for Textual.

PDF

Ruby summary

Per-language findings, task-family breakdown, and anomaly notes for Huginn.

PDF

Rust summary

Per-language findings, task-family breakdown, and anomaly notes for ripgrep.

PDF

TypeScript summary

Per-language findings, task-family breakdown, and anomaly notes for Vite.

PDF