Historical showcase

Published with the analyzer of its time; the results below are preserved as published. A later re-examination is available: Repository Context Strategies: Findings Addendum.

Cross-language evaluationLLM toolingRepository analysisAudit-ready evidenceReused raw baseline

Repository Context Strategies After Revision

A second cross-language evaluation cycle: new PViz-guided and bundle-only runs across the same seven repositories, presented alongside a historical raw-source-exploration baseline from the prior cycle.

New sessions this cycle

Tasks per repository

Repositories

Context strategies compared

The question

What changes when an agent receives different kinds of repository context?

This evaluation compares normal source exploration, PViz-guided source verification, and bundle-only analysis. The goal is not to crown a universal winner; it is to identify which context strategy is appropriate for which class of repository-analysis task.

Raw source exploration (historical baseline)

Repository source access with normal file discovery and inspection.

Intended use

Historical source-only baseline reused from the prior cycle on the same repositories. It was not re-run this cycle so the new evaluation could focus on the revised PViz-guided and bundle-only conditions; exact prompt-wording continuity with this cycle's PViz/Solo tasks has not been separately verified, so Raw is presented as reference context rather than a strictly controlled arm of this cycle's comparison.

PViz-assisted investigation

PViz structural bundle plus targeted source reads for verification.

Intended use

Source-grounded investigation with structural navigation, graph metrics, and scoped verification. New sessions this cycle.

PViz Solo

PViz bundle only; no repository source reads.

Intended use

Architectural orientation, structural risk discovery, and explicitly bounded first-pass analysis. New sessions this cycle.

Methodology

A controlled cross-language evaluation

Assessment sequence

1
Define the task record
Write the prompt, task-family classification, and source-grounded ground-truth card before reviewing any condition answer.
2
Run three context conditions
Evaluate Raw source exploration, PViz-assisted investigation, and PViz Solo against the same task.
3
Assess independently
Review each condition against the shared ground-truth record rather than against the other condition outputs.
4
Score and preserve evidence
Score D1–D5, retain transcripts and metadata, then synthesize results by task family rather than as a universal winner ranking.

Scoring dimensions

D1 — Core correctness: Accuracy and completeness of the task's required factual conclusions.
D2 — Critical depth: Whether the answer reached the task's decisive technical nuance rather than only its surface facts.
D3 — Confidence calibration: Clear separation of established evidence, inference, uncertainty, and source-only limits.
D4 — Evidence efficiency: Whether the investigation footprint was proportionate to the evidence gathered and the task's demands.
D5 — Answer discipline: Organization, relevance, and clarity of the final answer.

Model Used

All 84 sessions were run using Claude Sonnet 4.6 via Claude Code. Conditions differed only in the context supplied to the agent — Raw, PViz-assisted, or bundle-only — with the model and harness held constant across all sessions.

Central finding

Task family again predicted the useful context strategy more reliably than language or repository — with a sharper split inside structural-native tasks this cycle.

Behavioral-semantic questions still required source-level verification in every language. Mixed tasks again benefited most from PViz's combination of structural metrics and targeted source reads. Structural-native questions, now assessed across four languages instead of one, split by sub-skill: bundle-only analysis performed best when the requested topology metric was directly represented in the bundle (near-parity with PViz on one language's structural-native task) but remained vulnerable when correctness depended on complete edge extraction — an unresolved wildcard import produced this cycle's single largest solo shortfall on another language's structural-native task.

Task family	What it asks	Best-supported mode	Why
Behavioral / semantic	What happens, in what order, under which conditions?	Source-backed analysis	Method bodies, branches, assertions, and runtime semantics are decisive.
Mixed	What is structurally affected, and what behavior makes that matter?	PViz-assisted investigation	The bundle supplies structural metrics and orientation; source verifies behavioral detail.
Structural-native	What is connected, risky, cyclic, or widely depended on?	PViz Solo for topology questions; PViz-assisted when cycle/coupling detection is decisive	Topology metrics (importer counts, risk scores) are directly represented in the bundle, but cycle detection can be silently wrong when a wildcard import or intra-package reference is not fully resolved — bundle-only analysis has no way to detect this on its own.

Aggregate results

Answer quality and operating score are reported separately

Condition	D1 + D2	Total D1–D5	Interpretation
Raw source exploration (historical baseline)	192/196 (98.0%)	407/420 (96.9%)	Historical near-ceiling source baseline reused from the prior evaluation cycle on these same seven repositories; not re-run this cycle by design and shown here as reference context, not a strictly controlled arm of this cycle's comparison.
PViz-assisted investigation	196/196 (100.0%)	414/420 (98.6%)	Matched or exceeded the reused historical Raw baseline's total score in every language this cycle, a reversal from the prior cycle; D1+D2 correctness was already at parity-or-better against that baseline in both cycles and remains perfect within this cycle's own PViz sessions.
PViz Solo	133/196 (67.9%)	344/420 (81.9%)	Strong, consistently well-calibrated structural orientation, with markedly widened variance on structural-native tasks this cycle — near-parity with PViz on one language's structural-native task, this cycle's lowest single-task score on another.

Practical guidance

What each mode is good for

Raw source exploration (historical baseline)

PViz-assisted investigation

Source-grounded investigation with structural navigation, graph metrics, and scoped verification. New sessions this cycle.

PViz Solo

Architectural orientation, structural risk discovery, and explicitly bounded first-pass analysis. New sessions this cycle.

Representative evidence

Task-level records make the aggregate findings inspectable

Example task packet

Each task packet contains everything needed to verify a result independently — the original prompt, a source-grounded ground-truth card, per-condition transcripts and metadata, and the final assessment with dimension scores.

→Task packet — prompt, ground truth, assessment, scores, and anomaly notes where applicable.
→Condition records — full transcripts and run metadata for Raw, PViz-assisted, and PViz Solo.
→28 task packets — one for each of the 7 repositories × 4 tasks in the benchmark.

Browse the full evidence explorerFilter and inspect all 28 task packets, including scores, transcripts, metadata, ground truth, and assessment records.

Example

Java · Micrometer

Structural-native

Task D — Blast Radius and Coupling Analysis for MeterRegistry.java

Solo trusted the bundle's reported SCC=1 (no cycles) and stated MeterRegistry has no dependency cycles; PViz read source and found a real bidirectional cycle via a wildcard import (Search/RequiredSearch), the dataset's single largest solo miss.

Task packet

Ground truth ↗Assessment ↗Scores ↗Anomalies ↗Task methodology ↗

Condition records

raw

Transcript ↗Metadata ↗

pviz

Transcript ↗Metadata ↗

solo

Transcript ↗Metadata ↗

Limits and interpretation

What this benchmark does and does not establish

Scope set before evaluation

These boundaries were part of the evaluation design and define the claims the benchmark was intended to support.

One run per condition per cycle: this is an auditable evaluation set, not a variance study across repeated stochastic trials. A rerun of any condition, including Raw, could plausibly produce a different result even against unchanged source.
The Raw condition was not re-run this cycle. Its scores are reused from the prior evaluation cycle on these same seven repositories; see the Raw condition's own description for why it is presented as a historical reference baseline rather than a strictly controlled arm of this cycle.
This cycle's PViz/Solo task prompts were independently redrafted from the prior cycle. Exact wording equivalence with the reused Raw baseline's original prompts has not been verified — the two cycles target the same 28 task slots, families, and repository files, but not confirmed-identical prompt text. Comparisons against Raw should be read accordingly.
The 28-task mix remains heavily weighted toward behavioral-semantic work (21 of 28 tasks this cycle), where source access is structurally decisive.
Static structure cannot substitute for runtime behavior, implementation ordering, or test assertions.
Results should be interpreted by task family rather than as a universal ranking of context strategies.

Identified during evaluation

These findings emerged while running or scoring the benchmark and affect how specific results should be interpreted.

Two harness-level inefficiencies identified in the prior cycle (a Ruby PATH failure and a TypeScript bundle re-parsing overhead) did not reappear this cycle; this is the primary driver of PViz's improved total score against the reused Raw baseline.
Task-family fit mattered more than language or repository when determining which context strategy was useful.
The bundle's contribution varied by task: it could improve navigation, expose structural metrics, or — in four of the seven structural/mixed tasks this cycle — catch the bundle's own structural errors via a source cross-check.

Audit trail

Evidence and downloads

Cross-language summary report

Full write-up of this cycle's results across all seven repositories.

PDF

4th vs. 5th showcase comparison

Score deltas, resolved harness issues, and data-quality notes comparing this cycle against the prior one.

PDF

Bundle suitability analysis

Analysis of where the PViz bundle alone was sufficient versus where source verification was required.

PDF

What changes when an agent receives different kinds of repository context?

Raw source exploration (historical baseline)

Repository source access with normal file discovery and inspection.

Intended use

PViz-assisted investigation

PViz structural bundle plus targeted source reads for verification.

Intended use

Source-grounded investigation with structural navigation, graph metrics, and scoped verification. New sessions this cycle.

PViz Solo

PViz bundle only; no repository source reads.

Intended use

Architectural orientation, structural risk discovery, and explicitly bounded first-pass analysis. New sessions this cycle.

A controlled cross-language evaluation

Assessment sequence

1
Define the task record
Write the prompt, task-family classification, and source-grounded ground-truth card before reviewing any condition answer.
2
Run three context conditions
Evaluate Raw source exploration, PViz-assisted investigation, and PViz Solo against the same task.
3
Assess independently
Review each condition against the shared ground-truth record rather than against the other condition outputs.
4
Score and preserve evidence
Score D1–D5, retain transcripts and metadata, then synthesize results by task family rather than as a universal winner ranking.

Scoring dimensions

D1 — Core correctness: Accuracy and completeness of the task's required factual conclusions.
D2 — Critical depth: Whether the answer reached the task's decisive technical nuance rather than only its surface facts.
D3 — Confidence calibration: Clear separation of established evidence, inference, uncertainty, and source-only limits.
D4 — Evidence efficiency: Whether the investigation footprint was proportionate to the evidence gathered and the task's demands.
D5 — Answer discipline: Organization, relevance, and clarity of the final answer.

Model Used

Task family again predicted the useful context strategy more reliably than language or repository — with a sharper split inside structural-native tasks this cycle.

Task family	What it asks	Best-supported mode	Why
Behavioral / semantic	What happens, in what order, under which conditions?	Source-backed analysis	Method bodies, branches, assertions, and runtime semantics are decisive.
Mixed	What is structurally affected, and what behavior makes that matter?	PViz-assisted investigation	The bundle supplies structural metrics and orientation; source verifies behavioral detail.
Structural-native	What is connected, risky, cyclic, or widely depended on?	PViz Solo for topology questions; PViz-assisted when cycle/coupling detection is decisive	Topology metrics (importer counts, risk scores) are directly represented in the bundle, but cycle detection can be silently wrong when a wildcard import or intra-package reference is not fully resolved — bundle-only analysis has no way to detect this on its own.

Answer quality and operating score are reported separately

Condition	D1 + D2	Total D1–D5	Interpretation
Raw source exploration (historical baseline)	192/196 (98.0%)	407/420 (96.9%)	Historical near-ceiling source baseline reused from the prior evaluation cycle on these same seven repositories; not re-run this cycle by design and shown here as reference context, not a strictly controlled arm of this cycle's comparison.
PViz-assisted investigation	196/196 (100.0%)	414/420 (98.6%)	Matched or exceeded the reused historical Raw baseline's total score in every language this cycle, a reversal from the prior cycle; D1+D2 correctness was already at parity-or-better against that baseline in both cycles and remains perfect within this cycle's own PViz sessions.
PViz Solo	133/196 (67.9%)	344/420 (81.9%)	Strong, consistently well-calibrated structural orientation, with markedly widened variance on structural-native tasks this cycle — near-parity with PViz on one language's structural-native task, this cycle's lowest single-task score on another.

What each mode is good for

Raw source exploration (historical baseline)

PViz-assisted investigation

Source-grounded investigation with structural navigation, graph metrics, and scoped verification. New sessions this cycle.

PViz Solo

Architectural orientation, structural risk discovery, and explicitly bounded first-pass analysis. New sessions this cycle.

Task-level records make the aggregate findings inspectable

Example task packet

→Task packet — prompt, ground truth, assessment, scores, and anomaly notes where applicable.
→Condition records — full transcripts and run metadata for Raw, PViz-assisted, and PViz Solo.
→28 task packets — one for each of the 7 repositories × 4 tasks in the benchmark.

Browse the full evidence explorerFilter and inspect all 28 task packets, including scores, transcripts, metadata, ground truth, and assessment records.

Example

Java · Micrometer

Structural-native

Task D — Blast Radius and Coupling Analysis for MeterRegistry.java

Task packet

Ground truth ↗Assessment ↗Scores ↗Anomalies ↗Task methodology ↗

Condition records

raw

Transcript ↗Metadata ↗

pviz

Transcript ↗Metadata ↗

solo

Transcript ↗Metadata ↗

What this benchmark does and does not establish

Scope set before evaluation

These boundaries were part of the evaluation design and define the claims the benchmark was intended to support.

One run per condition per cycle: this is an auditable evaluation set, not a variance study across repeated stochastic trials. A rerun of any condition, including Raw, could plausibly produce a different result even against unchanged source.
The Raw condition was not re-run this cycle. Its scores are reused from the prior evaluation cycle on these same seven repositories; see the Raw condition's own description for why it is presented as a historical reference baseline rather than a strictly controlled arm of this cycle.
This cycle's PViz/Solo task prompts were independently redrafted from the prior cycle. Exact wording equivalence with the reused Raw baseline's original prompts has not been verified — the two cycles target the same 28 task slots, families, and repository files, but not confirmed-identical prompt text. Comparisons against Raw should be read accordingly.
The 28-task mix remains heavily weighted toward behavioral-semantic work (21 of 28 tasks this cycle), where source access is structurally decisive.
Static structure cannot substitute for runtime behavior, implementation ordering, or test assertions.
Results should be interpreted by task family rather than as a universal ranking of context strategies.

Identified during evaluation

These findings emerged while running or scoring the benchmark and affect how specific results should be interpreted.

Two harness-level inefficiencies identified in the prior cycle (a Ruby PATH failure and a TypeScript bundle re-parsing overhead) did not reappear this cycle; this is the primary driver of PViz's improved total score against the reused Raw baseline.
Task-family fit mattered more than language or repository when determining which context strategy was useful.
The bundle's contribution varied by task: it could improve navigation, expose structural metrics, or — in four of the seven structural/mixed tasks this cycle — catch the bundle's own structural errors via a source cross-check.