Addendum · reference analyzer preview, not yet enabled in the hosted product
This page re-examines Repository Context Strategies After Revision and Repository Context Strategies for Codebase Analysis Agents. The original page and its evidence are unchanged.
What the reference analyzer adds to the experiments' inputs, without changing any score.
Primary artifacts for auditing the evaluation and its source records.
This page does not change any score from the June or July evaluations and does not claim that richer analyzer output would change task results. It re-analyzes the 7 repositories at the exact commits used as experiment inputs and checks, for each structural task, whether the reference analyzer supplies information the ground-truth cards recorded as missing from the bundle. The reference analyzer is not yet enabled in the hosted product.
Which repository-context strategy (raw source exploration, PViz-assisted source reads, or bundle-only analysis) suits which task family?
June used bundles from the analyzer of that time (schema v1.2, published as inputs). July used bundles from the current legacy analyzer (schema v1.3, generated in July and not published). Both published transcripts, scores and ground-truth cards.
The commit behind each June input bundle was recovered by matching the bundle's file list and file contents to upstream history: 4,282 analyzed files checked, 0 differ. Running today's legacy analyzer on those commits reproduces the July ground-truth figures (for example Micrometer's 95 importers and Textual's 66-module cycle), so the July bundles used the same commits.
| Repository | Commit | Date |
|---|---|---|
| go-caddy | 39c9a85f80a3 | 2026-06-13 |
| java-micrometer | 8466feab03a1 | 2026-06-08 |
| kotlin-kotlinx | a4c6af96c15f | 2026-05-07 |
| python-textual | b24ef07d422b | 2026-05-19 |
| ruby-huginn | 9751c379dd94 | 2026-06-11 |
| rust-ripgrep | 43e2f08edef8 | 2026-06-05 |
| typescript-vite | 456901bb64de | 2026-06-15 |
Reference analyzer (preview: not enabled in the hosted product), release-candidate image revision `08823e7d7662`, run offline on the recovered source with the product's own analysis call. The legacy analyzer was run on the same commits as a control.
| Repository | Gap in the card | Reference analyzer at the same commit | Status |
|---|---|---|---|
| Caddy (Go) | No call-frequency data; symbol usage only by grep | `RegisterModule` (83 files); `NewReplacer` (40 files); `ReplaceAll` (33 files) | Closed: call data available |
| Micrometer (Java) | Missed the MeterRegistry ↔ Search/RequiredSearch cycle (wildcard import); 95 importers | Back-references resolved; MeterRegistry is in a 29-class cycle with the search classes; 170 referencing files (legacy 95). | Closed |
| kotlinx.coroutines (Kotlin) | Same-package users of Job.kt invisible | `JobSupport.kt` and `AbstractCoroutine.kt` now linked to `Job.kt` (implements); 123 referencing files (legacy 63). Name-level links stay out of cycle detection. | Partly closed |
| Textual (Python) | Cannot tell TYPE_CHECKING from runtime imports; no call edges | Call data available (`run` (58 files); `run_test` (38 files)). Cycle reported as 121 modules (legacy 66); the source has 141, 84 without TYPE_CHECKING imports, 70 at module load. No runtime split in either analyzer. | Partly closed |
| Huginn (Ruby) | 12-node cycle from the source; `receive!` said to have a single caller | Reference reproduces the 12-node cycle (legacy 10). The call graph lists more callers of `Agent.receive!` than the card: `event.rb` and `agents_controller.rb` call it too (checked in the source). | Closed; card corrected |
| ripgrep (Rust) | Bundle missed 3 importers of the ignore crate in `crates/core` | Unchanged: 7 importers; cross-crate `use ignore::...` paths are not resolved. | Open |
| Vite (TypeScript) | Cannot distinguish type-only imports; no per-export usage | Unchanged: 84-file cycle, no type-only marking, almost no resolved calls into `config.ts`. | Open |
The reference analyzer adds function-level call data in all 7 languages, resolves Java and Kotlin references that need no import, and reproduces Huginn's full cycle. It does not resolve Rust cross-crate imports, does not mark TypeScript type-only imports, and, like the legacy analyzer, does not separate `TYPE_CHECKING` imports in Python.
Each row was checked against the bundle of the same commit; the Micrometer back-references, the Huginn callers and the Textual AST figures were checked against the source. The checks, run records and bundles are in the download below.
This addendum describes analyzer output, not model behaviour. Whether these additions would change answer quality, efficiency or calibration can only be measured by repeating the evaluation with controls. The Raw baseline in July was reused from June, so it cannot serve as a control for a new cycle either.
For the structural tasks, the reference analyzer closes several gaps the ground-truth cards attributed to the bundle: call-frequency data, Java wildcard and same-package references, and Huginn's full cycle. Others remain: Rust cross-crate use, TypeScript type-only imports and Python `TYPE_CHECKING` separation. The original conclusion that task family predicts the useful context strategy is unaffected; this page provides no evidence either way about scores.
Re-run the structural-native and mixed tasks with reference-analyzer bundles, with a same-cycle Raw arm, identical prompt text across arms, and at least 3 runs per condition; score blind to condition with the existing rubric; report per-task deltas with intervals. Publish the input bundles, prompts and transcripts as in June.