Addendum · reference analyzer preview, not yet enabled in the hosted product
This page re-examines Scrapy: Structural Evolution Across 68 Iterations. The original page and its evidence are unchanged.
The study's starting point recovered and re-examined; the refactoring sequence itself is not reproducible.
Primary artifacts for auditing the evaluation and its source records.
This page re-examines the April showcase with the PViz reference analyzer, which is not yet enabled in the hosted product. The original report, metrics, snapshots and alignment notes are unchanged. Only the starting point of the study could be recovered; the intermediate refactoring states could not, so this addendum does not reproduce the refactoring sequence.
How was Scrapy's large dependency cycle traced, measured and broken through a sequence of targeted edge removals in a local working copy?
The study used the analyzer of that time (bundle schema v1.1) on a series of iterations of a locally modified Scrapy working copy, and published a report, a metrics file, snapshots of selected iterations and redacted alignment notes.
The first published snapshot (`Iteration 01`) is byte-identical to upstream commit `7010985e4f37` (2026-02-23) for every file it hashed: the study started from unmodified upstream code, and that starting point is reproducible. Later iterations differ from any upstream commit and exist only in the author's working copy, which is not available here. A separately labelled analysis of the current upstream commit `1cd9afb208fc` (2026-10-26) is included; it is not a reproduction of the experiment.
Reference analyzer (preview: not enabled in the hosted product), release-candidate image revision `08823e7d7662`, run offline on the recovered source with the product's own analysis call. The legacy analyzer was run as a control and the `scrapy/` package's imports were parsed independently with Python's `ast` module.
AST: every `import` in the scrapy package, classified by position. The analyzers report a runtime figure equal to their full figure.
| Figure | Published (v1.1) | Source (AST) | Legacy (current) | Reference |
|---|---|---|---|---|
| Largest cycle, all imports (modules) | 66 | 63 | 39 | 63 |
| Largest cycle without TYPE_CHECKING imports | 23 | 23 | 39 | 63 |
| Largest cycle, module-level imports only | — | 19 | — | — |
| Internal import relations in `scrapy/`: missing / extra vs AST | — | 827 | 82 / 1 | 0 / 0 |
The reference analyzer reproduces the package's import relations exactly (827 relations, none missing or extra) and finds the same 63-module cycle as the source. The published starting cycle had 66 files; 3 of them (`scrapy/http/cookies.py`, `scrapy/utils/curl.py`, `scrapy/utils/ossignal.py`) are not in the source's cycle and came from relations the earlier analyzer recorded but the source does not contain. The current legacy analyzer misses 82 relations and reports a 39-module cycle.
The study's runtime figure (23 modules without `TYPE_CHECKING` imports) matches the source exactly. Neither current analyzer reproduces that split: both report the runtime cycle equal to the full one. For this study, which used `TYPE_CHECKING` guards as a refactoring tool, that is a regression relative to the earlier analyzer.
| Original finding | Status | Addendum |
|---|---|---|
| Scrapy began as a single large conceptual cycle spanning orchestration, HTTP, spiders and settings | Retained | Confirmed on the recovered source: 63 modules, the same set in the source and the reference analyzer. |
| Runtime and conceptual coupling diverged: TYPE_CHECKING guards separate them | Retained (starting point only) | At the starting point the source's runtime cycle is 23 modules, as published. The later runtime figures cannot be checked. |
| The cycle was broken through targeted edge removals; one removal collapsed a large part of it | Not re-examined | The intermediate states are not available. |
| (new) Module-level imports | New | Imports executed at module load form a 19-module cycle at the starting point. |
| (new) Call graph and fixtures | New | 19,657 call sites at the starting point, 7,115 resolved inside the repository; 61 pytest fixtures recorded. |
The AST results, both bundles of both commits and the per-relation comparison are in the downloads below.
At `1cd9afb208fc` the source's package cycle has 67 modules (30 without `TYPE_CHECKING` imports, 25 at module load). The reference analyzer reports 65: it misses 31 relations and records 13 that the source does not contain, all from imports of underscore-prefixed modules such as `scrapy.utils._ssl` being attributed to the package's `__init__.py`. The legacy analyzer reports 61.
Only the starting point is comparable. The refactoring sequence, its iteration counts and its per-iteration figures remain the study's own evidence and are not restated or recomputed.
The cycle that motivated the study is real and is reproduced exactly by the reference analyzer at the study's starting point. The earlier analyzer's runtime view, which the study relied on, was accurate at the start and is not reproduced by either current analyzer. Whether the refactoring sequence would be measured the same way today cannot be determined, because its intermediate sources are not available.