Scrapy: Structural Evolution Across 68 Iterations

Scrapy was selected as a showcase candidate because it is a mature, well-maintained Python framework with a known history of complex internal dependencies. The initial intent was straightforward: run PViz against the codebase, identify the largest dependency cycles, and document a refactor campaign that reduced them.

The refactor work began against the conceptual SCC. PViz reported a 66-node cycle at baseline, and the work that followed was aimed at breaking it. After more than 25 iterations, the conceptual SCC had not moved. The number held at 66, then climbed briefly to 68, and stayed there. From the outside, the analysis looked like a complete stall — a 68-node cycle across 25+ iterations of active modification with nothing to show for it.

The problem was that the refactor steps were not targeting the conceptual graph. They were targeting runtime coupling — moving imports behind TYPE_CHECKING guards, restructuring execution-time dependencies, cleaning up the import chains that caused real problems at runtime. That work was succeeding. But PViz was only reporting one number, and that number was not the one being improved.

The dual-metric system — separating conceptual SCC from runtime SCC and introducing TC inflation as the signal for the gap between them — was built directly in response to this. The question it was designed to answer was simple: if the refactor steps are clearly doing something, why does the metric show nothing? The answer turned out to be that the metric and the work were measuring different layers of the same system. Once the two were separated, the stall resolved into a coherent picture: meaningful runtime progress running in parallel with an unchanged architectural graph, exactly as the data now shows.

The 68 iterations documented here are the complete trace, starting from the original baseline and running through the point where both layers were substantially resolved. The framework used to analyze them did not exist at the start. It was built because the work demanded it.

This analysis traces the structural evolution of the Scrapy codebase across 68 iterations using dependency graph analysis. The goal is not to evaluate code quality or runtime performance, but to measure how structural coupling changes under deliberate modification — and to identify where the system resists change.

The codebase is analyzed as a directed dependency graph: each module is a node, each import relationship is an edge. From this graph, four structural metrics are tracked at every iteration.

**Largest conceptual SCC** — the size of the largest strongly connected component in the full import graph, including TYPE_CHECKING annotations. A module inside this component cannot be fully understood or isolated without implicating every other module in the same cycle.

**Largest runtime SCC** — the size of the largest cycle in the runtime-only graph, excluding TYPE_CHECKING edges. This reflects coupling that affects actual import execution, not just type annotations.

**Cycle nodes** — the total count of modules participating in any cycle, across all SCCs. While the largest SCC represents the worst-case local entanglement, cycle nodes represent the overall footprint of cyclic dependency across the system. A reduction in largest SCC without a reduction in cycle nodes indicates redistribution rather than elimination of coupling.

**TC inflation** — the difference between conceptual and runtime cycle membership. A high TC inflation value means a large portion of the coupling exists only in type annotations — real architectural dependency, but not one that affects the interpreter at runtime.

At iteration 1: conceptual SCC = 66, conceptual cycle nodes = 70, runtime SCC = 23, runtime cycle nodes = 30, TC inflation = 43.

The two SCCs are not the same population. The orchestration core — crawler.py, engine.py, scraper.py, scheduler.py, spidermw.py — is fully embedded in the 66-node conceptual cycle, but carries a runtime SCC size of 1 at baseline. These modules are architecturally entangled through type annotations and import structure, but their execution-time coupling is already bounded.

The runtime cycle runs through the HTTP and spider layers: http/__init__.py, spiders/__init__.py, http/response/__init__.py, spiders/crawl.py, and linkextractors/lxmlhtml.py all carry runtime SCC = 23 at iteration 1.

This distinction — present from the very first measurement — means the system has two different coupling problems at the outset, operating on different layers of the stack. Runtime coupling is driven by import execution and is solvable through import control mechanisms such as deferral or TYPE_CHECKING guarding. Conceptual coupling is driven by dependency structure and is solvable only through structural change. These are not interchangeable problems, and improvements in one layer do not imply improvement in the other.

From a graph perspective, the baseline resembles a single dense cluster with embedded substructures: multiple subsystems — orchestration, HTTP, spiders, settings — fused into one large strongly connected component, with an additional runtime-visible cycle concentrated in the HTTP and spider interaction layer.

The first meaningful structural event in the Scrapy trace occurs at iteration 10, when a single file change collapses the runtime SCC from 23 to 8. The mechanism is TYPE_CHECKING guarding: runtime imports are moved behind annotation-only blocks, removing them from the execution graph without removing them from the dependency structure.

Over the following four iterations the runtime SCC continues to fall: 8 → 6 → 5 → 4, settling at 4 by iteration 14 where it holds through iteration 16. In the same window, the conceptual SCC does not move. It sits at 68 from iteration 7 onward and remains there through the end of this phase. Conceptual cycle nodes hold at 70 across every iteration from 7 through 16. This stability confirms a critical point: annotation-only dependencies still enforce structural coupling, even when removed from runtime execution.

TC inflation tracks the divergence directly. At iteration 9, TC inflation is 45. At iteration 10 it jumps to 60 — a 15-point increase in a single step. By iteration 14 it reaches 64, where it stabilizes. What this captures is the mechanism in quantitative form: each dependency moved behind TYPE_CHECKING exits the runtime graph and widens the gap between the two structural layers. This widening gap reflects a growing layer of hidden architecture — dependencies that no longer affect execution but still dictate how modules relate to one another. The conceptual SCC doesn't shrink because the coupling hasn't been removed — it has been reclassified.

This produces a meaningful but narrow result. Execution-time import cycles are reduced. Import-order sensitivity decreases. Runtime behavior is genuinely cleaner. But the dependency graph, evaluated as a whole, shows no architectural change. The 68-node conceptual SCC represents the same set of mutually entangled modules it did before iteration 10. The system appears cleaner at runtime, but remains structurally entangled.

The data makes the distinction concrete: runtime cycle nodes fall from 23 to 4 across this phase while conceptual cycle nodes do not move. Both metrics are measuring real structural properties — they simply measure different layers of the same system. Runtime SCC reflects what the interpreter encounters at execution time; conceptual SCC reflects what a developer encounters when reasoning about the codebase: which modules depend on which, and where changes will propagate.

Progress on one does not imply progress on the other. That is the central finding of this phase, and the lens through which the rest of the trace should be read.

The first true architectural event follows a period of conceptual stability. Between iterations 14 and 16, the conceptual SCC holds at 68 while runtime continues to stabilize. At iteration 17, that changes.

The largest conceptual SCC drops from 68 to 44. Total edges in the graph change from 1631 to 1630. One edge was removed.

The structural effect is disproportionate to the intervention. A single dependency relationship — a TYPE_CHECKING-only annotation edge — was responsible for holding 24 modules inside the largest cycle. Not all edges contribute equally to structural coupling; some act as bridges between otherwise separable clusters, completing the last path that keeps a large set of modules mutually reachable. Removing such an edge does not merely reduce connectivity — it reorganizes the graph. This is what happened at iteration 17: one well-placed removal caused those 24 modules to become separable from the main cluster.

This indicates that the original 68-node SCC was not uniformly dense. It was composed of multiple dense subgraphs connected by a small number of critical edges — and their removal exposes the latent boundaries between those subgraphs. The intervention required was not a refactoring campaign but the identification and removal of one specific connection.

TC inflation falls in the same step, from 64 to 40. This confirms the nature of the removed edge: it was a TYPE_CHECKING-only relationship, carrying no runtime weight, but sufficient to maintain conceptual entanglement across a large portion of the graph. Its removal doesn't just reduce the SCC — it also narrows the gap between conceptual and runtime structure.

The split is not uniform. Most of the tracked modules — crawler.py, engine.py, scraper.py, spidermw.py, scheduler.py, spiders/__init__.py, settings/__init__.py — land in the 44-node residual SCC. But http/__init__.py and http/response/__init__.py land at conceptual SCC = 20, forming a separate, smaller cycle rather than remaining in the main cluster. The break produced two distinct structures where one had existed before, revealing substructure that the original SCC had been concealing.

Runtime SCC is unchanged throughout. Every tracked module carries runtime SCC = 1 at iteration 17, identical to iteration 16. The execution graph is not touched. What changed is purely architectural — the coupling visible in the import structure and type annotations, not in runtime behavior.

This event establishes a key principle that recurs throughout the trace: structural change is non-linear and edge-sensitive. A single well-placed change can have system-wide impact, while large numbers of low-leverage changes may produce none at all.

Between iterations 27 and 50, the largest conceptual SCC holds at 30 modules for 24 consecutive iterations. TC inflation sits at 26. Runtime SCC sits at 4. Conceptual cycle nodes hold at 55. None of these numbers move.

The graph itself is not static. Total edges fluctuate across the plateau, ranging from 1615 to 1631 as modifications are made and partially reverted. At iteration 35, the conceptual SCC briefly rises to 31 — a new dependency added — before snapping back to 30 at iteration 36 when the change is reversed. The system responds to changes locally but preserves its overall structure. This edge churn demonstrates that the system is under active modification; these changes do not produce structural reorganization.

This is the plateau's primary diagnostic value: it shows that the 30-node cluster is not sensitive to the kinds of changes being applied during this period. This stands in direct contrast to the iteration 17 event, where a single edge removal reorganized 24 nodes out of the largest cycle. Here, across 24 iterations and continuous edge churn, no equivalent leverage point is found. The remaining structure absorbs modification without reorganizing, which reinforces the non-linear nature of structural evolution: it is not the volume of changes that determines structural impact, but whether those changes reach the edges that are actually holding the cycle together.

crawler.py characterizes this resistance. At iteration 27, it carries 160 importers and 17 outbound dependencies, with conceptual SCC = 30 and runtime SCC = 1. By iteration 50, importers have decreased slightly to 153 — but outbound dependencies have grown from 17 to 21, and SCC remains 30. The module is being actively modified throughout the plateau and is accumulating new responsibilities. This pattern reflects centralization under pressure: instead of decoupling, the module becomes more connected while remaining structurally embedded. As it takes on more coordination responsibilities, it becomes harder to extricate — which is precisely why the kernel holds. Its structural position does not change because its functional role is expanding to fill it.

This is what the architectural kernel looks like from the outside: a cluster that persists across iterations not because it is being ignored, but because it is load-bearing. The modules inside it — crawler.py, engine.py, scraper.py, and the components that depend on them — are coordinating the framework's core responsibilities. Their mutual dependencies are not accidental coupling that survived cleanup. They reflect how the system is designed to operate.

The distinction matters for how the plateau is interpreted. An SCC that stalls because no work is being done would look the same on the metrics. What distinguishes a kernel is that the plateau persists despite active modification — edges added, edges removed, modules changed — while the cluster holds its shape. The 30-node SCC survives 24 iterations of pressure without fracturing. That is the definition of structural resistance, and it marks the boundary between coupling that can be untangled incrementally and coupling that requires a fundamentally different intervention.

At iteration 51, seven edges are removed from the graph. The largest conceptual SCC drops from 30 to 20. crawler.py, engine.py, scraper.py, and spidermw.py — the orchestration core that anchored the kernel through 24 iterations of resistance — exit the shared cycle entirely, each reaching SCC = 1 in a single step.

Like the single-edge removal at iteration 17, these seven edges were high-leverage connections — the specific dependencies sustaining the kernel's structure. This intervention represents a targeted structural break, not incremental cleanup. The break is not staged. It happens at once. Seven edge removals produce a structural reorganization that 24 iterations of prior modification could not achieve. The modules that exit do so completely: from SCC 30 to SCC 1, with no intermediate state. This abrupt transition reinforces the earlier observation that structural change is non-linear and occurs in discrete collapses rather than gradual erosion. A second group — linkextractors, settings/__init__, spiders/__init__, spiders/base, spiders/crawl — moves from 30 to 17, falling into a smaller residual cluster rather than achieving full independence. TC inflation drops from 26 to 16 in the same step, reflecting that the removed edges were once again primarily annotation-based.

What follows is a series of smaller reductions. The 20-node SCC holds through iteration 57, then falls to 17 at iteration 58 as HTTP and response handling is restructured. It reaches 15 at iteration 62, where the runtime SCC simultaneously drops from 4 to 2 — the last runtime cycle outside the HTTP layer resolved. By iterations 66–67, the conceptual SCC has reached 13 and TC inflation has fallen to 11.

The final state at iteration 67 is structurally distinct from the baseline in every tracked dimension. crawler.py carries SCC = 1 and runtime SCC = 1, while still holding 147 importers — it remains the most-imported module in the framework, confirmed as ranking first among all modules by importer count, but its own dependencies are now directional rather than mutually entangled. engine.py, scraper.py, spidermw.py, scheduler.py, and downloader/__init__.py all reach SCC = 1 at both layers. The orchestration core that defined the system's structural identity from iteration 1 is fully acyclic.

The residual 13-node conceptual SCC is localized within the HTTP layer: http/__init__.py and http/response/__init__.py both carry SCC = 13 and runtime SCC = 1. This remaining cycle is localized — confined to a single subsystem — non-runtime, driven by annotation-only edges as confirmed by the absence of any runtime cycle in either module, and bounded, no longer spanning the system. It represents a shift from system-wide entanglement to contained structural coupling.

Iteration 68 introduces a slight uptick: 15 edges are added within the HTTP layer, expanding both modules from SCC 13 to SCC 15 and raising TC inflation from 11 to 13. Runtime structure remains unchanged. This is a localized regression, not a systemic one. It does not reintroduce cross-layer coupling or affect the overall architecture. The system remains decomposed into independent components with a single confined conceptual cluster.

The runtime graph, which opened at 23-node coupling, closes at 2 nodes. The system that began as a single 66-node cycle spanning orchestration, HTTP, spiders, settings, and utilities ends as a set of independent components with one localized, annotation-only cluster remaining. The coupling that persists is structurally visible and confined to a single layer. The coupling that was broken — across the orchestration core — required identifying the seven edges that were holding it together, not a campaign of incremental cleanup across the rest of the codebase.

Across the full 68-iteration trace — from the 66-node baseline at iteration 1 to the 13-node HTTP-layer residual at iteration 67 — three distinct categories of change emerge, each with a different relationship to structural outcome.

**Execution-level changes**, such as TYPE_CHECKING guarding, reduce runtime SCC without affecting conceptual structure. They are visible in the TC inflation signal: as runtime edges are reclassified as annotation-only, TC inflation rises while the conceptual SCC holds flat. The gap between iteration 9 (TC inflation 45) and iteration 14 (TC inflation 64) is the clearest instance of this pattern.

**Low-leverage structural changes** — incremental edge additions and removals that modify the graph without altering SCC boundaries — are the dominant activity during the plateau. Across iterations 27–50, total edges range from 1615 to 1631 while every structural metric remains fixed. These changes are real modifications to the codebase, but they do not reach the edges that are actually closing the cycle.

**High-leverage structural changes**, specifically the removal of cycle-closing edges, produce non-linear reorganization. The single edge removed at iteration 17 collapsed 24 nodes out of the conceptual SCC. The seven edges removed at iteration 51 broke the kernel entirely. In both cases, the magnitude of structural change was disproportionate to the number of edges touched — because those edges were the ones completing the cycle, not merely contributing to it.

The evolution documented here is not gradual. It is edge-sensitive and non-linear, characterized by long periods of resistance punctuated by abrupt structural shifts. Each step change corresponds not to a sustained campaign of modifications, but to a small number of changes that happened to reach the right edges.

The final system does not eliminate all coupling. Instead, it constrains coupling to a specific layer, making it visible, bounded, and structurally understandable — which is a materially different condition from where it began.

The structural analysis ends at iteration 67. What happened next is worth documenting separately, because it reveals something the dependency graph cannot show on its own.

The pre-refactor baseline passed 3565 tests. One failed — a pre-existing, unrelated issue. That is the ground truth the refactor had to return to.

Iteration 67 represented the structural endpoint: orchestration core fully acyclic, residual conceptual SCC confined to the HTTP layer, runtime SCC at 2. Running the test suite against that state produced immediate, widespread failures. Not edge cases — the suite failed inside the first 8 percent of tests, and the failures were structural in character.

The first revision surfaced a broken interface contract: SpiderMiddlewareManager was receiving an unexpected keyword argument (`spider_getter`) that the refactored adapter was injecting but the manager's `__init__` no longer accepted. The structural decoupling had changed how components were wired together without updating the full interface surface.

The second revision shifted the failure: `_CrawlerAdapter` was missing a `.stats` attribute. An adapter introduced to reduce coupling between the crawler and its consumers was not exposing the full contract that downstream middleware expected at runtime. The third revision hit the same class of problem in a different adapter: `_SchedulerCrawlerAdapter` was missing `.engine`.

The pattern across the first three revisions was consistent: the structural changes had introduced adapter objects as intermediaries between previously coupled components, but those adapters were incomplete. They decoupled the import graph without fully replicating the runtime interface. The test suite found every gap.

Revisions four and five shifted from interface failures to behavioral ones. With the adapters corrected, 61 failures appeared in the fourth run — engine slot assignment errors, spider start failures, response caching breakage, deprecation contract violations. By the fifth run that had narrowed to 14, concentrated in a broken import path (`scrapy.core.engine.Scraper` was no longer a valid module reference after the engine restructuring) and two remaining adapter gaps in the `Downloader` constructor.

Revisions six and seven closed the remaining failures. The sixth run returned 3537 passed with 0 failures — clean, though 28 fewer passing tests than the pre-refactor baseline. The collected count had also shifted back to 3838 from the 3667 collected in revisions 4 and 5, and the skip count returned to 282, suggesting test collection was stabilizing but not yet fully restored. The seventh and final run reached 3566 passed, 282 skipped, 32 xfailed, collected 3867: one net test gained over the pre-refactor baseline of 3565, with the pre-existing failure resolved as a side effect of the refactor.

The iteration 68 uptick in the structural metrics — 15 edges added, conceptual SCC expanding from 13 to 15 — reflects this recovery work. Some of the edges removed in the HTTP layer restructuring were load-bearing at runtime even though they appeared annotation-only in the conceptual graph. Restoring them was not a failure of the analysis; it was the test suite doing what structural analysis cannot: verifying that the system still behaves correctly after the structure changes.

The final state is not identical to iteration 67. It is better in the ways that matter: the orchestration core is acyclic, the runtime graph is reduced, and the test suite passes. The two edges added back to the HTTP layer are the cost of making that true in a working system rather than a dependency graph.