Fixes: #1053

Summary

This PR adds lazy per-bundle loading to the model registry so individual extension bundles are imported on demand rather than all at once. This is Phase 2 of the lazy loading optimization started in #1050.

The Problem

After #1050 (lazy per-registry loading), commands that need model extensions still load all model bundles at once via modelRegistry.ensureLoaded(). With @swamp/aws/ec2 + @swamp/aws/s3 installed (116 bundles), every model command takes 3-6 seconds because it imports all 116 bundles even when only 1 type is needed. This scales linearly — every new extension installed makes every command slower.

The Solution

An ExtensionCatalogStore (SQLite database at .swamp/_extension_catalog.db) indexes all known extension types without importing any JavaScript. The loading flow becomes:

1. On startup, ensureLoaded() calls buildIndex() which reads the catalog and does a lightweight mtime scan (directory walk + stat calls, no imports). Types are registered as lazy entries — the registry knows they exist but hasn't imported their bundles.
2. types() returns both loaded and lazy entries — model type search works with zero bundle imports.
3. When a specific type is needed (e.g. model get, model create), ensureTypeLoaded(type) queries the catalog for the bundle path, imports just that one bundle, and also imports any extension bundles that target the base type (for modelRegistry.extend() support).
4. First run after install (catalog doesn't exist): falls back to the existing full-import path, then populates the catalog. One-time cost, same speed as before.

Performance Results

Benchmarked with @swamp/aws/ec2 + @swamp/aws/s3 (116 bundles):

The key insight: these numbers stay constant regardless of how many extensions are installed. Before this change, installing more extensions made every command proportionally slower.

What Changed

New Files

• src/infrastructure/persistence/extension_catalog_store.ts — SQLite-backed catalog for extension type metadata. Completely independent of CatalogStore (data queries) — separate DB file, separate class, no shared state. Schema includes a kind column (model, extension, vault, driver, datastore, report) so the same store supports all registry types.
• src/infrastructure/persistence/extension_catalog_store_test.ts — 14 tests including concurrent-open test (busy_timeout before WAL pragma, per fix: set SQLite busy_timeout before WAL pragma to prevent concurrent locking errors #1057).

Modified Files

• src/domain/models/model.ts — ModelRegistry gains lazy entry support:

  • LazyModelEntry type — metadata for an unloaded type (type name, bundle path, version)
  • registerLazy() — registers a type as known-but-not-imported
  • ensureTypeLoaded(type) — imports a single bundle on demand with per-type promise memoization
  • promoteFromLazy() — converts a lazy entry to a fully loaded definition
  • isLazy() — checks if a type needs loading
  • has() — now returns true for both loaded and lazy types
  • types() — now returns both loaded and lazy type names
  • setTypeLoader() — configures the per-type loader callback

• src/domain/models/model_test.ts — 11 new tests covering lazy entry registration, promotion, type enumeration, ensureTypeLoaded with concurrent callers, and no-op behavior for already-loaded/unknown types.

• src/domain/models/user_model_loader.ts — Two new loading modes:

  • buildIndex() — discovers files, checks mtimes against catalog, rebundles only changed files, registers lazy entries from catalog. Falls back to full import when catalog isn't populated.
  • loadSingleType() — imports one bundle + its extensions from catalog paths.
  • Supporting methods for mtime-based staleness detection, catalog population from registry, and direct bundle-by-path import.

• src/cli/mod.ts — loadUserModels() now creates an ExtensionCatalogStore, sets the type loader on the registry, and calls buildIndex() instead of loadModels().

• src/domain/extensions/extension_auto_resolver.ts — resolveModelType() now calls ensureTypeLoaded() before checking the registry, so lazy types are loaded on demand before falling back to auto-resolution.

• src/libswamp/models/get.ts — getModelDef in ModelGetDeps now supports async return (for lazy loading). createModelGetDeps calls ensureTypeLoaded() before get().

• design/extension.md — New "Lazy Per-Bundle Loading" section documenting the architecture, self-healing behavior, and roadmap.

Risks Discussed During Planning

Extension methods going missing

If ensureTypeLoaded("base/type") loads the base bundle but misses an extension bundle that adds methods via modelRegistry.extend(), those methods would silently be absent. Mitigation: The catalog tracks extends_type for extension bundles. loadSingleType() queries findExtensionsForType(baseType) and imports all matching extensions after the base type.

Stale catalog data

If a user edits extension source files, the catalog could serve stale metadata. Mitigation: buildIndex() always runs a directory scan + mtime comparison on every command. Changed files are detected, rebundled, and the catalog is updated. Deleted files are removed from the catalog. No manual intervention needed.

First-run migration

Users upgrading have cached bundles but no catalog. Mitigation: When the catalog's populated flag is false, buildIndex() falls back to the existing full-import path (same speed as before), then populates the catalog from the loaded registry. Self-healing: deleting _extension_catalog.db triggers the same rebuild.

Concurrent access

Multiple swamp processes hitting the same catalog. Mitigation: SQLite WAL mode + busy_timeout=5000 (set before WAL pragma, per #1057 fix). Concurrent-open test verifies two handles can write simultaneously without "database is locked" errors.

Independence from data catalog

The ExtensionCatalogStore must not couple with or affect the data query CatalogStore. Mitigation: Completely separate database file (.swamp/_extension_catalog.db vs .swamp/data/_catalog.db), separate class, no shared code paths, no shared imports.

Forward-Facing Work

• Lazy per-bundle loading for vault, driver, datastore, and report registries #1062 — Wire the remaining registries (vault, driver, datastore, report) through the same ExtensionCatalogStore. The schema already supports all registry types via the kind column. This is incremental wiring work — the catalog and patterns are in place.
• Explicit write-through at extension pull/rm/update mutation points (currently handled implicitly by the mtime scan, but direct catalog updates would be marginally faster).

Test Plan

• 4038 unit tests pass (including 11 new ModelRegistry lazy entry tests and 14 new ExtensionCatalogStore tests)
• deno check, deno lint, deno fmt all clean
• UAT tests from systeminit/swamp-uat verified against the compiled binary:
  • model/type/search_test.ts — 2/2 pass
  • extension/broken_extensions_test.ts — 5/5 pass
  • e2e/extension_model_test.ts — 3/3 pass
  • e2e/model_lifecycle_test.ts — 3/3 pass
  • model/type/describe_test.ts — 3/3 pass
  • extension/pull_test.ts — 4/4 pass
  • model/search_test.ts — 3/3 pass
  • e2e/error_paths_test.ts — 9/9 pass
  • e2e/global_flags_test.ts — 7/7 pass
  • adversarial/state_corruption_test.ts — 6/6 pass
  • adversarial/concurrency_test.ts — 5/6 pass (1 pre-existing flaky failure, reproduces on current release too)
• Performance benchmarked with 116 bundles (@swamp/aws/ec2 + @swamp/aws/s3), results consistent across multiple runs

🤖 Generated with Claude Code