Skip to content

perf: defer self-contained bundle creation to first out-of-process execution#743

Merged
stack72 merged 2 commits intomainfrom
fix/lazy-self-contained-bundle
Mar 17, 2026
Merged

perf: defer self-contained bundle creation to first out-of-process execution#743
stack72 merged 2 commits intomainfrom
fix/lazy-self-contained-bundle

Conversation

@stack72
Copy link
Copy Markdown
Contributor

@stack72 stack72 commented Mar 17, 2026

Problem

With 104+ user extension models, swamp model type search (and any other command that loads extensions) took ~46 seconds to start. After the fix in #741 that parallelised the 4 loader types and skipped loading for help/version commands, the remaining bottleneck was in UserModelLoader itself.

At startup, loadModels() eagerly built a self-contained bundle for every model file — a separate deno bundle subprocess per model that inlines all dependencies (including zod) so the bundle can run inside Docker containers without network access. With 104 models, this was 104 sequential subprocesses on every single CLI invocation, regardless of whether Docker execution would ever be used.

Architecture Decision & Tradeoffs

What changed

bundleSource?: string on ModelDefinition (a pre-built JS string stored at load time) has been replaced with bundleSourceFactory?: () => Promise<string> — a memoizing closure that defers the expensive work to the point of actual need.

// Before: runs at startup, for every model, every invocation
modelDef.bundleSource = await bundleExtension(absolutePath, denoPath, { selfContained: true });

// After: closure set at load time, executed only on first Docker execution
let cachedBundle: string | undefined;
modelDef.bundleSourceFactory = async () => {
  if (!cachedBundle) {
    cachedBundle = await bundleExtension(absolutePath, denoPath, { selfContained: true });
  }
  return cachedBundle;
};

Tradeoff: startup cost vs. first-execution cost

Before After
Every CLI invocation Pays bundling cost for all N models Pays nothing
First Docker execution of model A Already paid at startup Pays bundling cost for model A only
Second Docker execution of model A Pre-built Memoized in-process — instant

The first out-of-process execution of a given model will be slightly slower than before — it now bundles on demand rather than having it pre-built. This is the right tradeoff because:

  1. Docker execution is rare relative to everyday CLI usage (type search, model get, data list, etc.)
  2. Cost is proportional to actual need — only the models you actually run out-of-process are ever bundled
  3. Memoization ensures the cost is paid at most once per model per process invocation
  4. The startup tax was paid unconditionally regardless of what command you ran — even read-only commands that never touch Docker

Why only models?

Vaults, drivers, and datastores do not create self-contained bundles at all — they only use the externalized (cached) bundle for in-process execution. This change is model-specific because only models support out-of-process/Docker execution via bundleSource.

User Impact

Measured on a real repo with 104 extension models:

Command Before After
swamp model type search aws ~46 seconds ~2.8 seconds
swamp model get my-model ~46 seconds ~2.8 seconds
swamp data list ~46 seconds ~2.8 seconds
First Docker model run Instant (pre-built) ~same as before (built on demand)

16x speedup for the everyday command path. The remaining ~2.8s is the warm-cache cost of the externalized bundle loading (disk reads + dynamic imports for 104 files), which is a separate optimization opportunity.

Files Changed

File Change
src/domain/models/model.ts bundleSource?: stringbundleSourceFactory?: () => Promise<string>
src/domain/models/user_model_loader.ts Remove eager bundling, set memoizing factory closure
src/domain/models/method_execution_service.ts Await bundleSourceFactory?.() at execution time

Test Plan

  • deno check — type checking passes
  • deno lint — no lint errors
  • deno fmt --check — formatting correct
  • deno run test — 3175 tests passed, 0 failed
  • deno run compile — binary compiled successfully
  • Manual: swamp model type search in 104-model repo — 46s → 2.8s

…ecution

Previously UserModelLoader eagerly built a self-contained (fully inlined)
bundle for every model at startup, regardless of whether Docker/out-of-process
execution would ever be used. With 104+ models this added ~43 seconds of
sequential deno bundle subprocess calls to every CLI invocation.

Replace bundleSource?: string on ModelDefinition with
bundleSourceFactory?: () => Promise<string> — a memoizing closure set at
load time that defers the expensive bundleExtension({ selfContained: true })
call to the first actual out-of-process execution. Subsequent executions of
the same model within a process hit the in-memory memo and are instant.

Tradeoff: first Docker execution of a model incurs the bundling cost that
was previously paid at startup. This is acceptable — Docker execution is
rare relative to CLI usage, and the cost is now proportional to actual need.

Measured improvement: 104-model repo, swamp model type search
  Before: ~46 seconds
  After:  ~2.8 seconds (16x faster)

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
github-actions[bot]

This comment was marked as outdated.

github-actions[bot]

This comment was marked as outdated.

… bundling failure

- Move bundleSourceFactory call inside the non-raw driver branch so raw
  executions never trigger self-contained bundling (HIGH #2)
- Switch to promise-based memoization to avoid duplicate concurrent builds
- Restore error handling: log a warning and rethrow on bundling failure,
  resetting the cached promise so subsequent calls can retry (HIGH #1)

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Copy link
Copy Markdown

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

This is a well-implemented performance optimization that defers self-contained bundle creation from startup time to first out-of-process execution (Docker). The change delivers a 16x speedup for everyday CLI commands while maintaining correctness.

No Blocking Issues

Code Quality & TypeScript Strict Mode:

  • No any types introduced
  • All exports are named exports (per CLAUDE.md requirements)
  • Proper typing with bundleSourceFactory?: () => Promise<string>

Domain-Driven Design:

  • bundleSourceFactory is appropriately placed on ModelDefinition as model metadata
  • Factory invocation happens in MethodExecutionService (Domain Service) - correct orchestration boundary
  • No persistence concerns leaked into the domain model

Implementation Quality:

  • Promise-based memoization (bundlePromise ??=) correctly handles concurrent calls, avoiding duplicate bundling work
  • Error handling properly resets bundlePromise = undefined on failure, allowing retries
  • Logging preserved via logger.warn on bundle failure
  • Bundle population moved to only non-raw drivers (method_execution_service.ts:582-586), avoiding unnecessary work for in-process execution

Security: No new security concerns - bundling logic unchanged, just deferred.

Test Coverage: PR description confirms 3175 tests pass. The existing comprehensive tests for user_model_loader_test.ts and method_execution_service_test.ts cover the behavioral contract, which hasn't changed - only the timing of when bundling occurs.

Suggestions (non-blocking)

  1. Documentation update opportunity: design/execution-drivers.md:171 still references bundleSource rather than bundleSourceFactory. This is a minor documentation artifact that could be updated in a follow-up.

LGTM - clean implementation of a valuable performance optimization.

Copy link
Copy Markdown

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial Review

I systematically attempted to break this code across logic, error handling, concurrency, and API contract dimensions. The implementation is solid.

Critical / High

None found.

Medium

  1. Interface change on public type (model.ts:606): ModelDefinition.bundleSource?: string was replaced with bundleSourceFactory?: () => Promise<string>. While ModelDefinition is exported, this field was only ever set by UserModelLoader internally and consumed by MethodExecutionService. External code directly inspecting bundleSource would now find it undefined. The risk is low since this was an implementation detail, but it's technically a breaking change to the interface shape.

Low

  1. Model file deletion timing (user_model_loader.ts:357-369): The closure captures absolutePath at load time but bundling happens at first Docker execution. If a user deletes the model file between these events, bundling will fail. This is a minor behavioral change from eager bundling (which would have failed at load time instead). In practice this is rare and the error message is clear.

  2. Non-null assertion in Docker driver (docker_execution_driver.ts:273): The driver uses request.bundle! without null check. If bundleSourceFactory is undefined (e.g., built-in models) and someone attempts Docker execution, they'll get a runtime crash. However, this is unchanged from the prior behavior—built-in models never had bundleSource set either. The error timing is just different now.

Verification of Memoization Pattern

The promise-based memoization is correct:

let bundlePromise: Promise<string> | undefined;
modelDef.bundleSourceFactory = () => {
  bundlePromise ??= bundleExtension(...).catch((error) => {
    bundlePromise = undefined;  // Allow retry on failure
    throw error;
  });
  return bundlePromise;
};
  • Concurrent calls share the same promise (no duplicate bundling)
  • On failure, bundlePromise resets to undefined, allowing retry
  • Each model's closure has its own bundlePromise (loop creates new scope each iteration)

Error Semantics Change

Old: Bundle failure at load time → warning logged → bundleSource undefined → Docker execution later fails at driver

New: Bundle failure at execution time → error thrown immediately with clear message

This is better behavior—you get an actionable error message rather than a mysterious driver failure when bundle is missing.

Verdict

PASS — Clean implementation with correct memoization, proper error propagation, and no regressions. The performance improvement (46s → 2.8s startup) is significant with no observable downside for users.

@stack72 stack72 merged commit 57baec4 into main Mar 17, 2026
6 checks passed
@stack72 stack72 deleted the fix/lazy-self-contained-bundle branch March 17, 2026 22:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant