perf: defer self-contained bundle creation to first out-of-process execution#743
perf: defer self-contained bundle creation to first out-of-process execution#743
Conversation
…ecution
Previously UserModelLoader eagerly built a self-contained (fully inlined)
bundle for every model at startup, regardless of whether Docker/out-of-process
execution would ever be used. With 104+ models this added ~43 seconds of
sequential deno bundle subprocess calls to every CLI invocation.
Replace bundleSource?: string on ModelDefinition with
bundleSourceFactory?: () => Promise<string> — a memoizing closure set at
load time that defers the expensive bundleExtension({ selfContained: true })
call to the first actual out-of-process execution. Subsequent executions of
the same model within a process hit the in-memory memo and are instant.
Tradeoff: first Docker execution of a model incurs the bundling cost that
was previously paid at startup. This is acceptable — Docker execution is
rare relative to CLI usage, and the cost is now proportional to actual need.
Measured improvement: 104-model repo, swamp model type search
Before: ~46 seconds
After: ~2.8 seconds (16x faster)
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
… bundling failure - Move bundleSourceFactory call inside the non-raw driver branch so raw executions never trigger self-contained bundling (HIGH #2) - Switch to promise-based memoization to avoid duplicate concurrent builds - Restore error handling: log a warning and rethrow on bundling failure, resetting the cached promise so subsequent calls can retry (HIGH #1) Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
There was a problem hiding this comment.
Code Review Summary
This is a well-implemented performance optimization that defers self-contained bundle creation from startup time to first out-of-process execution (Docker). The change delivers a 16x speedup for everyday CLI commands while maintaining correctness.
No Blocking Issues
Code Quality & TypeScript Strict Mode:
- No
anytypes introduced - All exports are named exports (per CLAUDE.md requirements)
- Proper typing with
bundleSourceFactory?: () => Promise<string>
Domain-Driven Design:
bundleSourceFactoryis appropriately placed onModelDefinitionas model metadata- Factory invocation happens in
MethodExecutionService(Domain Service) - correct orchestration boundary - No persistence concerns leaked into the domain model
Implementation Quality:
- Promise-based memoization (
bundlePromise ??=) correctly handles concurrent calls, avoiding duplicate bundling work - Error handling properly resets
bundlePromise = undefinedon failure, allowing retries - Logging preserved via
logger.warnon bundle failure - Bundle population moved to only non-raw drivers (
method_execution_service.ts:582-586), avoiding unnecessary work for in-process execution
Security: No new security concerns - bundling logic unchanged, just deferred.
Test Coverage: PR description confirms 3175 tests pass. The existing comprehensive tests for user_model_loader_test.ts and method_execution_service_test.ts cover the behavioral contract, which hasn't changed - only the timing of when bundling occurs.
Suggestions (non-blocking)
- Documentation update opportunity:
design/execution-drivers.md:171still referencesbundleSourcerather thanbundleSourceFactory. This is a minor documentation artifact that could be updated in a follow-up.
LGTM - clean implementation of a valuable performance optimization.
There was a problem hiding this comment.
Adversarial Review
I systematically attempted to break this code across logic, error handling, concurrency, and API contract dimensions. The implementation is solid.
Critical / High
None found.
Medium
- Interface change on public type (
model.ts:606):ModelDefinition.bundleSource?: stringwas replaced withbundleSourceFactory?: () => Promise<string>. WhileModelDefinitionis exported, this field was only ever set byUserModelLoaderinternally and consumed byMethodExecutionService. External code directly inspectingbundleSourcewould now find it undefined. The risk is low since this was an implementation detail, but it's technically a breaking change to the interface shape.
Low
-
Model file deletion timing (
user_model_loader.ts:357-369): The closure capturesabsolutePathat load time but bundling happens at first Docker execution. If a user deletes the model file between these events, bundling will fail. This is a minor behavioral change from eager bundling (which would have failed at load time instead). In practice this is rare and the error message is clear. -
Non-null assertion in Docker driver (
docker_execution_driver.ts:273): The driver usesrequest.bundle!without null check. IfbundleSourceFactoryis undefined (e.g., built-in models) and someone attempts Docker execution, they'll get a runtime crash. However, this is unchanged from the prior behavior—built-in models never hadbundleSourceset either. The error timing is just different now.
Verification of Memoization Pattern
The promise-based memoization is correct:
let bundlePromise: Promise<string> | undefined;
modelDef.bundleSourceFactory = () => {
bundlePromise ??= bundleExtension(...).catch((error) => {
bundlePromise = undefined; // Allow retry on failure
throw error;
});
return bundlePromise;
};- Concurrent calls share the same promise (no duplicate bundling)
- On failure,
bundlePromiseresets toundefined, allowing retry - Each model's closure has its own
bundlePromise(loop creates new scope each iteration)
Error Semantics Change
Old: Bundle failure at load time → warning logged → bundleSource undefined → Docker execution later fails at driver
New: Bundle failure at execution time → error thrown immediately with clear message
This is better behavior—you get an actionable error message rather than a mysterious driver failure when bundle is missing.
Verdict
PASS — Clean implementation with correct memoization, proper error propagation, and no regressions. The performance improvement (46s → 2.8s startup) is significant with no observable downside for users.
Problem
With 104+ user extension models,
swamp model type search(and any other command that loads extensions) took ~46 seconds to start. After the fix in #741 that parallelised the 4 loader types and skipped loading for help/version commands, the remaining bottleneck was inUserModelLoaderitself.At startup,
loadModels()eagerly built a self-contained bundle for every model file — a separatedeno bundlesubprocess per model that inlines all dependencies (including zod) so the bundle can run inside Docker containers without network access. With 104 models, this was 104 sequential subprocesses on every single CLI invocation, regardless of whether Docker execution would ever be used.Architecture Decision & Tradeoffs
What changed
bundleSource?: stringonModelDefinition(a pre-built JS string stored at load time) has been replaced withbundleSourceFactory?: () => Promise<string>— a memoizing closure that defers the expensive work to the point of actual need.Tradeoff: startup cost vs. first-execution cost
The first out-of-process execution of a given model will be slightly slower than before — it now bundles on demand rather than having it pre-built. This is the right tradeoff because:
type search,model get,data list, etc.)Why only models?
Vaults, drivers, and datastores do not create self-contained bundles at all — they only use the externalized (cached) bundle for in-process execution. This change is model-specific because only models support out-of-process/Docker execution via
bundleSource.User Impact
Measured on a real repo with 104 extension models:
swamp model type search awsswamp model get my-modelswamp data listmodel run16x speedup for the everyday command path. The remaining ~2.8s is the warm-cache cost of the externalized bundle loading (disk reads + dynamic imports for 104 files), which is a separate optimization opportunity.
Files Changed
src/domain/models/model.tsbundleSource?: string→bundleSourceFactory?: () => Promise<string>src/domain/models/user_model_loader.tssrc/domain/models/method_execution_service.tsbundleSourceFactory?.()at execution timeTest Plan
deno check— type checking passesdeno lint— no lint errorsdeno fmt --check— formatting correctdeno run test— 3175 tests passed, 0 faileddeno run compile— binary compiled successfullyswamp model type searchin 104-model repo — 46s → 2.8s