feat: move AWS, Azure, and 1Password vault providers to extensions#736
feat: move AWS, Azure, and 1Password vault providers to extensions#736
Conversation
Move the aws-sm, azure-kv, and 1password vault providers from built-in types to extension vaults, installable via swamp extension pull or auto-resolved from the registry on first use. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Vault create now auto-resolves @-prefixed vault types from the registry,
matching the behavior of model commands. This allows users to run
swamp vault create @swamp/aws-sm my-vault --config '{"region":"us-east-1"}'
without manually installing the extension first.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
There was a problem hiding this comment.
Adversarial Review
I systematically traced code paths, examined edge cases, and stress-tested the error handling. The code is well-structured and handles the migration cleanly.
Critical / High
None found. The migration logic is sound, error handling is appropriate, and no security issues are introduced.
Medium
-
UX gap in
vault_create.ts:127-132: When a user provides a deprecated type name likeaws-sm, the error message is:Unknown vault type: aws-sm. Available types: local_encryption. Use 'swamp vault type search' to see available types.This doesn't mention that
aws-smhas been renamed to@swamp/aws-sm. Contrast withregisterVaultinvault_service.ts:149-155which callssuggestVaultType()and helpfully says "The type 'aws-sm' has been renamed to '@swamp/aws-sm'". Sincevault_createvalidates the type before callingregisterVault, users hit the less helpful error first.Breaking example:
swamp vault create aws-sm my-vault --config '{"region":"us-east-1"}'gives a confusing error instead of a migration hint.Suggested fix: Call
suggestVaultType()in the vault_create error path, or auto-remap old type names before validation (similar tofromRepository). -
Extension vaults require
--configeven when empty (vault_create.ts:154-159): If an extension vault type has no required configuration, users still must pass--config '{}'. This is awkward but not broken.
Low
-
Theoretical:
resolveProviderConfigdefault case (vault_create.ts:52-53): Returns{}for unknown built-in types. Currently fine since onlylocal_encryptionandmockare built-in, but future built-in types would silently get empty config. Mock vaults already work with{}per tests. -
fromRepositorysilently swallows non-"Unsupported vault type" errors at debug level (vault_service.ts:103-105): If config schema validation fails insideregisterVaultfor a reason other than unsupported type, the error is logged at debug level and the vault is skipped without user-visible warning. This is intentional (fallback behavior) but could mask real config errors.
Verdict
PASS — No blocking issues. The medium UX gap (missing rename hint in vault_create) is a polish issue that doesn't affect correctness. The migration path is documented in the PR body, breaking changes are intentional, and the code handles edge cases appropriately.
There was a problem hiding this comment.
Review Summary
This PR cleanly moves AWS, Azure, and 1Password vault providers to extensions, reducing core dependencies while maintaining backward compatibility.
No blocking issues found.
Code Quality ✓
- TypeScript strict mode compliance - the only
anyusage (AnyOptions) is properly annotated withdeno-lint-ignore - Named exports used consistently
- AGPLv3 headers present in all modified files
- Clean separation of concerns
DDD Compliance ✓
VaultServiceremains a proper Domain Service (stateless operations)- Repository pattern used appropriately (
YamlVaultConfigRepository) - Clean encapsulation of migration logic in
RENAMED_VAULT_TYPES assertVaultProvider()provides good runtime validation for extension providers
Migration Path ✓
RENAMED_VAULT_TYPESmaps old names (aws,aws-sm,azure,azure-kv,1password) to new extension namesfromRepository()auto-remaps and logs deprecation warnings- Error messages updated to guide users to install extensions
Test Coverage ✓
vault_service_test.ts: Tests error handling, renamed type suggestions, mock vault, auto-remapping behaviorvault_types_test.ts: Verifies onlylocal_encryptionremains as built-invault_expression_test.ts: Updated for new error message format
Suggestions (non-blocking)
vault_create.tshas no dedicated unit tests for the command itself - this is a pre-existing gap, not introduced by this PR
LGTM - well-executed migration with proper backward compatibility.
…on (#910) ## Summary - Add auto-resolution infrastructure for datastore extensions (mirrors the vault pattern from #736) - Add name remapping: `type: s3` in `.swamp.yaml` → `@swamp/s3-datastore` extension - Backward compatible — built-in S3 code is still present as a fallback This is the first step toward moving the S3 datastore to the `@swamp/s3-datastore` extension package (published as v2026.03.28.1 in the [swamp-extensions](https://github.com/systeminit/swamp-extensions) repo). A follow-up PR will remove the built-in S3 code once this is validated. ### Phase 1: Auto-resolution infrastructure - `resolveDatastoreType()` helper in `extension_auto_resolver.ts` - `hotLoadDatastores()` in `ExtensionInstallerPort` and auto-resolver adapter - Datastores are now hot-loaded alongside models and vaults after extension install ### Phase 2: Name remapping - `RENAMED_DATASTORE_TYPES` map: `"s3" → "@swamp/s3-datastore"` - `resolveDatastoreConfig` and `parseDatastoreEnvVar` are now async - Old `type: s3` configs auto-resolve the extension, with deprecation warning - Falls back to built-in S3 if extension unavailable ## User impact - **No breaking changes** — existing `type: s3` configs work as before - Deprecation warning logged when old `s3` type name is used - First use auto-installs extension (~5s one-time); subsequent runs use cache - Requires `@swamp/s3-datastore` extension published to registry (done: v2026.03.28.1) ## Test plan - [x] Compiled binary, tested against real S3-backed repository - [x] First run: extension auto-resolved, downloaded, bundled, cached - [x] Second run: loaded from cache, no registry call, completed cleanly - [x] All unit tests pass (38 tests across modified files) - [ ] CI passes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
…tension Remove all built-in S3 datastore code from the core binary and delegate S3 storage entirely to the `@swamp/s3-datastore` extension. This follows the same pattern as PR #736 which moved vault providers to extensions. ## What changed ### Built-in S3 code removed - Deleted 6 infrastructure files: `s3_client.ts`, `s3_lock.ts`, `s3_cache_sync.ts`, `s3_datastore_verifier.ts`, and their tests - Removed `S3DatastoreConfig`, `S3ConnectionConfig` from the type system - Removed `@aws-sdk/client-s3` from `deno.json` (no longer bundled) - Removed all `config.type === "s3"` branches from `repo_context.ts`, `resolve_datastore.ts`, `setup.ts`, `sync.ts`, `status.ts` - Removed `swamp datastore setup s3` CLI command ### Generic extension setup command added - New `swamp datastore setup extension <type> --config '{...}'` command works with any extension datastore, not just S3 - Validates config against extension schema, verifies health via provider, migrates data via sync service, updates `.swamp.yaml` - Supports legacy type names: `swamp datastore setup extension s3 ...` auto-remaps to `@swamp/s3-datastore` ### Backward compatibility preserved - `type: s3` in `.swamp.yaml` auto-remaps to `@swamp/s3-datastore` with deprecation warning (from Phase 1+2 in PR #910) - `SWAMP_DATASTORE=s3:bucket/prefix` env var format still works - Extension auto-resolves on first use for logged-in users ### Docs and skills updated - `design/datastores.md`: S3 now documented as extension, not built-in - `README.md`: Updated setup examples to use extension command - `.claude/skills/swamp-repo/SKILL.md`: Updated command reference ## Design choices ### Why the provider pattern is better than built-in The old code created `S3Client` at 6+ call sites in `repo_context.ts`, manually destructuring config fields each time. PR #917 found that `endpoint` and `forcePathStyle` were silently dropped at several sites, breaking S3-compatible stores (DigitalOcean Spaces, MinIO). The extension's provider pattern stores config once at construction and passes it through everywhere — structurally immune to this class of bug. ### Sync behavioral differences The built-in S3 path used `pullAll()`/`pushAll()` for sync operations. The extension path uses `pullChanged()`/`pushChanged()` via the `DatastoreSyncService` interface. The "changed" variants are better for normal use (only transfer what's needed). For empty caches, they're equivalent since everything is "changed". The built-in also had `pullChangedForModel()` for per-model scoped pulls — an optimization not available through the generic interface, acceptable trade-off for now. ### `datastore status` no longer shows S3-specific fields Bucket, prefix, region, endpoint are no longer displayed in `swamp datastore status`. The generic extension interface doesn't expose backend-specific details. The health check still works via the provider's `createVerifier()`. ### Known limitation: offline + legacy S3 config If a user has `type: s3` in `.swamp.yaml`, is offline, and the extension isn't cached, `resolveDatastoreConfig` throws before any command runs. The error message is clear ("Install it with: swamp extension pull @swamp/s3-datastore"). This is an edge case — the extension is cached after first use. ## Test plan - [x] `deno check` — clean - [x] `deno lint` — 910 files, clean - [x] `deno fmt` — 998 files, clean - [x] `deno run test` — 3672 passed, 0 failed - [x] `deno run compile` — binary compiles without `@aws-sdk/client-s3` - [x] Manual test: compiled binary against real S3-backed repo, extension auto-resolved and command completed cleanly - [ ] CI passes Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
Closes #665
Summary
Moves the
aws-sm,azure-kv, and1passwordvault providers from built-in types to extension vaults published at swamp.club. After this change, onlylocal_encryption(andmockfor testing) remain as built-in vault types. The three cloud/external vault providers are now independently versioned extensions that auto-resolve from the registry on first use.What changed
Removed from core:
aws_vault_provider.ts,azure_kv_vault_provider.ts,onepassword_vault_provider.tsand their test files (-1,545 lines)aws-sm,azure-kv,1passwordfromBUILT_IN_VAULT_TYPESinvault_types.ts— onlylocal_encryptionremainsVaultService.registerVault()@aws-sdk/client-secrets-manager,@azure/identity,@azure/keyvault-secretsfromdeno.jsondependenciesMigration path via
RENAMED_VAULT_TYPES:aws/aws-sm→@swamp/aws-smazure/azure-kv→@swamp/azure-kv1password→@swamp/1passwordWhen
VaultService.fromRepository()loads an existing vault config with an old type name, it remaps to the@swamp/*extension type and auto-resolves it from the registry (installed by PR #725's auto-resolution infrastructure).vault createsimplified:--region,--vault-url,--op-vault,--op-accountflags--config <json>for provider configurationresolveProviderConfig()only handleslocal_encryptionnowensureDefaultVaults()is now a no-op:AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY, andAWS_REGIONwere setError messages updated:
swamp extension pull @swamp/aws-sminstead of setting AWS env varsPublished extensions
The three vault providers have been published to swamp.club as:
@swamp/[email protected]— shells out toopCLI, no npm SDK deps@swamp/[email protected]— uses@aws-sdk/[email protected]@swamp/[email protected]— uses@azure/[email protected]+@azure/[email protected]Source lives at https://github.com/systeminit/swamp-extensions
User impact
Existing users with vault configs on disk
No action required. Existing
.swamp/vault/*.yamlfiles withtype: aws-sm,type: azure-kv, ortype: 1passwordcontinue to work. On first use, swamp will:@swamp/*extension typeCreating new vaults
The CLI syntax changes from dedicated flags to
--config <json>:Offline users
Users without registry access can manually install extensions by placing the
.tssource files inextensions/vaults/.Binary size
The compiled binary no longer includes the AWS SDK, Azure SDK, or 1Password provider code. These dependencies are now bundled into the extensions at publish time.
Known issues
extensions/models/directory doesn't exist for vault-only extensions (Auto-resolver fails when extensions/models/ directory does not exist for vault-only extensions #734)Verification
deno check— passesdeno lint— passesdeno fmt— passesdeno run test— 3138 passed, 0 faileddeno run compile— binary compiled successfullyopCLI check, aws-sm fails at credential check — both expected)🤖 Generated with Claude Code