🦋 Changeset detected

Latest commit: ee38236

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agents-api	Ready	Preview, Comment	Mar 31, 2026 5:46pm
agents-docs	Ready	Preview, Comment	Mar 31, 2026 5:46pm
agents-manage-ui	Ready	Preview, Comment	Mar 31, 2026 5:46pm

TL;DR — Adds gateway-backed fallback model support so agents can automatically retry on alternate models when the primary fails, and gates the Cost nav item and fallback-models UI behind runtime capability flags derived from AI_GATEWAY_API_KEY presence.

Key changes

• Add fallbackModels field to ModelSettings — Extends the core Zod schema and OpenAPI spec with an ordered string array of fallback models at the validation, API, and UI type layers.
• Extend /capabilities endpoint with modelFallback and costTracking flags — The capabilities route now checks for AI_GATEWAY_API_KEY at runtime and reports whether fallback and cost-tracking features are available.
• Inject fallback models into gateway providerOptions in ModelFactory — prepareGenerationConfig merges fallbackModels into the gateway provider options when the gateway key is configured, preserving existing gateway settings.
• Add GATEWAY_ROUTABLE_PROVIDERS constant — New constant restricts the fallback model selector to providers that the AI Gateway can route to (anthropic, openai, google).
• Add gatewayOnly mode to ModelSelector — Filters the model picker to only gateway-routable providers and hides custom/LLM-gateway options when selecting fallback models.
• Wire fallback models UI into ModelConfiguration — The shared model configuration component conditionally renders an ordered fallback-model list with add/remove controls, gated behind the modelFallback capability flag.
• Thread fallbackModels through agent and project model forms — Model section, project models section, serialization, and deserialization all propagate fallback models alongside existing model and providerOptions fields.
• Gate Cost sidebar nav item behind costTracking capability — The sidebar now conditionally renders the Cost link only when the server reports cost tracking is enabled.
• Add tests for capabilities endpoint and model factory fallback behavior — New and updated tests cover capability flag responses and prepareGenerationConfig fallback injection logic.

_{Summary ｜ 20 files ｜ 1 commit ｜ base: main ← implement/fallback-models-capabilities}

Capabilities endpoint: modelFallback and costTracking flags

Before: /capabilities only returned sandbox configuration.
After: /capabilities returns modelFallback and costTracking objects, both enabled when AI_GATEWAY_API_KEY is set.

The handler checks process.env.AI_GATEWAY_API_KEY at request time and derives both flags from its presence. The response schema adds two required fields so the UI can feature-gate without hardcoded env checks.

agents-api/src/routes/capabilities.ts · agents-api/src/__tests__/capabilities.test.ts · agents-manage-ui/src/lib/actions/capabilities.ts

Core: fallbackModels schema and model factory integration

Before: ModelSettings had model and providerOptions only; no failover mechanism existed.
After: ModelSettings accepts an optional fallbackModels string array that gets injected into gateway.models provider options at generation time.

The ModelSettingsSchema in agents-core validates the new field. ModelFactory.prepareGenerationConfig merges it into streamProviderOptions.gateway.models only when AI_GATEWAY_API_KEY is present, preserving any existing gateway configuration.

How does fallback injection work at runtime?

When fallbackModels is non-empty and the gateway key exists, the factory spreads existing gateway options and adds a models key with the ordered fallback list. The AI Gateway uses this array to retry failed requests against alternate models in order. Without the gateway key, the field is silently ignored.

packages/agents-core/src/utils/model-factory.ts · packages/agents-core/src/validation/schemas.ts · packages/agents-core/src/constants/models.ts · packages/agents-core/src/__tests__/utils/model-factory.test.ts

UI: Capabilities-gated fallback model editor and cost navigation

Before: The model configuration component had no fallback support; the Cost nav link was always visible.
After: A numbered fallback-model list with add/remove appears when modelFallback.enabled is true; the Cost sidebar link renders only when costTracking.enabled is true.

The ModelConfiguration component queries capabilities and conditionally renders ModelSelector instances in gatewayOnly mode (filtered to GATEWAY_ROUTABLE_PROVIDERS). Fallback models thread through agent sidepane model sections, project form sections, and the serialize/deserialize layer for persistence.

agents-manage-ui/src/components/shared/model-configuration.tsx · agents-manage-ui/src/components/agent/sidepane/nodes/model-selector.tsx · agents-manage-ui/src/components/sidebar-nav/app-sidebar.tsx · agents-manage-ui/src/features/agent/domain/serialize.ts

  ^{｜ View workflow run ｜ Triggered by Pullfrog ｜ Using Claude Opus ｜ 𝕏}

TL;DR — Adds gateway-backed fallback model support and provider restriction so agents can automatically retry on alternate models when the primary fails and limit which providers serve requests. Introduces fallbackModels and allowedProviders inheritance through the project → agent → sub-agent hierarchy. Gates the Cost nav item, fallback-models UI, and allowed-providers UI behind runtime capability flags derived from AI_GATEWAY_API_KEY presence. Surfaces provider routing telemetry in the cost dashboard.

Key changes

• Add fallbackModels and allowedProviders fields to ModelSettings — Extends the core Zod schema and OpenAPI spec with optional ordered string arrays for model failover and provider restriction at the validation, API, and UI type layers.
• Extend /capabilities endpoint with modelFallback and costTracking flags — The capabilities route checks for AI_GATEWAY_API_KEY at runtime and reports whether fallback and cost-tracking features are available.
• Inject fallback models and allowed providers into gateway providerOptions in ModelFactory — prepareGenerationConfig merges fallbackModels into gateway.models and allowedProviders into gateway.order/gateway.only when the gateway key is configured; strips provider prefix from model ID when allowedProviders routes traffic.
• Add inheritGatewayFields to model-resolver.ts for hierarchical field inheritance — Sub-agents inherit fallbackModels and allowedProviders from parent agent and project levels when not explicitly set, with agent taking precedence over project.
• Forward all ModelSettings fields through model config functions — getPrimaryModel, getStructuredOutputModel, and getSummarizerModel now spread the full config object instead of cherry-picking model and providerOptions, fixing a bug where new fields were silently dropped.
• Extract GATEWAY_ROUTABLE_PROVIDERS_SET into shared constant — Moved from ModelFactory into constants/models.ts so both the backend and the UI ModelSelector can restrict fallback choices to providers the AI Gateway can route to.
• Add gatewayOnly, defaultOpen, and onClose props to ModelSelector — Filters the model picker to only gateway-routable providers, hides custom/LLM-gateway options when selecting fallback models, auto-opens the popover for newly added slots, and fires onClose when dismissed without a selection.
• Build AllowedProvidersSection with drag-to-reorder UI — New component with radio toggle (all vs. specific), a draggable ordered provider list with add/remove, and inheritance support from parent configurations.
• Wire fallbackModels and allowedProviders through agent, sub-agent, and project model forms — Both the agent sidepane MetadataEditor and the sub-agent ModelSection propagate both fields across all three model slots with full inheritance chains.
• Gate Cost sidebar nav and cost pages behind costTracking capability — The sidebar conditionally renders the Cost link only when the server reports cost tracking is enabled; both cost page components call notFound() when the capability is disabled.
• Add provider routing telemetry and "Cost by Provider" dashboard table — setGatewayAttributesOnSpan captures gen_ai.response.provider and gen_ai.request.provider from gateway routing metadata; cost dashboard shows a new breakdown table and per-event provider column.
• Normalize gateway model IDs via overrideModelId middleware — gatewayCostMiddleware strips known provider prefixes (e.g. anthropic/) from model IDs so cost tracking attributes report the bare model name regardless of routing.
• Add comprehensive documentation for AI Gateway features — New "AI Gateway" docs section covering fallback models, allowed providers, combined usage, and manual gateway provider options; sub-agents docs updated with fallbackModels and allowedProviders fields.
• Add tests for capabilities, model resolver inheritance, model factory fallback, and overrideModelId — New tests cover gateway field inheritance across project/agent/sub-agent hierarchy, capability flag responses, prepareGenerationConfig fallback/provider injection, and normalizeModelId.

_{Summary ｜ 35 files ｜ 9 commits ｜ base: main ← implement/fallback-models-capabilities}

Capabilities endpoint: modelFallback and costTracking flags

Before: /capabilities only returned sandbox configuration.
After: /capabilities returns modelFallback and costTracking objects, both enabled when AI_GATEWAY_API_KEY is set.

The handler checks process.env.AI_GATEWAY_API_KEY at request time and derives both flags from its presence. The response schema adds two required fields so the UI can feature-gate without hardcoded env checks.

capabilities.ts · capabilities.test.ts · capabilities.ts (UI)

Core: fallbackModels and allowedProviders schema and model factory integration

Before: ModelSettings had model and providerOptions only; no failover or provider-restriction mechanism existed.
After: ModelSettings accepts optional fallbackModels and allowedProviders string arrays that get injected into gateway provider options at generation time, with provider prefix stripping when allowedProviders routes traffic.

The ModelSettingsSchema in agents-core validates both new fields. ModelFactory.prepareGenerationConfig conditionally builds gateway provider options when AI_GATEWAY_API_KEY is present: fallbackModels maps to gateway.models, allowedProviders maps to gateway.order and gateway.only. When allowedProviders is set, createModel strips the provider prefix from the model ID so the gateway can route to the first allowed provider. Both fields are silently ignored without the gateway key.

How do fallback and provider restrictions interact at runtime?

When the gateway key exists, the factory merges both into the same gateway options object. models tells the gateway which alternate models to try on failure, while order/only restrict which providers can serve any request (including fallback attempts). They compose naturally — e.g., setting allowedProviders: ['bedrock', 'anthropic'] with fallbackModels: ['anthropic/claude-haiku-4-5'] routes all traffic through Bedrock or Anthropic and retries on Haiku if the primary fails.

schemas.ts · model-factory.ts · models.ts · model-factory.test.ts

Gateway field inheritance in model resolver

Before: resolveModelConfig returned model settings verbatim from the first level in the hierarchy that defined a base model.
After: An inheritGatewayFields function fills in fallbackModels and allowedProviders from parent levels (agent → project) when the resolved level does not define them, respecting explicit overrides.

The resolver walks sub-agent → agent → project. For each model slot, if the resolved config lacks fallbackModels or allowedProviders, it checks progressively higher levels. Explicit values at lower levels are never overridden. Five new tests cover all inheritance paths and the no-parent edge case.

model-resolver.ts · model-resolver.test.ts · model-config.ts

Provider routing telemetry and cost-by-provider dashboard

Before: Cost middleware only set gen_ai.usage.cost; the dashboard had no provider dimension.
After: setGatewayAttributesOnSpan captures gen_ai.response.provider and gen_ai.request.provider from gateway routing metadata; the cost dashboard shows a "Cost by Provider" breakdown table and a per-event provider column. overrideModelId normalizes gateway model IDs by stripping provider prefixes.

usage-cost-middleware.ts · otel-attributes.ts · cost-dashboard.tsx · signoz-stats.ts

UI: Capabilities-gated fallback model editor and allowed providers

Before: The model configuration component had no fallback or provider-restriction support; the Cost nav link was always visible.
After: A numbered fallback-model list with add/remove appears when modelFallback.enabled is true; an allowed-providers section with drag-to-reorder and all/specific radio toggle appears alongside it; the Cost sidebar link and cost pages render only when costTracking.enabled is true.

model-configuration.tsx · model-selector.tsx · model-section.tsx · metadata-editor.tsx

Documentation: AI Gateway section in models docs

Before: Gateway usage was documented inline with a providerOptions.gateway.models code example.
After: A dedicated "AI Gateway" section documents fallback models, allowed providers, combined usage, and manual gateway provider options. Sub-agents docs updated with new configurable fields.

The old inline gateway model routing example was replaced with a brief mention linking to the new section. The new docs cover inheritance behavior (project → agent → sub-agent), provider/model format requirements, and interaction semantics when both features are combined.

models.mdx · sub-agents.mdx

  ^{｜ View workflow run ｜ Triggered by Pullfrog ｜ Using Claude Opus ｜ 𝕏}

Ito Test Report ❌

19 test cases ran. 5 failed, 14 passed.

Across 19 test cases, 14 passed and 5 failed, with core gateway capability behavior, cost-navigation gating/routing, agent fallback persistence, unsaved-change handling, save-race stability, and security checks (unauthorized API access, XSS, malformed/oversized IDs, and capability-fetch failure handling) all working as expected. The most important confirmed defects are in fallback-model UX and persistence: project settings drops models.*.fallbackModels during save/reload (including duplicate and concurrent-edit scenarios, causing silent data loss despite success feedback), inherited sub-agent fallback rows can be edited before explicit override, and pressing Escape in the selector leaves an orphan pending fallback row.

❌ Failed (5)

Category	Summary	Screenshot
Edge	🟠 Inherited structured/summarizer fallback settings are editable before explicit override.
Edge	🟠 Duplicate fallback entries can be added, but fallback rows disappear after save/reload due to fallbackModels being omitted from submit payloads.
Edge	⚠️ Saving project drops fallbackModels from the PATCH payload despite success feedback.
Logic	🟠 Pressing Escape closes the fallback model popover but leaves an orphan pending fallback row instead of clearing it.
Happy-path	⚠️ Fallback model selections are dropped on save/reload because project form serialization omits fallbackModels.

const savedModels = fallbackModels ?? inheritedFallbackModels ?? [];
const isInherited = !fallbackModels && !!inheritedFallbackModels;

{!showPendingSelector && (
  <Button
    variant="outline"
    size="sm"
    className="w-full"
    onClick={() => setShowPendingSelector(true)}
    disabled={disabled}
  >

fallbackModels={models?.structuredOutput?.fallbackModels}
inheritedFallbackModels={structuredOutputInheritance.fallbackModels}
onFallbackModelsChange={(models) =>
  updatePath('models.structuredOutput.fallbackModels', models.length ? models : undefined)
}

base: {
  model: data.models.base.model,
  providerOptions: cleanProviderOptions(data.models.base.providerOptions),
},
structuredOutput: data.models?.structuredOutput?.model
  ? {
      model: data.models.structuredOutput.model,
      providerOptions: cleanProviderOptions(data.models.structuredOutput.providerOptions),
    }
  : undefined,
summarizer: data.models?.summarizer?.model
  ? {
      model: data.models.summarizer.model,
      providerOptions: cleanProviderOptions(data.models.summarizer.providerOptions),
    }
  : undefined,

{capabilities?.modelFallback?.enabled && effectiveModel && onFallbackModelsChange && (
  <FallbackModelsSection
    editorNamePrefix={editorNamePrefix}
    fallbackModels={fallbackModels}
    inheritedFallbackModels={inheritedFallbackModels}
    onFallbackModelsChange={onFallbackModelsChange}
    disabled={disabled}
  />
)}

const { field: fallbackModelsField } = useController({
  control,
  name: 'models.base.fallbackModels',
});

models: {
  ...data.models,
  base: {
    model: data.models.base.model,
    providerOptions: cleanProviderOptions(data.models.base.providerOptions),
  },
  structuredOutput: data.models?.structuredOutput?.model
    ? {
        model: data.models.structuredOutput.model,
        providerOptions: cleanProviderOptions(data.models.structuredOutput.providerOptions),
      }
    : undefined,

const [updated] = await db
  .update(projects)
  .set({
    ...params.data,
    updatedAt: now,
  })

{showPendingSelector && (
        <div className="flex items-center gap-2">
          <span className="text-xs text-muted-foreground w-4 shrink-0">
            {savedModels.length + 1}.
          </span>
          <div className="flex-1">
            <ModelSelector
              value=""
              gatewayOnly
              defaultOpen
              onValueChange={(newValue) => {
                setShowPendingSelector(false);
                if (newValue) {
                  onFallbackModelsChange([...(fallbackModels ?? []), newValue]);
                }
              }}

<Button
            variant="ghost"
            size="icon"
            className="h-8 w-8 shrink-0"
            onClick={() => setShowPendingSelector(false)}
          >

const [open, setOpen] = useState(defaultOpen);

  useEffect(() => {
    if (defaultOpen) {
      setOpen(true);
    }
  }, [defaultOpen]);

  <Popover open={open} onOpenChange={disabled ? undefined : setOpen}>

return {
  ...data,
  models: {
    ...data.models,
    base: {
      model: data.models.base.model,
      providerOptions: cleanProviderOptions(data.models.base.providerOptions),
    },
    structuredOutput: data.models?.structuredOutput?.model
      ? {
          model: data.models.structuredOutput.model,
          providerOptions: cleanProviderOptions(data.models.structuredOutput.providerOptions),
        }
      : undefined,

const { field: fallbackModelsField } = useController({
  control,
  name: 'models.base.fallbackModels',
});

<ModelConfiguration
  fallbackModels={fallbackModelsField.value ?? undefined}
  onFallbackModelsChange={(models) => {
    fallbackModelsField.onChange(models.length ? models : undefined);
  }}
/>

const modelSettingsSchema = z.object({
  model: z.string().optional(),
  providerOptions: z.record(z.string(), z.any()).optional().nullable(),
  fallbackModels: z.array(z.string()).optional().nullable(),
});

✅ Passed (14)

Category	Summary	Screenshot
Adversarial	XSS payload in custom model ID persisted as inert text with no script execution.
Adversarial	Malformed and oversized custom model IDs are treated as opaque values with no executable path.
Adversarial	Unauthenticated capabilities and project update calls were rejected with 401, and authenticated readback showed no project mutation.
Adversarial	Cost links were hidden while direct cost deep-links remained accessible with non-sensitive empty/zero data.
Adversarial	Save replay stress in project settings and agent editor stayed deterministic after double-submit attempts and reload.
Edge	Fallback selector exposed gateway-routable providers and blocked unsupported openrouter selection.
Edge	Mobile viewport fallback controls remained reachable, and project update completed successfully.
Edge	Unsaved edits were discarded on back/forward and refresh, with saved values preserved.
Edge	Capability-fetch error handling remains defensive and UI gating stays null-safe.
Happy-path	Authenticated capabilities check returned gateway-off flags with sandbox present as expected.
Happy-path	Capabilities returned modelFallback/costTracking as enabled and project settings showed fallback controls in gateway-on mode.
Happy-path	Fallback model persisted after save and hard refresh in agent editor.
Happy-path	Verified capability-gated cost navigation in org and project sidebars across gateway-off and gateway-on modes, with correct destination routes.
Happy-path	Capability flags stayed consistent across project settings, Cost navigation, and agent editor gating in one authenticated session.

Commit: 7d9c6de

View Full Run

Tell us how we did: Give Ito Feedback

Ito Test Report ❌

18 test cases ran. 2 failed, 16 passed.

Overall, 16 of 18 test cases passed, confirming that capability-gated behavior, API contracts, sidebar visibility, deep-link blocking when disabled, enabled cost-page rendering paths, project model-configuration logic, persistence, inheritance/serialization, and adversarial security checks all behaved as expected without mocked route behavior. The key issue was 2 high-severity failures reproducing the same real regression introduced by this PR: tenant/project cost routes can incorrectly hit 404 in enabled environments during fresh load/refresh because the guard calls notFound() while capabilities are still unresolved (null initial query data).

❌ Failed (2)

Category	Summary	Screenshot
Edge	⚠️ Enabled tenant cost deep-link can render 404 before capabilities finish loading.
Edge	⚠️ Cost pages can render persistent 404 before capabilities resolve, despite enabled cost tracking.

const { data: capabilities, isLoading: capabilitiesLoading } = useCapabilitiesQuery();

if (!capabilitiesLoading && !capabilities?.costTracking?.enabled) {
  notFound();
}

const { data: capabilities, isLoading: capabilitiesLoading } = useCapabilitiesQuery();

if (!capabilitiesLoading && !capabilities?.costTracking?.enabled) {
  notFound();
}

return useQuery<Capabilities | null>({
  queryKey: capabilitiesQueryKeys.current,
  async queryFn() {
    const response = await getCapabilitiesAction();
    if (!response.success || !response.data) {
      throw new Error(response.error);
    }

    return response.data;
  },
  enabled,
  initialData: null,
  initialDataUpdatedAt: 0,
  staleTime: 30_000,
});

return useQuery<Capabilities | null>({
  queryKey: capabilitiesQueryKeys.current,
  async queryFn() {
    const response = await getCapabilitiesAction();
    if (!response.success || !response.data) {
      throw new Error(response.error);
    }

    return response.data;
  },
  enabled,
  initialData: null,
  initialDataUpdatedAt: 0,
  staleTime: 30_000,
});

const { data: capabilities, isLoading: capabilitiesLoading } = useCapabilitiesQuery();

if (!capabilitiesLoading && !capabilities?.costTracking?.enabled) {
  notFound();
}

const { data: capabilities, isLoading: capabilitiesLoading } = useCapabilitiesQuery();

if (!capabilitiesLoading && !capabilities?.costTracking?.enabled) {
  notFound();
}

✅ Passed (16)

Category	Summary	Screenshot
Adversarial	Malicious custom model strings were handled as inert text with no alert/dialog execution, and the UI remained stable through reload.
Adversarial	Manual URL re-entry and navigation-based bypass attempts remained blocked, matching expected capability-gating behavior.
Adversarial	Concurrent tab saves converged to one consistent persisted state after reload, with no malformed model configuration.
Adversarial	Tampered project settings payload was handled safely (no 5xx), and settings UI remained stable after reload.
Edge	Rapid add/remove/re-add fallback interactions stayed stable with no phantom rows and persisted intended values after reload.
Edge	Outside-click and explicit-cancel flows removed pending fallback entries, and save payload/reload contained only valid fallback models.
Logic	Enabled mode exposed Allowed providers and Fallback models after base-model selection, and disabled mode correctly hid gated sections while retaining base model controls.
Logic	Fallback selector correctly limited options to gateway-routable providers and excluded disallowed provider groups.
Logic	Allowed provider ordering persisted after save/reload, and switching to All providers cleared allowedProviders while preserving valid fallback behavior.
Logic	Sub-agent pane displayed inherited base model/provider indicators and allowed switching structured/summarizer sections to specific providers with editable controls.
Logic	Re-execution confirmed save works when the form is dirty, and serialization included fallbackModels/allowedProviders while cleared/default fields were omitted.
Happy-path	Capabilities endpoint returned HTTP 200 with sandbox present and both modelFallback/costTracking disabled when the gateway key was unset.
Happy-path	Re-execution confirmed /manage/capabilities returned modelFallback.enabled=true and costTracking.enabled=true with gateway capability configured.
Screen	Tenant and project sidebars correctly hid Cost navigation when capability flags were disabled.
Screen	Direct deep-links to tenant and project cost routes stayed blocked and showed 404-style pages through refresh/back-forward checks.
Screen	Tenant and project cost pages rendered successfully and showed the Cost by Provider section with a valid empty state.

Commit: d2b86a7

View Full Run

Tell us how we did: Give Ito Feedback