Skip to content

fix: flush OTEL spans after Slack webhook fire-and-forget handlers #2006

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
amikofalvy merged 2 commits into from
Feb 14, 2026
Merged

fix: flush OTEL spans after Slack webhook fire-and-forget handlers #2006

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
49e20e0
fix: flush OTEL spans after Slack webhook fire-and-forget handlers
amikofalvy Feb 13, 2026
e2bba53
address pr review: add unit tests and warning logging for flushTraces
amikofalvy Feb 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions packages/agents-core/src/utils/__tests__/flushTraces.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
import { trace } from '@opentelemetry/api';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { flushTraces } from '../tracer-factory';

describe('flushTraces', () => {
afterEach(() => {
vi.restoreAllMocks();
});

it('should call forceFlush on delegate when provider has getDelegate', async () => {
const mockForceFlush = vi.fn().mockResolvedValue(undefined);
vi.spyOn(trace, 'getTracerProvider').mockReturnValue({
getTracer: vi.fn(),
getDelegate: () => ({ forceFlush: mockForceFlush }),
} as any);

await flushTraces();
expect(mockForceFlush).toHaveBeenCalledOnce();
});

it('should call forceFlush directly when provider has it but no getDelegate', async () => {
const mockForceFlush = vi.fn().mockResolvedValue(undefined);
vi.spyOn(trace, 'getTracerProvider').mockReturnValue({
getTracer: vi.fn(),
forceFlush: mockForceFlush,
} as any);

await flushTraces();
expect(mockForceFlush).toHaveBeenCalledOnce();
});

it('should not throw when provider has no forceFlush method', async () => {
vi.spyOn(trace, 'getTracerProvider').mockReturnValue({
getTracer: vi.fn(),
} as any);

await expect(flushTraces()).resolves.toBeUndefined();
});

it('should not throw when forceFlush rejects', async () => {
const mockForceFlush = vi.fn().mockRejectedValue(new Error('Export failed'));
vi.spyOn(trace, 'getTracerProvider').mockReturnValue({
getTracer: vi.fn(),
forceFlush: mockForceFlush,
} as any);

await expect(flushTraces()).resolves.toBeUndefined();
});

it('should not throw when getTracerProvider throws', async () => {
vi.spyOn(trace, 'getTracerProvider').mockImplementation(() => {
throw new Error('OTEL not initialized');
});

await expect(flushTraces()).resolves.toBeUndefined();
});
});
26 changes: 26 additions & 0 deletions packages/agents-core/src/utils/tracer-factory.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,3 +106,29 @@ export function getTracer(serviceName: string, serviceVersion?: string): Tracer
return noopTracer;
}
}

/**
* Force-flush all pending trace spans to the configured exporter.
*
* Uses the global TracerProvider registered by the OpenTelemetry SDK.
* This is safe to call from any package — it accesses the same provider
* that was set up in the host application's instrumentation.
*
* Use this in fire-and-forget handlers (e.g. Slack webhooks) where the
* HTTP response is sent before background work completes, so the
* per-request flush middleware runs too early to capture those spans.
*/
export async function flushTraces(): Promise<void> {
try {
const provider = trace.getTracerProvider() as {
forceFlush?: () => Promise<void>;
getDelegate?: () => { forceFlush?: () => Promise<void> };
};
const delegate = typeof provider.getDelegate === 'function' ? provider.getDelegate() : provider;
if (typeof delegate.forceFlush === 'function') {
await delegate.forceFlush();
}
Comment on lines +128 to +130
Copy link
Copy Markdown
Contributor

@claude claude Bot Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💭 Consider: Silent no-op when forceFlush unavailable

Issue: When forceFlush is not available on the delegate, the function silently does nothing with no indication to operators.

Why: This is intentional for environments without OTEL configured, but could mask configuration issues where the TracerProvider wasn't properly initialized.

Fix: Consider adding debug-level logging to help diagnose configuration issues:

Suggested change
if (typeof delegate.forceFlush === 'function') {
await delegate.forceFlush();
}
if (typeof delegate.forceFlush === 'function') {
await delegate.forceFlush();
}

Note: The current implementation is acceptable since debug logging here may be noisy. This is a minor consideration — the no-op behavior is correct for unconfigured environments.

} catch (error) {
logger.warn({ error }, 'Failed to flush traces');
}
}
100 changes: 59 additions & 41 deletions packages/agents-work-apps/src/slack/routes/events.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import {
deleteAllWorkAppSlackChannelAgentConfigsByTeam,
deleteAllWorkAppSlackUserMappingsByTeam,
deleteWorkAppSlackWorkspaceByNangoConnectionId,
flushTraces,
} from '@inkeep/agents-core';
import { SpanStatusCode } from '@opentelemetry/api';
import runDbClient from '../../db/runDbClient';
Expand Down Expand Up @@ -209,14 +210,16 @@ app.post('/events', async (c) => {
threadTs: event.thread_ts || event.ts || '',
messageTs: event.ts || '',
teamId,
}).catch((err: unknown) => {
const errorMessage = err instanceof Error ? err.message : String(err);
const errorStack = err instanceof Error ? err.stack : undefined;
logger.error(
{ errorMessage, errorStack },
'Failed to handle app mention (outer catch)'
);
});
})
.catch((err: unknown) => {
const errorMessage = err instanceof Error ? err.message : String(err);
const errorStack = err instanceof Error ? err.stack : undefined;
logger.error(
{ errorMessage, errorStack },
'Failed to handle app mention (outer catch)'
);
})
.finally(() => flushTraces());
} else {
outcome = 'ignored_unknown_event';
span.setAttribute(SLACK_SPAN_KEYS.OUTCOME, outcome);
Expand Down Expand Up @@ -258,21 +261,26 @@ app.post('/events', async (c) => {
actionValue: action.value,
teamId,
responseUrl: responseUrl || '',
}).catch(async (err: unknown) => {
const errorMessage = err instanceof Error ? err.message : String(err);
logger.error(
{ errorMessage, actionId: action.action_id },
'Failed to open agent selector modal'
);
if (responseUrl) {
await sendResponseUrlMessage(responseUrl, {
text: 'Sorry, something went wrong while opening the agent selector. Please try again.',
response_type: 'ephemeral',
}).catch((e) =>
logger.warn({ error: e }, 'Failed to send error notification via response URL')
})
.catch(async (err: unknown) => {
const errorMessage = err instanceof Error ? err.message : String(err);
logger.error(
{ errorMessage, actionId: action.action_id },
'Failed to open agent selector modal'
);
}
});
if (responseUrl) {
await sendResponseUrlMessage(responseUrl, {
text: 'Sorry, something went wrong while opening the agent selector. Please try again.',
response_type: 'ephemeral',
}).catch((e) =>
logger.warn(
{ error: e },
'Failed to send error notification via response URL'
)
);
}
})
.finally(() => flushTraces());
}

if (action.action_id === 'modal_project_select') {
Expand Down Expand Up @@ -354,6 +362,8 @@ app.post('/events', async (c) => {
{ err, selectedProjectId },
'Failed to update modal on project change'
);
} finally {
await flushTraces();
}
})();
}
Expand All @@ -370,13 +380,15 @@ app.post('/events', async (c) => {
actionValue: action.value,
teamId,
responseUrl: responseUrl || undefined,
}).catch((err: unknown) => {
const errorMessage = err instanceof Error ? err.message : String(err);
logger.error(
{ errorMessage, actionId: action.action_id },
'Failed to open follow-up modal'
);
});
})
.catch((err: unknown) => {
const errorMessage = err instanceof Error ? err.message : String(err);
logger.error(
{ errorMessage, actionId: action.action_id },
'Failed to open follow-up modal'
);
})
.finally(() => flushTraces());
}
}

Expand Down Expand Up @@ -435,10 +447,12 @@ app.post('/events', async (c) => {
messageText: message.text || '',
threadTs: message.thread_ts,
responseUrl,
}).catch((err: unknown) => {
const errorMessage = err instanceof Error ? err.message : String(err);
logger.error({ errorMessage, callbackId }, 'Failed to handle message shortcut');
});
})
.catch((err: unknown) => {
const errorMessage = err instanceof Error ? err.message : String(err);
logger.error({ errorMessage, callbackId }, 'Failed to handle message shortcut');
})
.finally(() => flushTraces());
} else {
outcome = 'ignored_unknown_event';
span.setAttribute(SLACK_SPAN_KEYS.OUTCOME, outcome);
Expand Down Expand Up @@ -491,10 +505,12 @@ app.post('/events', async (c) => {
span.setAttribute(SLACK_SPAN_KEYS.OUTCOME, outcome);
logger.info({ callbackId }, 'Handling view_submission: agent_selector_modal');

handleModalSubmission(view).catch((err: unknown) => {
const errorMessage = err instanceof Error ? err.message : String(err);
logger.error({ errorMessage, callbackId }, 'Failed to handle modal submission');
});
handleModalSubmission(view)
.catch((err: unknown) => {
const errorMessage = err instanceof Error ? err.message : String(err);
logger.error({ errorMessage, callbackId }, 'Failed to handle modal submission');
})
.finally(() => flushTraces());

span.end();
return new Response(null, { status: 200 });
Expand All @@ -510,10 +526,12 @@ app.post('/events', async (c) => {
span.setAttribute(SLACK_SPAN_KEYS.OUTCOME, outcome);
logger.info({ callbackId }, 'Handling view_submission: follow_up_modal');

handleFollowUpSubmission(view).catch((err: unknown) => {
const errorMessage = err instanceof Error ? err.message : String(err);
logger.error({ errorMessage, callbackId }, 'Failed to handle follow-up submission');
});
handleFollowUpSubmission(view)
.catch((err: unknown) => {
const errorMessage = err instanceof Error ? err.message : String(err);
logger.error({ errorMessage, callbackId }, 'Failed to handle follow-up submission');
})
.finally(() => flushTraces());

span.end();
return new Response(null, { status: 200 });
Expand Down