# Delete project artifact Source: https://docs.uselemma.ai/api-reference/artifacts/delete-project-artifact https://api.uselemma.ai/openapi.json delete /projects/{project_id}/artifacts/{artifact_id} Deletes one uploaded project artifact. # Get Agent understanding version Source: https://docs.uselemma.ai/api-reference/artifacts/get-agent-understanding-version https://api.uselemma.ai/openapi.json get /projects/{project_id}/artifacts/versions/{version} Returns the Agent understanding content and metadata for a specific project, agent, and version. # Get uploaded project artifact content Source: https://docs.uselemma.ai/api-reference/artifacts/get-uploaded-project-artifact-content https://api.uselemma.ai/openapi.json get /projects/{project_id}/artifacts/{artifact_id}/content Returns stored text content and metadata for one uploaded text artifact. # List Agent understanding versions Source: https://docs.uselemma.ai/api-reference/artifacts/list-agent-understanding-versions https://api.uselemma.ai/openapi.json get /projects/{project_id}/artifacts/versions Returns Agent understanding version metadata for a project and agent, newest first. The response omits artifact content. # List project artifacts Source: https://docs.uselemma.ai/api-reference/artifacts/list-project-artifacts https://api.uselemma.ai/openapi.json get /projects/{project_id}/artifacts Returns Agent understanding and uploaded artifacts for a project, scoped to the selected agent. # Regenerate project artifacts Source: https://docs.uselemma.ai/api-reference/artifacts/regenerate-project-artifacts https://api.uselemma.ai/openapi.json post /projects/{project_id}/artifacts/regenerate Starts Agent understanding and diagram regeneration for a tenant-owned project and optional agent. Clients should poll list_project_artifacts for a new latest_version to observe completion. # Update project artifact Source: https://docs.uselemma.ai/api-reference/artifacts/update-project-artifact https://api.uselemma.ai/openapi.json patch /projects/{project_id}/artifacts/{artifact_id} Replaces the stored text of one uploaded project artifact, preserving its id, creation time, and agent. Send either content for a full replacement or patch for ordered exact-match edits, never both. The response carries metadata and the refreshed content digest, not the document. # Upload project artifact Source: https://docs.uselemma.ai/api-reference/artifacts/upload-project-artifact https://api.uselemma.ai/openapi.json post /projects/{project_id}/artifacts Upload an agent-scoped .txt or .md file (text/plain or text/markdown). # Write Agent understanding version Source: https://docs.uselemma.ai/api-reference/artifacts/write-agent-understanding-version https://api.uselemma.ai/openapi.json post /projects/{project_id}/artifacts/versions Records the next Agent understanding version for a project and agent. Versions are append-only: this never edits an existing version, and the first version for an agent must supply both files in full. The response carries the new version identity, not its documents. # Export audit events as CSV Source: https://docs.uselemma.ai/api-reference/audit-log/export-audit-events-as-csv https://api.uselemma.ai/openapi.json get /audit-events/export Organization-wide audit log for owners, admins, and org-wide admin API keys. Project-scoped keys are rejected. Defaults to the last 7 days; the window may not exceed 90 days. Returns every matching event as text/csv (columns: occurred_at, event_id, actor_type, actor_id, impersonator_user_id, action, target_type, target_id, ip, user_agent, metadata). More than 10,000 matching events is a 400; narrow the filters. Each export is itself recorded in the audit log as data.audit_log.export. # List audit events Source: https://docs.uselemma.ai/api-reference/audit-log/list-audit-events https://api.uselemma.ai/openapi.json get /audit-events Organization-wide audit log for owners, admins, and org-wide admin API keys. Project-scoped keys are rejected. Defaults to the last 7 days; the window may not exceed 90 days. Pass the returned next_cursor (JSON) back as cursor to page. # Get GitHub connection Source: https://docs.uselemma.ai/api-reference/github-repositories/get-github-connection https://api.uselemma.ai/openapi.json get /projects/{project_id}/github/connection Returns retained GitHub connection health and last-known repository identity. Disabled rollout state does not remove stored history. # Get insight Source: https://docs.uselemma.ai/api-reference/insights/get-insight https://api.uselemma.ai/openapi.json get /insights/{insight_id} Get a single visible insight for a project, including its analysis, suggested fix and processed member issues. Draft insights, insights outside their agent's top 15, unauthorized projects and unknown projects return 404. # List insights Source: https://docs.uselemma.ai/api-reference/insights/list-insights https://api.uselemma.ai/openapi.json get /insights List project-scoped visible insights: each agent's top 15 active insights by priority, ordered by priority (0 is most urgent). Lower-ranked insights are not returned. member_count counts processed member issues only. Pass limit and offset to page; defaults match issues (50 / max 200). # Attach issue tag Source: https://docs.uselemma.ai/api-reference/issues/attach-issue-tag https://api.uselemma.ai/openapi.json put /issues/{issue_id}/tags/{tag_id} Attach a tag to an issue idempotently. # Create issue tag Source: https://docs.uselemma.ai/api-reference/issues/create-issue-tag https://api.uselemma.ai/openapi.json post /issues/tags Create a project-scoped issue tag. Names are normalized; duplicate normalized names return the existing tag with 200. # Delete issue tag Source: https://docs.uselemma.ai/api-reference/issues/delete-issue-tag https://api.uselemma.ai/openapi.json delete /issues/tags/{tag_id} Delete a project-scoped tag and cascade removal from all issues. # Detach issue tag Source: https://docs.uselemma.ai/api-reference/issues/detach-issue-tag https://api.uselemma.ai/openapi.json delete /issues/{issue_id}/tags/{tag_id} Detach a tag from an issue idempotently. # Get issue Source: https://docs.uselemma.ai/api-reference/issues/get-issue https://api.uselemma.ai/openapi.json get /issues/{issue_id} Get a single issue. # Get issue markdown Source: https://docs.uselemma.ai/api-reference/issues/get-issue-markdown https://api.uselemma.ai/openapi.json get /issues/{issue_id}/markdown Build markdown for an issue. # Get issue metric series Source: https://docs.uselemma.ai/api-reference/issues/get-issue-metric-series https://api.uselemma.ai/openapi.json get /issues/{issue_id}/metric-series Returns zero-filled daily occurrence buckets for an issue over the last 30 days. # Get issues in bulk Source: https://docs.uselemma.ai/api-reference/issues/get-issues-in-bulk https://api.uselemma.ai/openapi.json post /issues/bulk-get Public REST + MCP batch issue read. First-party dashboard and platform-api load issues one-at-a-time via get_issue / list_issues. Fetch many issues by id. One invalid or unauthorized id does not hide the rest of the batch. Results preserve input order. Mixed results are HTTP 200. # List issue occurrences Source: https://docs.uselemma.ai/api-reference/issues/list-issue-occurrences https://api.uselemma.ai/openapi.json get /issues/{issue_id}/occurrences List occurrences for an issue. Pass release to filter; empty string matches unreleased occurrences. # List issue tags Source: https://docs.uselemma.ai/api-reference/issues/list-issue-tags https://api.uselemma.ai/openapi.json get /issues/tags List project-scoped issue tags in normalized name order. # List issues Source: https://docs.uselemma.ai/api-reference/issues/list-issues https://api.uselemma.ai/openapi.json get /issues List issues for a project. By default, returns issues at or above the frequency cutoff. Pass expanded=true for less-common issues and include_context=true for dashboard metric context. expand=dashboard and projection=dashboard are deprecated aliases. # Merge issues Source: https://docs.uselemma.ai/api-reference/issues/merge-issues https://api.uselemma.ai/openapi.json post /issues/{issue_id}/merge Merge one active same-agent issue into this issue. The loser is dismissed as a merged duplicate; its occurrences are immediately moved onto the survivor. # Mutate issues in batch Source: https://docs.uselemma.ai/api-reference/issues/mutate-issues-in-batch https://api.uselemma.ai/openapi.json post /issues/batch Apply one discriminated batch mutation: status, start-review, flag, assignee, or tag. Mixed item-level failures return HTTP 200. Successful items are not rolled back. # Patch issue Source: https://docs.uselemma.ai/api-reference/issues/patch-issue https://api.uselemma.ai/openapi.json patch /issues/{issue_id} Apply exactly one scalar mutation: status (with optional dismissal fields), flagged, or user_id assignee. Empty, mixed, or incompatible bodies are rejected before any write. start-review stays on its dedicated route. # Start issue review Source: https://docs.uselemma.ai/api-reference/issues/start-issue-review https://api.uselemma.ai/openapi.json patch /issues/{issue_id}/start-review Set status to in_progress and assign the acting user when the issue is unassigned, in one atomic write. An existing assignee is preserved; an issue already in progress is returned unchanged. # Create Linear ticket Source: https://docs.uselemma.ai/api-reference/linear-tickets/create-linear-ticket https://api.uselemma.ai/openapi.json post /issues/{issue_id}/linear-ticket # Inspect compact Linear ticket states Source: https://docs.uselemma.ai/api-reference/linear-tickets/inspect-compact-linear-ticket-states https://api.uselemma.ai/openapi.json post /projects/{project_id}/linear-ticket-summaries Returns no ticket history or workspace directory; only active compact states for the requested project issues. # Inspect issue Linear ticket Source: https://docs.uselemma.ai/api-reference/linear-tickets/inspect-issue-linear-ticket https://api.uselemma.ai/openapi.json get /issues/{issue_id}/linear-ticket # Inspect Linear connection Source: https://docs.uselemma.ai/api-reference/linear-tickets/inspect-linear-connection https://api.uselemma.ai/openapi.json get /projects/{project_id}/linear/connection Returns retained Linear connection health and project defaults. Disabled rollout state does not remove stored history. # Link confirmed Linear ticket Source: https://docs.uselemma.ai/api-reference/linear-tickets/link-confirmed-linear-ticket https://api.uselemma.ai/openapi.json put /issues/{issue_id}/linear-ticket # Preview exact Linear ticket Source: https://docs.uselemma.ai/api-reference/linear-tickets/preview-exact-linear-ticket https://api.uselemma.ai/openapi.json post /issues/{issue_id}/linear-ticket/preview # Unlink Linear ticket Source: https://docs.uselemma.ai/api-reference/linear-tickets/unlink-linear-ticket https://api.uselemma.ai/openapi.json delete /issues/{issue_id}/linear-ticket # Accept monitor suggestion Source: https://docs.uselemma.ai/api-reference/monitors/accept-monitor-suggestion https://api.uselemma.ai/openapi.json post /projects/{project_id}/monitors/{monitor_id}/suggestions/{suggestion_id}/accept Activates a new version that replaces one phrasing with the suggested sentence. Past results stay as they are. # Create monitor Source: https://docs.uselemma.ai/api-reference/monitors/create-monitor https://api.uselemma.ai/openapi.json post /projects/{project_id}/monitors Creates a signal or tool monitor and starts learning. A tool monitor requires the tool name and how an issue in those traces is recognized. Optional guidance is used only while learning and is not returned. # Delete monitor Source: https://docs.uselemma.ai/api-reference/monitors/delete-monitor https://api.uselemma.ai/openapi.json delete /projects/{project_id}/monitors/{monitor_id} Deletes a monitor. Deleted monitors are omitted from reads and are no longer judged. # Dismiss a monitor issue Source: https://docs.uselemma.ai/api-reference/monitors/dismiss-a-monitor-issue https://api.uselemma.ai/openapi.json post /projects/{project_id}/monitors/{monitor_id}/issues/{issue_id}/dismiss Dismisses the issue for the whole project. The close records this monitor as the source. Optional reason is an issue dismissal reason. # Dismiss monitor suggestion Source: https://docs.uselemma.ai/api-reference/monitors/dismiss-monitor-suggestion https://api.uselemma.ai/openapi.json post /projects/{project_id}/monitors/{monitor_id}/suggestions/{suggestion_id}/dismiss Drops a pending phrasing suggestion. The saved phrasing stays. # Get monitor summary Source: https://docs.uselemma.ai/api-reference/monitors/get-monitor-summary https://api.uselemma.ai/openapi.json get /projects/{project_id}/monitors/{monitor_id}/summary Returns the fit rate, fit count, traces seen, and a chart series for the window. Counts are exact over the latest judgment of each trace. recalibrating is true during a cron or manual recalibrate, and suggestions lists pending phrasing changes. feedback_consumed_through is the latest succeeded cron or manual run's start, or null when none has succeeded. The window includes start and excludes end. # List monitor feedback Source: https://docs.uselemma.ai/api-reference/monitors/list-monitor-feedback https://api.uselemma.ai/openapi.json get /projects/{project_id}/monitors/{monitor_id}/feedback Lists corrections on the monitor, newest first, including ones that were undone. # List monitor issues Source: https://docs.uselemma.ai/api-reference/monitors/list-monitor-issues https://api.uselemma.ai/openapi.json get /projects/{project_id}/monitors/{monitor_id}/issues Lists issues that have occurrences from this monitor in the window. Counts and last_seen_at cover only those occurrences. Dismissed issues are included. # List monitor summaries Source: https://docs.uselemma.ai/api-reference/monitors/list-monitor-summaries https://api.uselemma.ai/openapi.json get /projects/{project_id}/monitors/summaries Returns one summary per live monitor for the window. Each row includes whether a cron or manual recalibrate is running and any pending phrasing suggestions. The window includes start and excludes end. # List monitor traces Source: https://docs.uselemma.ai/api-reference/monitors/list-monitor-traces https://api.uselemma.ai/openapi.json get /projects/{project_id}/monitors/{monitor_id}/traces Lists traces the monitor judged as fitting in the window, newest first. `total` is the fitting-trace count in the window. Pass offset for a page. The window includes start and excludes end. # List monitors Source: https://docs.uselemma.ai/api-reference/monitors/list-monitors https://api.uselemma.ai/openapi.json get /projects/{project_id}/monitors Lists monitors in the project. Deleted monitors are omitted. # List monitors for a trace Source: https://docs.uselemma.ai/api-reference/monitors/list-monitors-for-a-trace https://api.uselemma.ai/openapi.json get /projects/{project_id}/traces/{trace_id}/monitors Lists live monitors and whether the trace currently counts toward each one. An active correction is included when one exists. # Recalibrate monitor Source: https://docs.uselemma.ai/api-reference/monitors/recalibrate-monitor https://api.uselemma.ai/openapi.json post /projects/{project_id}/monitors/{monitor_id}/recalibrate Starts a manual recalibrate from feedback. The saved format changes only when the new one scores better, and phrasing changes stay suggestions. # Record monitor feedback Source: https://docs.uselemma.ai/api-reference/monitors/record-monitor-feedback https://api.uselemma.ai/openapi.json post /projects/{project_id}/monitors/{monitor_id}/feedback Records a correction for a trace. Verdict exclude drops the trace from the fit count, chart, and trace list. Verdict include puts the trace on those reads, including a trace the monitor has not judged. An optional reason is stored with the correction. Repeating the same verdict returns the existing row. # Restore a monitor issue Source: https://docs.uselemma.ai/api-reference/monitors/restore-a-monitor-issue https://api.uselemma.ai/openapi.json post /projects/{project_id}/monitors/{monitor_id}/issues/{issue_id}/restore Sets the issue back to open. The change is global, not limited to this monitor. # Restore monitor feedback Source: https://docs.uselemma.ai/api-reference/monitors/restore-monitor-feedback https://api.uselemma.ai/openapi.json post /projects/{project_id}/monitors/{monitor_id}/feedback/{feedback_id}/restore Restores an undone correction and makes it the active correction for that trace. # Undo monitor feedback Source: https://docs.uselemma.ai/api-reference/monitors/undo-monitor-feedback https://api.uselemma.ai/openapi.json post /projects/{project_id}/monitors/{monitor_id}/feedback/{feedback_id}/undo Undoes a correction. Repeating undo returns the same row. # Update monitor Source: https://docs.uselemma.ai/api-reference/monitors/update-monitor https://api.uselemma.ai/openapi.json patch /projects/{project_id}/monitors/{monitor_id} Changes the name, trace_phrasing, or issue_phrasing (tool monitors only). Send at least one; omitted fields keep their value. A phrasing change activates a new version with that phrasing, keeps the saved formats, and supersedes pending suggestions for it. Past results stay as they are. Any other field is rejected. # Create project Source: https://docs.uselemma.ai/api-reference/projects/create-project https://api.uselemma.ai/openapi.json post /projects Create a project for the authenticated tenant. # Describe project analytics catalog Source: https://docs.uselemma.ai/api-reference/projects/describe-project-analytics-catalog https://api.uselemma.ai/openapi.json get /projects/{project_id}/analytics/catalog Returns the events, fields, filter operations, aggregations, and cost measures that a project analytics query can use. # Get instrumentation diagnostics settings Source: https://docs.uselemma.ai/api-reference/projects/get-instrumentation-diagnostics-settings https://api.uselemma.ai/openapi.json get /projects/{project_id}/instrumentation-diagnostics Returns the instrumentation-diagnostics settings for a project owned by the authenticated tenant. # Get project Source: https://docs.uselemma.ai/api-reference/projects/get-project https://api.uselemma.ai/openapi.json get /projects/{project_id} Returns a single project if it belongs to the authenticated tenant. # Get project analytics Source: https://docs.uselemma.ai/api-reference/projects/get-project-analytics https://api.uselemma.ai/openapi.json get /projects/{project_id}/analytics Returns one analytics payload selected by the required view discriminator. Specialized views do not compute unrequested panels. Cost figures are Lemma estimates from observed token counts and public LiteLLM list prices — not invoice amounts. Backfilled rows carry the price as of the backfill run, not the price as of the call. # Get project briefs settings Source: https://docs.uselemma.ai/api-reference/projects/get-project-briefs-settings https://api.uselemma.ai/openapi.json get /projects/{project_id}/briefs Returns the Slack briefs settings for a project owned by the authenticated tenant. # Inspect project setup status Source: https://docs.uselemma.ai/api-reference/projects/inspect-project-setup-status https://api.uselemma.ai/openapi.json get /projects/{project_id}/setup-status Returns setup state, scope, permissions, and stable blocker codes without exposing credentials or provider-private data. # List instrumentation diagnostics findings Source: https://docs.uselemma.ai/api-reference/projects/list-instrumentation-diagnostics-findings https://api.uselemma.ai/openapi.json get /projects/{project_id}/instrumentation-diagnostics/findings Returns the complete active instrumentation-diagnostics findings for a project owned by the authenticated tenant. Empty when the rollout gate or project preference is off. # List projects Source: https://docs.uselemma.ai/api-reference/projects/list-projects https://api.uselemma.ai/openapi.json get /projects Returns projects available to the authenticated tenant. # Query project analytics Source: https://docs.uselemma.ai/api-reference/projects/query-project-analytics https://api.uselemma.ai/openapi.json post /projects/{project_id}/analytics/query Runs one catalog aggregate over stored analytics for a project. # Reorder projects Source: https://docs.uselemma.ai/api-reference/projects/reorder-projects https://api.uselemma.ai/openapi.json post /projects/reorder Persist project ordering for the authenticated tenant. # Update instrumentation diagnostics settings Source: https://docs.uselemma.ai/api-reference/projects/update-instrumentation-diagnostics-settings https://api.uselemma.ai/openapi.json patch /projects/{project_id}/instrumentation-diagnostics Enable or disable the instrumentation-diagnostics digest for the authenticated tenant. # Update project Source: https://docs.uselemma.ai/api-reference/projects/update-project https://api.uselemma.ai/openapi.json patch /projects/{project_id} Update a project for the authenticated tenant. # Update project briefs settings Source: https://docs.uselemma.ai/api-reference/projects/update-project-briefs-settings https://api.uselemma.ai/openapi.json patch /projects/{project_id}/briefs Enable or disable Slack briefs and optionally set cadence for the authenticated tenant. # Bulk delete traces Source: https://docs.uselemma.ai/api-reference/traces/bulk-delete-traces https://api.uselemma.ai/openapi.json post /traces/bulk-delete Delete multiple traces for a project. # Check per-trace ingest status Source: https://docs.uselemma.ai/api-reference/traces/check-per-trace-ingest-status https://api.uselemma.ai/openapi.json get /traces/ingest-status Returns whether a specific otel_trace_id is queued (`enqueued`), stored (`ingested`), ready (`ready`), or not found (`not_found`). # Check ready traces Source: https://docs.uselemma.ai/api-reference/traces/check-ready-traces https://api.uselemma.ai/openapi.json get /traces/has-ready Returns whether a project has ready traces. # Delete trace Source: https://docs.uselemma.ai/api-reference/traces/delete-trace https://api.uselemma.ai/openapi.json delete /traces/{trace_id} Delete a trace for a project. # Discover trace search facets Source: https://docs.uselemma.ai/api-reference/traces/discover-trace-search-facets https://api.uselemma.ai/openapi.json get /traces/search/facets Returns curated, bounded facets for the active trace-search context. # Get trace Source: https://docs.uselemma.ai/api-reference/traces/get-trace https://api.uselemma.ai/openapi.json get /traces/{trace_id} Loads a single trace with spans and stats. Pass expand=context to return the agent context bundle (row, stats, spans, and issue occurrences) instead of the default detail payload. Default (no expand) responds with the get_trace detail schema; expand=context responds with the get_trace_context bundle schema. With expand=context, spans_limit/spans_offset page the bundle's spans (spans_total reports the stored count) and include_attributes/include_events=false drop the payload-heavy span fields. # List thread issue summaries Source: https://docs.uselemma.ai/api-reference/traces/list-thread-issue-summaries https://api.uselemma.ai/openapi.json post /projects/{project_id}/thread-issue-summaries Batch-fetch issue extraction completion and dev issues for thread_ids (join key for eval trial_hash). # List thread traces Source: https://docs.uselemma.ai/api-reference/traces/list-thread-traces https://api.uselemma.ai/openapi.json get /traces/{trace_id}/thread Thread traversal for a ready trace. scope=all (default) returns every ready member oldest first using the threadTraces schema; scope=neighbors returns previous/next ids using the threadTraceNeighbors schema. Empty for standalone traces. # List trace agent names Source: https://docs.uselemma.ai/api-reference/traces/list-trace-agent-names https://api.uselemma.ai/openapi.json get /traces/agent-names Returns agent names seen in project traces. # List trace issue occurrences Source: https://docs.uselemma.ai/api-reference/traces/list-trace-issue-occurrences https://api.uselemma.ai/openapi.json get /traces/{trace_id}/issue_occurrences Returns issue occurrence evidence linked to a ready trace. # List trace spans Source: https://docs.uselemma.ai/api-reference/traces/list-trace-spans https://api.uselemma.ai/openapi.json get /traces/{trace_id}/spans Returns spans for a ready trace in start-time order. Optional limit/offset pages the result and include_attributes/include_events=false drop the payload-heavy span keys. Default returns every span with all fields. The bare array is intentional (continuation is detected by a short or empty page, no total is reported); the expand=context bundle reports spans_total instead. # List traces Source: https://docs.uselemma.ai/api-reference/traces/list-traces https://api.uselemma.ai/openapi.json get /traces/dashboard Canonical first-party dashboard list with cursor pagination and independent has_error/has_issues filters. Kept separate from GET /traces because that legacy window list uses a different backend and response shape. # Search trace evidence Source: https://docs.uselemma.ai/api-reference/traces/search-trace-evidence https://api.uselemma.ai/openapi.json get /traces/search/evidence Search visible span content, grouped into trace-level results. Distinct from structural search. # Search traces (structural) Source: https://docs.uselemma.ai/api-reference/traces/search-traces-structural https://api.uselemma.ai/openapi.json get /traces/search Structural search over traces and span attributes. Distinct from evidence search, which searches visible span content. # Platform Source: https://docs.uselemma.ai/changelog Changes to the Lemma platform, including new features, improvements, and bug fixes. ## Audit logs * **Read:** An owner or admin lists organization audit events with `GET /audit-events`. An org-wide admin API key can list them too. The default window is the last 7 days, and a window wider than 90 days is rejected. A project-scoped key is rejected. * **CSV export:** `GET /audit-events/export` downloads the same window and filters as CSV, under the same access rules. More than 10,000 matching events is rejected, and Lemma records the export. ## Issues * **Completed tasks:** After the agent delivers the requested outcome, detection looks for an [issue](/platform/issues), including a confirmed hallucination. ## Analytics * **Catalog:** `GET /projects/{project_id}/analytics/catalog` lists each event and its measures. The MCP tool is `describe_project_analytics`. * **Query:** `POST /projects/{project_id}/analytics/query` selects one event and aggregates its measures. The MCP tool is `query_project_analytics`. [Querying analytics](/guides/query-analytics) shows the query body. ## Analytics * **90-day history:** [Analytics](/platform/analytics) keeps traces, generations, tools, and errors for 90 days. REST and MCP use the same views. ## Traces * **Processed traces:** With **Show unprocessed** off, each [traces](/platform/traces) page fills with processed traces, so you don’t get a page of empty slots. The traces API accepts `include_unprocessed`. ## Analytics * **Errors headline:** The [Analytics](/platform/analytics) errors headline counts span errors. **Error rate** and the agent scorecard count runs whose root span failed. ## Issues * **Failure definition:** Detection uses one failure definition from extraction through investigation for an [issue](/platform/issues). ## Slack * **Priority alerts:** A priority alert shows the issue name, agent, confirmation writeup, sparkline, and actions. Each export mints a fresh link that lasts 7 days. Alerts no longer append a markdown dump of the thread. See [Slack](/connections/slack). ## Audit logs * **Sensitive actions:** Audit logs record auth, membership, API key, integration, export, delete, and billing actions. Lemma keeps them for at least 90 days. ## Bug fixes * **API keys settings:** The API keys settings page finishes loading. A failed request shows the error instead of spinning. ## API keys * **Project scope:** You can limit an API key to one project. Calls through REST or the Model Context Protocol (MCP) to any other project are rejected. ## Issues * **Confirmation:** Issue lists, issue details, and the issue API show whether an issue is unconfirmed or confirmed. They also show the proof writeup and any dismissal note. * **Merging:** Merging matches issues on where the failure happened and what failed. It ignores a parameter that appears on only one occurrence, which cuts duplicate issue groups. * **Validation fields:** The issue API and the dashboard no longer return the old validation status, confidence, and evidence fields. ## Slack * **Confirmed alerts:** Slack alerts wait until Lemma confirms the issue, and a confirmed alert includes the proof. * **Instrumentation error alerts:** Project Slack settings include an **Instrumentation error alerts** toggle. Turning it off stops the Slack digests and hides the dashboard banner. The control explains what these errors are. * **Channel picker:** The Slack alert channel picker searches every public channel, including a workspace with more than 500 channels. ## Support access * **Owner controls:** An organization owner can turn Lemma support access off, or limit how long it lasts. An Enterprise organization starts with support access off. ## Settings * **Project settings:** Project settings split into **General**, **Integrations**, and **Developer**. Cards show whether a connection is live. ## API keys * **Scopes:** API keys have ingest, read, and admin scopes. An existing key stays admin. A new key defaults to ingest. The page shows who created the key, when it was last used, and when it expires. Admins get an alert before expiry. ## Artifacts * **Coding agents:** A coding agent can update uploaded [artifacts](/platform/artifacts) and append learned context through the [Lemma MCP server](/connections/mcp). It reads the current text before it writes, so it doesn’t overwrite a newer edit. Constraints you wrote stay in place. ## Slack * **Alert channel:** You choose the Slack alert channel in the dashboard. Lemma retired `/lemma link` and `/lemma unlink`. An unlinked mention points you to settings. ## Issues * **Flag fields:** When you flag an issue, the public JSON fields use a `flagged_` prefix. Those fields are separate from confirmation fields. ## Webhooks * **Delivery:** [Webhook](/connections/webhooks) delivery rejects an unsafe or internal destination. Only an owner or admin can change organization settings. Lemma publishes `/.well-known/security.txt`. ## Billing * **Annual billing:** Pro checkout offers annual billing in billing settings, onboarding, and starter trial plan changes. ## API * **Trace search errors:** An invalid trace search returns HTTP 400 with a message you can act on, in REST and MCP. It no longer returns a generic server error. * **Catalog:** The public API and MCP catalog drops unused context and release tools. Trace and issue tools use their current names. An analytics call has to name one view. ## Billing * **Checkout retry:** Lemma replaces an expired Stripe Checkout session, so you can retry checkout. You no longer stay stuck on the organization checkout lock. ## Issues * **Issue type:** Issue details label the issue type in the side rail. ## MCP * **Missing capabilities:** An MCP client can call `get_more_tools` to report a capability the current catalog doesn’t include. ## Traces * **Unprocessed rows:** The traces list hides unprocessed rows by default. **Show unprocessed** keeps your choice. * **Failed tool calls:** A failed tool call in the trace drawer is one line, with the failure in red. The full error stays on the span. ## Issues * **Fingerprints:** An issue fingerprint attaches to the failed operation. Grouping treats the same failure at equivalent sites as one issue, and it keeps tool names exact. * **Copy to markdown:** **Copy to markdown** is a compact icon in the issue actions. ## Slack * **Instrumentation digests:** An instrumentation Slack digest includes every agent that has findings, even when more than three agents are affected. ## Issues * **Fingerprints:** Issue grouping keys each new occurrence by where it failed, what failed, and the parameter involved. Headers, the issue API, and [webhooks](/connections/webhooks) show that fingerprint instead of category, subsystem, failure class, or canonical mechanism. * **Thread status:** Thread extraction status covers every analyzed turn, so an earlier turn no longer looks unsampled. * **Root causes:** REST and MCP can read recurring root causes and the issues linked to them. ## Traces * **Span paging:** REST and MCP trace reads can page through spans and return a trimmed span list. A context bundle reports the total span count. A time bound accepts an ISO 8601 date or epoch milliseconds. Lemma rejects an invalid value. * **Diagnostics:** Diagnostics name incomplete model, history, or input and output instrumentation, instead of calling the generation untyped. ## Bug fixes * **API edge errors:** If the API edge cannot reach the service, the browser gets a JSON 502 and consistent cross-origin headers. ## Billing * **Pro trials:** A new Pro trial stays on Pro when it ends. You can move to Lemma Jr. before the trial ends. * **Plan controls:** Trial plan controls live in **Manage organization → Billing**, including **Keep Pro** and **Reactivate**. ## Inspect * **Product questions:** [Inspect](/platform/inspect) and the Slack agent answer product, SDK, API, and instrumentation questions from the public docs. Answers cite the heading. ## Bug fixes * **Project pages:** Project pages in the dashboard load again. * **Search loading:** Trace search shows a loading indicator while results load. ## Billing * **Pro trials:** A new organization starts a card-required 14-day Pro trial. Billing lives in **Manage organization**, with plan, status, upgrade, and cancel. A planless or lapsed organization picks a plan during onboarding. That trial was set to end on Lemma Jr. unless you kept Pro. ## Traces * **Search:** Trace search opens a dropdown built from your data. Clicking a row selects that result. You can edit a saved view without losing it, and you can save a search in one click. ## Slack * **Download markdown:** A Slack issue alert includes **Download .md**. Each click mints a new private link. ## Issues * **Empty projects:** A project with no issues shows the explainer video. ## Bug fixes * **Invites:** A signed-out invitee creates an account and returns to the invite. Deleting an organization sends you to a safe page instead of a crashed dashboard. ## Inspect * **Conversation layout:** Inspect centers the conversation title and uses shorter history controls. Suggestions line up with the conversation. While it answers, it shows **Thinking**. ## Issues * **Dismissed issues:** The issues list hides dismissed issues unless you filter for them. Rows that are already on screen stay there while the list refreshes. An empty list and a failed load each state what happened. ## Billing * **Trials and subscriptions:** You can still create an account and an organization. The dashboard and Slack support require an active trial or subscription. ## Issues * **Review budget:** The remaining review budget counts threads, not traces. A long chat no longer uses the entire sample. * **Grouping metadata:** A newly grouped issue keeps where it failed, what failed, and the parameter involved. ## Slack * **Priority alerts:** A Slack priority alert leads with why the issue needs attention. It uses a compact sparkline instead of a tall 14-day chart. Reconnect Slack if an existing connection cannot upload the chart. * **Token counts:** Instrumentation diagnostics flag a generation span that is missing input or output token counts. They also explain the effect on the cost estimate. ## Issues * **Recurring issues:** The issues list opens on recurring issues. Less common issues sit below a divider. Add `expanded=true` to a shared link to show them. * **Grouping:** Grouping matches occurrences that share the same code defect, using the cited trace evidence. * **Merge:** You can merge compatible issues from the issues table. Pick the issue that should survive, then merge in the dashboard, the REST API, or MCP. * **Bulk actions:** You can flag, unflag, review, resolve, dismiss, assign, and tag up to 50 selected issues. A partial failure lists the rows to retry. * **Categories:** Issue categories stay consistent in filters, sorting, list rows, and detail headers. * **Issue details:** Issue details hide a raw ID for a removed member. An active issue title can refresh from recent occurrences. ## API * **Bulk updates:** REST and MCP can read and update up to 50 issues in one call. Lemma consolidated the trace, issue, and analytics routes. Older compatible aliases still work. Lemma deprecated those aliases. * **Public responses:** Customer REST and MCP responses omit internal-only fields and tools. An unconfigured API origin returns “Service unavailable”. * **OpenAPI:** OpenAPI and MCP descriptions use product language. Deprecated REST routes stay callable, and the public spec hides them. ## Dashboard * **Setup to-dos:** Dashboard setup to-dos clear after you connect MCP, upload context, or dismiss an issue. ## Accounts * **Deletion:** An owner can delete an organization. If you belong to no organization, you can delete your account from **Account Security**. # Docs Source: https://docs.uselemma.ai/changelog/docs Changes to the Lemma docs, listed by the day they were published. ## Guides * **Analytics query:** [Querying analytics](/guides/query-analytics) explains the query body and response, with volume, tool-failure, model-cost, errors-by-agent, and join examples for REST and MCP. * **Analytics skill:** [Querying analytics](/guides/query-analytics) opens with how to install the `lemma-analytics` skill so a coding agent can answer volume, latency, tool-failure, and cost questions. ## Reference * **Analytics query API:** [Analytics query API](/reference/analytics-query) lists every analytics event and field, request key, limit, error, and preset view. ## Platform * **Analytics:** The [Analytics](/platform/analytics) page points to [Querying analytics](/guides/query-analytics) and the analytics skill. ## Changelog * **Platform:** The [platform changelog](/changelog) lists user-facing product changes by day. * **SDKs:** The [TypeScript SDK](/changelog/typescript) and [Python SDK](/changelog/python) changelogs list changes by version. * **Docs:** This page lists documentation changes by day. ## Reference * **Ingest status:** The [trace contract](/reference/trace-contract) says ingest keeps exact `OK` and `ERROR`, ignores every other value, and lets a non-empty error force `ERROR`. ## Getting started * **Page titles:** Page titles name the task. The introduction title is “Production monitoring for AI agents.” ## Integrations * **LangChain:** The LangChain guide documents that an idle open trace is finalized and sent after the open-trace TTL. ## Guides * **From issue to fix:** [Turning issues into fixes](/guides/turning-issues-into-fixes) walks one production issue from the evidence through a code change and a check that it does not recur. ## Platform * **Issues:** The issues and overview pages link that guide. ## Guides * **Runnable examples:** [Runnable examples](/guides/examples) points at one docs-chat agent implemented for each first-party stack, plus a direct SDK path. ## Integrations * **Wiring:** The LangChain, LangGraph, Mastra, OpenAI Agents, and Vercel AI SDK pages show the handler, exporter, or telemetry helper for that example. ## Tracing * **Cross-process turns:** [Cross-process turns](/tracing/instrumentation/cross-process-turns) explains how a host and a sandbox record one trace. * **Fail open:** The traces page says automatic SDK delivery catches network errors, and `ingest()` still raises. ## Platform * **Product pages:** Platform and connection pages match the public product: workspace sequence, ready traces, and the coding-agent setup path. ## Connections * **Webhooks and MCP:** Webhook and MCP pages replace the previous incident and monitor copy. ## Navigation * **Page menu:** Each page can be copied, opened as markdown, sent to the docs MCP, or opened in Cursor or Claude Code. * **Introduction:** The introduction screenshot sits under the opening paragraph. ## Platform * **Platform guides:** Pages for overview, concepts, traces, issues, Inspect, artifacts, and analytics describe the dashboard. ## Connections * **Connections:** Pages for overview, Slack, Linear, MCP, and webhooks describe how Lemma connects to your tools. ## Integrations * **Vercel AI SDK:** The integration page says a generation records the provider system when the call sets messages. ## Navigation * **Setup path:** Instrumentation setup is the getting-started path for the SDK. The standalone quickstart and tracing overview pages were folded into that guide. * **Platform group:** Platform pages have their own nav group, separate from tracing. ## Tracing * **Release:** Setup explains how to set `release`, including identifiers produced in CI. ## Getting started * **Trace quality:** Getting started documents the diagnostics skill for reviewing trace quality. ## Tracing * **Metadata:** The context page says any metadata key is searchable. ## Integrations * **Token usage:** LangChain, LangGraph, Mastra, OpenAI Agents, and Vercel AI SDK pages document generation token usage. ## Tracing * **Release:** Setup, the trace contract, and the integration pages document the client `release` field. ## Tracing * **Inputs and outputs:** Integration pages and the trace contract say prompts, tool payloads, model output, and error messages are always recorded. ## Reference * **Analytics attributes:** [Analytics telemetry](/reference/analytics-telemetry) lists the generation attributes the SDK emits, and the generations page documents them. ## Tracing * **Tool messages:** [Tool calls](/tracing/instrumentation/tool-calls) documents `userFacingMessage` and `user_facing_message`. ## Integrations * **Mastra:** The Mastra page says the trace name comes from the agent entity. ## Integrations * **Trace contract:** The Vercel AI SDK, OpenAI Agents, LangChain, and LangGraph pages describe current-turn input and output, thread and user, and nested tool calls. ## Integrations * **Mastra:** A Mastra page documents `LemmaMastraExporter`. ## Tracing * **One delivery:** Traces, context, and troubleshooting say each execution sends one complete trace. `TraceHandle.flush` is no longer documented. * **Second delivery:** Troubleshooting describes a root whose input and output are null after a second delivery. ## Guides * **Quality guide:** Building high-quality traces lives under Guides. ## Reference * **Trace contract:** The trace contract specifies what clients send and which fields unlock product behavior. The undocumented environment field is gone from the contract. ## Guides * **Quality:** A guide covers what makes a trace useful to Lemma. * **Migrations:** Guides cover moving from Raindrop, Langfuse, and OpenTelemetry. ## Tracing * **Debug verify:** The debug mode page says verification reads per-trace ingest status. ## Tracing * **Debug mode:** The debug mode page documents delivery logs and the smoke test. * **Debug flag:** `LEMMA_DEBUG=1` is documented as the preferred value. `true` still works. ## Getting started * **Skill install:** Getting started shortens the tracing skill install path. ## Getting started * **Public docs:** The docs site opens with an introduction and the tracing skill install path. ## Tracing * **Instrumentation:** Tracing pages cover setup, traces, generations, tool calls, spans, and context. ## Integrations * **First-party stacks:** Pages cover LangChain, OpenAI Agents, and the Vercel AI SDK. # Python SDK Source: https://docs.uselemma.ai/changelog/python Changes to uselemma-tracing, listed by version. ## LangChain * **Root output:** Reasoning content blocks become the answer string on the root trace output. The generation span keeps the full block list. Tool-call payloads stay unchanged. * **Empty results:** A successful non-root chain or tool that returns `None` ends with `{"result": "none"}`. Root traces and errors are unchanged. ## LangChain * **Idle traces:** An open trace with no activity for `open_trace_ttl` (default 2 hours) is finalized and sent. A long-running loop is not closed mid-run. The sweep runs from chain, model, and tool starts, at most once per eviction interval. ## Tracing * **Before send:** `before_send` can redact or drop an ingest payload without a custom HTTP transport. * **Payload cap:** `max_payload_bytes` caps the payload. `deduplicate_span_content` shrinks repeated history before the cap is applied. ## LangChain * **Span filter:** `exclude_span_names` and `include_span` drop spans, including noisy LangGraph middleware, while children stay attached to the nearest kept ancestor. * **Model name:** When a chat model is not serializable, the handler reads `ls_model_name` from run metadata after invocation params. ## Ingest * **User-Agent:** A custom `transport` sends `uselemma-tracing/` unless you set `User-Agent` yourself. ## LangChain * **LangSmith middleware:** When LangSmith wraps LangChain middleware and skips `on_chain_start`, nested callbacks stay on the active turn instead of becoming extra root traces. ## Tool calls * **Failure payloads:** A tool result with `success: false` or `status` of `error` or `failed` is recorded as an error. The span uses `structuredContent.error` when that field is set. ## LangChain * **Callback flags:** The handler subclasses LangChain’s callback base when `langchain_core` is installed, and sets `run_inline` plus the other 1.x flags. LangChain 1.6 can call the handler without crashing before a span is recorded. ## Generations * **Token counts:** Non-finite numbers, including infinity, are dropped from numeric usage fields. ## Tracing * **One turn, two processes:** [start\_turn](/tracing/instrumentation/cross-process-turns), `attach_turn`, and `apply` record a host and a sandbox as one trace. The child keeps a journal and does not call ingest. The host sends once. `turn.end()` still raises on failure. ## Ingest * **Fail open:** `trace()`, `async_trace()`, and `TraceHandle.end()` catch HTTP and network errors, log them, and do not rethrow. [ingest()](/tracing/instrumentation/traces) still raises. A delivery failure does not replace your agent’s error. ## Generations * **Model identity:** A typed generation span copies the model from the invocation, the response, or response metadata when params omit it. The model is omitted on end when it is still unset. Completion text is not used as a model id. ## Tool calls * **Encoded failures:** A tool payload that encodes a failure is stored as a span error. ## Generations * **Token usage:** LangChain and OpenAI Agents record token usage, including cache and reasoning tokens from Agents SDK details. * **Provider:** Generation spans set the provider system from the model provider. ## Tracing * **Release:** Pass `release` to the client, or set `LEMMA_RELEASE`, to stamp the running revision on each trace. * **Provenance:** Spans record the SDK language and integration so missing usage is distinct from a real zero. ## Tracing * **Inputs and outputs:** Prompts, tool inputs, tool outputs, model output, and error messages are always recorded. The `record_inputs` and `record_outputs` arguments are removed. Passing them raises `TypeError`. Redact sensitive values before they reach the agent. ## Tracing * **Error messages:** Empty and missing failure text uses the same message in the Python and TypeScript SDKs. Serialized payloads use compact JSON. ## Ingest * **User-Agent:** SDK HTTP requests send `uselemma-tracing/`. ## Tool calls * **User-facing text:** Set [`user_facing_message`](/tracing/instrumentation/tool-calls) when a tool sends text directly to the person. Lemma shows that text as the message they received. ## OpenAI Agents * **Trace contract:** The processor records current-turn input and output, terminal errors, and span-bounded times. Soft tool errors are included. ## LangChain * **Trace contract:** Owned runs keep current-turn input, output, and errors. Tool calls stay nested under one root until the final answer. LangGraph uses the LangChain adapter. ## Tool calls * **MCP errors:** A tool result with `isError` is recorded as an error, without an invented output. * **Child timing:** A tool span no longer ends after its parent generation. ## Ingest * **One delivery:** The `replace` option is removed. Each execution sends one complete trace. Patching a trace over time is not supported. ## Tracing * **Environment field:** The undocumented `environment` client field is removed. ## Debug * **Ingest status:** `LEMMA_DEBUG_VERIFY` checks `GET /traces/ingest-status` for up to 15 seconds, so a backlog does not look like a missed delivery. * **Debug flag:** `LEMMA_DEBUG=1` is the preferred value. `true` still works. ## Debug * **Delivery logs:** Debug mode logs client config and ingest responses, including failure hints. * **Smoke test:** Debug verification checks that a trace reached Lemma. ## Ingest * **Send on end:** Trace handles and the OpenAI Agents processor send a trace once, when it ends. They no longer send partial snapshots along the way. ## Tracing * **First public release:** `uselemma-tracing` sends traces to Lemma, with debug mode and experiment mode. ## LangChain * **Callback handler:** A LangChain callback handler records chains, model calls, and tools. ## OpenAI Agents * **Processor:** An OpenAI Agents processor records agent runs. # TypeScript SDK Source: https://docs.uselemma.ai/changelog/typescript Changes to @uselemma/tracing, listed by version. ## LangChain * **Root output:** Reasoning content blocks become the answer string on the root trace output. The generation span keeps the full block list. Tool-call payloads stay unchanged. ## LangChain * **Model name:** When a chat model is not serializable, the handler reads `ls_model_name` from run metadata after invocation params. ## Tool calls * **Failure payloads:** A tool result with `success: false` or `status` of `error` or `failed` is recorded as an error. The span uses `structuredContent.error` when that field is set. ## LangChain * **Callback handler:** `LemmaLangChainCallbackHandler` accepts LangChain 1.x callback arguments and sets `run_inline`, `raise_error`, and the `ignore_*` flags. ## Vercel AI * **Telemetry types:** The Vercel AI telemetry object is assignable to the AI SDK `Telemetry` type without a cast. ## Mastra * **Exporter types:** `exportTracingEvent` accepts the framework event shape, so you can pass the exporter without a cast. ## Tracing * **One turn, two processes:** [startTurn](/tracing/instrumentation/cross-process-turns), `attachTurn`, and `apply` record a host and a sandbox as one trace. The child keeps a journal and does not call ingest. The host sends once. `turn.end()` still raises on failure. ## Ingest * **Fail open:** `trace()` and `TraceHandle.end()` catch HTTP and network errors, log them, and do not rethrow. [ingest()](/tracing/instrumentation/traces) still raises. A delivery failure does not replace your agent’s error. ## Generations * **Model identity:** A typed generation span copies the model from the invocation, the response, or response metadata when params omit it. Completion text is not used as a model id. ## Vercel AI * **Provider system:** A generation records the provider system when the call sets `messages`. ## Tool calls * **Encoded failures:** A tool payload that encodes a failure is stored as a span error. ## Tracing * **OpenCode:** `codingAgentTurnTrace` accepts the `opencode` harness id. ## Tracing * **Coding-agent turns:** `codingAgentTurnTrace` builds one trace from a completed coding-agent turn: prompt, tool calls, and response. ## Generations * **Token usage:** LangChain, OpenAI Agents, Vercel AI SDK, and Mastra record token usage, including cache and reasoning tokens from Agents SDK and AI SDK 7 shapes. * **Provider:** Generation spans set the provider system from the model provider. ## Tracing * **Release:** Pass `release` to the client, or set `LEMMA_RELEASE`, to stamp the running revision on each trace. * **Provenance:** Spans record the SDK language and integration so missing usage is distinct from a real zero. ## Tracing * **Inputs and outputs:** Prompts, tool inputs, tool outputs, model output, and error messages are always recorded. The `recordInputs` and `recordOutputs` options are removed. Redact sensitive values before they reach the agent. ## Tracing * **Error messages:** Empty and missing failure text uses the same message in the TypeScript and Python SDKs. Serialized payloads use compact JSON. ## Tool calls * **User-facing text:** Set [`userFacingMessage`](/tracing/instrumentation/tool-calls) when a tool sends text directly to the person. Lemma shows that text as the message they received. ## Mastra * **Agent name:** The trace name comes from the agent entity, then the entity name, then the id inside the run span name. An explicit `agentName` still wins. ## Vercel AI * **Trace contract:** One integration owns one in-flight run. The root records the current turn, thread, and user, with wall-clock times and structured output. ## OpenAI Agents * **Trace contract:** The processor records current-turn input and output, terminal errors, and span-bounded times. Soft tool errors match the Python SDK. ## LangChain * **Trace contract:** Owned runs keep current-turn input, output, and errors. Tool calls stay nested under one root until the final answer. LangGraph uses the LangChain adapter. ## Mastra * **Exporter:** `LemmaMastraExporter` maps agent and workflow spans to Lemma traces, with nested generations and tools. Soft tool failures and the current user turn are included. ## Tool calls * **MCP errors:** A tool result with `isError` is recorded as an error, without an invented output. * **Child timing:** A tool span no longer ends after its parent generation. ## Tracing * **Registry:** An ended trace handle is released from the client registry. ## Ingest * **One delivery:** The `replace` option is removed. Each execution sends one complete trace. Patching a trace over time is not supported. ## Ingest * **end() sends:** `TraceHandle.flush` is removed. `end()` is how a trace handle is sent. ## Tracing * **Environment field:** The undocumented `environment` client field is removed. ## Debug * **Ingest status:** `LEMMA_DEBUG_VERIFY` and `debugSmokeTest` check `GET /traces/ingest-status` for up to 15 seconds, so a backlog does not look like a missed delivery. * **Debug flag:** `LEMMA_DEBUG=1` is the preferred value. `true` still works. ## Debug * **Delivery logs:** Debug mode logs client config and ingest responses, including failure hints. * **Smoke test:** `debugSmokeTest` checks that a trace reached Lemma. ## Ingest * **Send on end:** Trace handles and the OpenAI Agents processor send a trace once, when it ends. They no longer send partial snapshots along the way. ## Tracing * **First public release:** `@uselemma/tracing` sends traces to Lemma, with debug mode and experiment mode. ## LangChain * **Callback handler:** A LangChain callback handler records chains, model calls, and tools. ## OpenAI Agents * **Processor:** An OpenAI Agents processor records agent runs. ## Vercel AI * **Telemetry:** A Vercel AI SDK telemetry integration records model and tool calls. # Linear Source: https://docs.uselemma.ai/connections/linear Linear tickets created or linked from Lemma issues Connecting Linear never creates or links a ticket by itself. It makes those actions available at three scopes: organization, project, and person. Set each one up before you expect ticket creation to work end to end. Linear ticket linking is enabled per organization. If you don’t see **Linear** in project settings, it isn’t enabled for your workspace. You’ll configure this from three places: **Manage organization → Integrations** for the workspace connection, **Settings → Linear** on the project for defaults, and **Account → Profile → Connected accounts** for your own identity. ## Connect the organization’s Linear workspace Open **Manage organization → Integrations**. Have an organization owner or admin connect or reconnect the workspace through OAuth. One workspace serves the whole Lemma organization, and a given Linear workspace can only belong to one Lemma organization at a time. If the connection starts failing, reconnect from organization settings. Lemma keeps existing ticket identities and last-known state through a degraded connection. Members and coding agents can’t install the workspace themselves, though they can check connection health for a project they have access to. ## Set project defaults Open the project’s **Settings → Linear**. Select a default Linear team, required before ticket creation works, and optionally a Linear project under that team. Saving defaults doesn’t create a ticket by itself. If Linear is disconnected at the organization level, an owner has to reconnect it before defaults do anything. ## Map your personal identity Each Lemma member who wants tickets assigned to them personally verifies their own Linear identity from **Account → Profile → Connected accounts**. This is separate from the organization’s workspace connection, and Lemma doesn’t expose a directory, so you can only map your own account, not pick someone else’s. Ordinary issue assignment inside Lemma doesn’t write to Linear. Combined assignment only happens when you create a ticket and choose an eligible, already-mapped Lemma member; that single action assigns both the Lemma issue and the new ticket. Any other ticket creation defaults to unassigned, and API keys or other non-human principals have no personal mapping at all. ## Create, link, and unlink tickets A Lemma issue can have at most one active Linear ticket, and a Linear ticket can back at most one active Lemma issue. From there: * **Create** a ticket explicitly, one issue at a time, using the project’s team and project defaults. Lemma writes a safe summary and a link back to the issue, never raw prompts or tool output. * **Link** to an exact Linear URL or identifier, with a preview and confirmation step. There’s no workspace-wide browsing or backfill. * **Unlink** explicitly when you need to replace a link; unlink first, then create or link again. Feature flags, open issues, and detection don’t auto-file tickets under any circumstance. The actions above are the only way a ticket gets created. ## How ticket status stays in sync * **Recurrence**: never creates, reopens, or replaces a ticket on its own. * **Reopen from matching evidence**: if a resolved issue reopens, Lemma comments once on the already-completed ticket instead of reopening it. Matching evidence never reopens a dismissed issue. * **Dismiss**: requests cancellation of the active ticket. A Linear `completed` status resolves the Lemma issue; `canceled` only updates the cached snapshot. * **Manual reopen**: reopening an issue in Lemma never alters the linked ticket. ## Related pages Status, assignment, and the issue that a ticket attaches to. When to use Linear versus Slack, MCP, or webhooks. The same create, link, and unlink capabilities through MCP where authorized. Alerts for issues; Slack never creates Linear tickets. # Lemma MCP server Source: https://docs.uselemma.ai/connections/mcp MCP for coding agents, plus external servers Lemma can call Lemma has two opposite Model Context Protocol (MCP) directions. Don’t mix them: * **Lemma MCP server**: your coding agent calls Lemma to list traces and issues, inspect evidence, and take authorized actions * **External MCP servers**: Lemma calls tools on a server you connect, so Inspect and analysis can use those tools This page is not the same connection as [Inspect](/platform/inspect). Inspect runs in-product. The Lemma MCP server is for editors and coding agents. The endpoint is `https://api.uselemma.ai/mcp/` (trailing-slash variants work the same), authenticated with `Authorization: Bearer your_organization_api_key`. Pass a project ID on project-scoped tools. Lemma only records MCP activation for a project after a successful project-scoped tool call, so saving the URL in Cursor isn’t activation by itself. ## Prerequisites * A Lemma organization with at least one project that has, or will have, traces * An organization API key from **API Keys** or the project get-started flow. Copy it immediately; Lemma won’t show it again. Keep it server-side in your coding tool’s MCP config, never in a public repo * An MCP client that supports Streamable HTTP, such as Cursor, Claude Desktop, or Claude Code ## Connect your coding agent Create an organization API key in the [Lemma dashboard](https://platform.uselemma.ai). Copy your key immediately after creating it. You won’t be able to see it again. In **Settings > MCP Servers**, add: ```json theme={null} { "mcpServers": { "lemma": { "url": "https://api.uselemma.ai/mcp/", "headers": { "Authorization": "Bearer your_organization_api_key" } } } } ``` Add to `~/Library/Application Support/Claude/claude_desktop_config.json`: ```json theme={null} { "mcpServers": { "lemma": { "type": "http", "url": "https://api.uselemma.ai/mcp/", "headers": { "Authorization": "Bearer your_organization_api_key" } } } } ``` Restart Claude Desktop after saving. Run this in your terminal: ```bash theme={null} claude mcp add --transport http lemma \ https://api.uselemma.ai/mcp/ \ --header "Authorization: Bearer your_organization_api_key" ``` Any MCP client that supports Streamable HTTP transport uses the same shape: ```json theme={null} { "mcpServers": { "lemma": { "url": "https://api.uselemma.ai/mcp/", "headers": { "Authorization": "Bearer your_organization_api_key" } } } } ``` Replace `your_organization_api_key` with the organization API key you copied. Ask your agent: ```text theme={null} Use Lemma to show me my last 5 traces, including any errors. ``` Include the project ID if your client hasn’t already selected one. Activation is recorded only after that project-scoped call succeeds. ## What tools exist REST and MCP share one operation list; see the [API Reference](/api-reference) for the full set. Results get truncated once they’re oversized, so prefer pagination over asking for every span at once. Use it to pull the last traces with errors, inspect a specific issue, export an issue as markdown, and create or link a Linear ticket when you’re authorized to. MCP clients require confirmation before running any write tool. ## Connect an external MCP server Connect an external MCP server when **Lemma** needs that server’s tools, not when a coding agent needs Lemma. Open **Settings → External MCP Connections**. Enter the server URL, then authenticate with OAuth or custom headers. Wait until the connection is `active`. `pending`, `expired`, `needs_reauth`, and `revoked` are not healthy. Add project validation instructions so Lemma knows how to use the tools. Remove a connection when Lemma should stop calling those tools. External MCP is not a substitute for the Lemma MCP server above, and it does not deliver [webhook](/connections/webhooks) events. ## Related pages The issues your coding agent can list and act on. In-product investigation. Inspect doesn’t use this MCP connection. When to use MCP versus Slack, Linear, or webhooks. The shared REST and MCP operation list. # Route findings into your workflow Source: https://docs.uselemma.ai/connections/overview Connect Lemma findings to Slack, Linear, MCP, and webhooks Integrations and connections solve two different problems. Integrations (OpenAI Agents, Vercel AI SDK, LangChain, LangGraph, and Mastra) attach Lemma tracing to your agent framework; they’re how traces get into Lemma in the first place. Connections take what Lemma finds after those traces exist and route it to the rest of your team’s workflow. They never instrument the agent itself. Pick a connection based on what should happen next: | Connection | Use it when | | - | - | | **[Slack](/connections/slack)** | The team should see issue alerts and optional Issue Briefs in a Slack channel you choose | | **[Linear](/connections/linear)** | Issues should become or link to Linear tickets, with an explicit create or link action, not automatic filing | | **[Lemma MCP server](/connections/mcp)** | A coding agent in Cursor, Claude Code, or similar should query this project’s traces and issues | | **External MCP servers** | Lemma itself should call tools on a server you connect. See [Connect an external MCP server](/connections/mcp#connect-an-external-mcp-server) | | **[Webhooks](/connections/webhooks)** | Another system should receive signed HTTP POST requests for issue lifecycle events | You can use several at once, and none of them replace the dashboard as the source of truth. ## Suggested order Follow the [workspace sequence](/platform/overview#recommended-workspace-sequence) on Platform overview after ready traces exist. Two connection-specific caveats: * The Lemma support channel is a Slack Connect channel with the Lemma team, not a product alert path. See [Slack](/connections/slack). * Lemma MCP and external MCP servers are opposite directions. See [Lemma MCP server](/connections/mcp). Organization owners and admins connect Slack and Linear at the workspace level. Project members can bind the project’s Slack channel and Linear defaults on top of that connection, though setting defaults still requires an owner or admin. API keys and webhook secrets are shown exactly once at creation. ## Related pages The 10-step setup sequence and scope table. Issue alerts and Issue Briefs in your workspace. Create, link, and unlink Linear tickets from issues. Let a coding agent query traces and issues, or connect an external server into Lemma. Signed HTTP POST requests for issue lifecycle events. # Slack Source: https://docs.uselemma.ai/connections/slack Issue alerts and Issue Briefs in your Slack workspace, plus the Lemma support channel Lemma has two separate Slack setups that don’t share configuration: a product integration that delivers alerts into your workspace, and a support channel that connects you directly to the Lemma team. Open a project’s **Settings** in the [Lemma dashboard](https://platform.uselemma.ai) and find **Slack** and **Lemma support channel** to configure either one. ## Set up issue alerts in your workspace This delivers issue alerts, and optionally Issue Briefs, into your Slack workspace. Have an organization owner or admin connect the Slack workspace through OAuth. This connection applies to the whole organization, not one project. Select the alert channel for this project. Channel binding is project-scoped, so a connected workspace without a channel selected won’t deliver anything. Confirm delivery works before you rely on it for real alerts. Turn on Issue Briefs and pick daily or weekly cadence. Briefs post to the same project alert channel, so they need that binding too. Daily briefs cover the prior 24 hours; weekly briefs cover the prior seven days and send Mondays in UTC. Disconnecting the workspace stops delivery for the whole organization; reconnect from the same settings to restore it. Connecting Slack never creates Linear tickets and never files issues on its own; it only notifies you about issues Lemma already detected. Owners and admins are the only ones who can connect or disconnect the workspace, but once it’s connected, any project member can set the alert channel and briefs, subject to their own Slack channel membership. ## Request the Lemma support channel The Lemma support channel is a private Slack Connect channel owned by Lemma, for working directly with the Lemma team. It doesn’t deliver product alerts and doesn’t require the integration above. In Project Settings, find **Lemma support channel**. An organization owner or admin enters the email they use for Slack and requests the channel. Accept Slack’s emailed Connect invitation. Status moves from creating to invited to connected. Availability depends on plan and environment, so treat an unavailable or failed status as accurate rather than a glitch. ## Related pages The lifecycle Slack alerts and briefs report on. In-Slack investigations use the customer integration, not support Connect. The same issue lifecycle events outside Slack. When to use Slack, Linear, MCP, or webhooks. # Webhooks Source: https://docs.uselemma.ai/connections/webhooks Signed HTTP callbacks for issue lifecycle events Project webhooks send a signed HTTP POST request whenever an issue changes state. They are separate from both [Slack](/connections/slack) alerts and [Lemma MCP](/connections/mcp). Open the project’s **Settings → Webhooks** in the [Lemma dashboard](https://platform.uselemma.ai) to configure one. Lemma doesn’t emit `incident.*` events. Customer alerting follows issues instead. If you find incident-named events in older samples or docs, they’re stale. ## Webhook events | Event | When it fires | | - | - | | `issue.created` | A new issue was created | | `issue.resolved` | The issue was marked resolved | | `issue.dismissed` | The issue was dismissed | | `issue.reopened` | A resolved issue reopened from matching evidence. Dismissed issues never reopen this way. | ## Set up an endpoint Open the project’s **Settings → Webhooks**, add an HTTPS endpoint, and select the issue events you want. Copy the signing secret when Lemma shows it. It won’t be shown again, and regenerating it replaces the old value, so store it in your receiver’s secret manager immediately. Every request includes an `X-Lemma-Signature` header containing `sha256=` and an `X-Lemma-Timestamp` header with the Unix delivery time in seconds. Verify the signature against the raw request body, then check the timestamp is recent, before you process anything. A 5-minute tolerance (the convention Stripe and GitHub also use) absorbs normal delivery and retry delay while still rejecting a replayed request. The signature is a Hash-based Message Authentication Code (HMAC) with SHA-256 over the exact raw body. ```typescript theme={null} import { createHmac, timingSafeEqual } from "crypto"; const TOLERANCE_SECONDS = 5 * 60; function verifyLemmaSignature( rawBody: string, secret: string, signature: string, timestamp: string, ): boolean { const expected = "sha256=" + createHmac("sha256", secret).update(rawBody).digest("hex"); const a = Buffer.from(expected); const b = Buffer.from(signature); const signatureValid = a.length === b.length && timingSafeEqual(a, b); const ageSeconds = Math.abs(Date.now() / 1000 - Number(timestamp)); const timestampFresh = Number.isFinite(ageSeconds) && ageSeconds <= TOLERANCE_SECONDS; return signatureValid && timestampFresh; } ``` ```python theme={null} import hmac, hashlib, time TOLERANCE_SECONDS = 5 * 60 def verify_lemma_signature( raw_body: bytes, secret: str, signature: str, timestamp: str ) -> bool: expected = "sha256=" + hmac.new( secret.encode(), raw_body, hashlib.sha256 ).hexdigest() signature_valid = hmac.compare_digest(expected, signature) try: age_seconds = abs(time.time() - float(timestamp)) timestamp_fresh = age_seconds <= TOLERANCE_SECONDS except ValueError: timestamp_fresh = False return signature_valid and timestamp_fresh ``` Always verify against the raw request body. Parsing and re-serializing the JSON changes whitespace and breaks the signature check. Send a test event, confirm your endpoint returns `2xx`, and leave it enabled. ## Headers and verification Every request carries three headers you’ll use to verify and route it: * **`X-Lemma-Signature`**: the `sha256=` HMAC-SHA256 of the raw body * **`X-Lemma-Timestamp`**: Unix seconds; reject requests older than your tolerance to block replays * **`X-Lemma-Event`**: the event type; route on this header ## Payload shape The envelope always includes `event`, `timestamp`, `projectId`, and an `issue` object with the issue’s ID, status, and related fields. Don’t expect an `incident` object; that shape doesn’t exist anymore. ```json theme={null} { "event": "issue.created", "timestamp": "2026-08-28T12:00:00.000Z", "projectId": "12345678-1234-1234-1234-123456789012", "issue": { "id": "iss_1234567890123", "status": "open" } } ``` ## Delivery and retries Lemma counts a `2xx` response within 10 seconds as a successful delivery and retries failures with exponential backoff. You can review delivery history, status, and latency (not signing secrets) in **Settings → Webhooks → Delivery log**. Offload slow processing to a background job so you can return `2xx` immediately. ## Security checklist * Verify `X-Lemma-Signature` and reject stale `X-Lemma-Timestamp` values (a 5-minute tolerance works well) on every request * Route on `X-Lemma-Event` * Use HTTPS endpoints only * Store the signing secret in an environment variable, never in source control * Remember this is distinct from Slack alerts and from Linear’s inbound webhooks to Lemma ## Related pages The lifecycle these events report. Human-readable issue alerts in Slack instead of HTTP. When to use webhooks versus Slack, Linear, or MCP. # Production monitoring for AI agents Source: https://docs.uselemma.ai/getting-started/introduction Find, investigate, and fix failures in production AI agents Lemma is an observability and evaluation platform for AI agents. It records what each agent did in production, finds recurring problems, and gives you the evidence to investigate. Project dashboard showing to-do items, headline stats, and resolved-versus-dismissed and traces-per-month charts Traditional monitoring catches explicit errors. It misses agents that return incorrect answers, forget earlier messages, or choose the wrong tool. How Lemma works: 1. **Instrument** the production agent so each execution becomes a ready, inspectable [trace](/tracing/instrumentation/concepts#trace). 2. **Analyze** completed traces for behavior that materially hurt the agent’s task. 3. **Group** repeated evidence into an [issue](/platform/concepts#issue). 4. **Investigate** in the dashboard, [Inspect](/platform/inspect), Slack, Linear, or a coding agent. See [Turning issues into fixes](/guides/turning-issues-into-fixes). See [Product boundaries](/platform/concepts#product-boundaries) for what Lemma evaluates and which surfaces are not shipped. ## Start sending traces Choose how you’ll send agent data to Lemma. Send your first complete agent trace to Lemma. Trace agents built with OpenAI Agents, Vercel AI SDK, LangChain, LangGraph, or Mastra. Learn how Lemma defines traces, spans, generations, and threads. What to do after the first ready trace arrives. ## Investigate and fix failures Use Lemma to review detected failures, receive alerts, and investigate from your editor. Triage a production issue, inspect the evidence, ship a change, and confirm it does not recur. Review recurring failures that Lemma detects across your agent traces. Ask questions about the trace, issue, or page you are on. Receive issue alerts and optional Issue Briefs in Slack. Give your coding agent the traces and issues it needs to investigate a failure. # Building high-quality traces Source: https://docs.uselemma.ai/guides/building-high-quality-traces The ideal instrumentation path, with bad → better → best examples for structure, timing, nesting, generations, and tools A high-quality Lemma trace is more than data arriving. It is a complete, nested record of one agent execution: every span has real timing, every successful span has input and output, generations carry the full message list sent to the model, and tools show what they were asked and what they returned. This page is the ideal instrumentation path. For the exact ingest fields, validation rules, and client-vs-normalization mapping, see the [trace contract](/reference/trace-contract). Use [Common issues](/tracing/troubleshooting/common-issues) when something is missing or malformed. **One agent execution = one trace.** Child LLM calls, tool calls, retrieval, and app logic belong inside that root, not as separate top-level traces. ## The ideal trace
trace root support-agent 2.41s
input:  "Where is my order #1843?"
output: "It ships tomorrow and arrives Friday."
span retrieve-context 412ms
input:  
output: 
tool search\_docs 183ms
args:   
result: 
tool lookup\_order 96ms
args:   
result: 
generation answer 1.87s
model:    gpt-4o
messages: \[system, prior turns, user, tool results]
output:   "It ships tomorrow and arrives Friday."
What “ideal” means on each node: * **Root**: current user input, final answer (or error), measured duration * **Spans / tools**: real input and output, nested under the step that caused them, measured start → end * **Generations**: model name plus the **full message list** sent to that call (system, prior turns, tool results, current user), not only the latest user string ## Quality checklist | Rule | Why it matters | | - | - | | One root per agent execution | Otherwise one run scatters across unrelated traces | | Root has input and final output (or error) | You can read what the user asked and whether the run succeeded | | Correct typed children | Generations, tools, and generic spans render and filter correctly | | Every span has a real start and end | Duration, latency, and timelines stay trustworthy | | Every successful span has input and output | Opaque spans hide what happened; failures keep input plus error | | Every failure is recorded on the span where it occurred | You can see which operation failed and what input caused it | | Nest under the work that caused the child | Flat trees hide retrieval, retries, and sub-steps | | Generations record the full message list for that call | Prior turns and tool results are part of what the model saw | | Tools record arguments and result (or error) | Tool behavior becomes inspectable | Record complete inputs and outputs only when safe. Redact secrets, credentials, and sensitive user data before tracing them. ## Structure and nesting **Bad**: each call is its own top-level trace: ```text theme={null} draft-reply ← separate trace search_docs ← separate trace final-answer ← separate trace ``` **Better**: one root, but everything is flat under it: ```text theme={null} support-agent ├── plan ├── embed-query ├── search_docs ├── rerank └── final-answer ``` **Best**: one root, children nested under the step that performed them: ```text theme={null} support-agent ├── plan │ └── embed-query ├── retrieve │ ├── search_docs │ └── rerank └── final-answer ``` Wrap the run with `lemma.trace()` and record children from the trace or span handle that owns them. See [Agent traces](/tracing/instrumentation/traces) and [Spans](/tracing/instrumentation/spans). ## Input / output **Bad**: named span with nothing useful: ```text theme={null} retrieve-context ← no input, no output ``` **Better**: partial payload: ```text theme={null} retrieve-context ← input only (or output only) ``` **Best**: every successful span carries both sides: ```text theme={null} retrieve-context ← input + output ``` Failures should still keep the input and record the error instead of an output. Opaque spans hide what happened; input and output make the tree inspectable. ## Measure duration **Bad**: no duration at all: ```text theme={null} retrieve-context ← duration missing ``` **Better**: duration inferred from the parent: ```text theme={null} retrieve-context ← duration inferred ``` **Best**: measured from real start → end: ```text theme={null} retrieve-context ← duration from start → end ``` Prefer live handles (`startSpan` / `start_span`, and the matching tool and generation helpers) so `.end(...)` establishes duration after the work completes. Pass `durationMs` / `duration_ms` only when you already measured the operation externally. Lemma can infer missing child durations from a parent, but measured bounds are better. ## Record failures Record a failure on the exact span, generation, or tool where it happened. Keep the input, pass the error instead of an output, and end the span so timing remains accurate. **Bad**: the exception is swallowed and the failed operation is missing: ```text theme={null} support-agent ← lookup_order failed, but no failed span ``` **Better**: the root fails, but the source is unclear: ```text theme={null} support-agent ← error: "Order service unavailable" └── lookup_order ← no error recorded here ``` **Best**: the child records its input, error, and duration; the root also fails when the execution cannot recover: ```text theme={null} support-agent ← error: "Order service unavailable" └── lookup_order ← input + error + duration ``` With a live handle, call `tool.end({ error })` / `tool.end(error=error)` and rethrow when the whole execution should fail. Callback traces automatically record uncaught exceptions on the root. If you catch an error at the root boundary, call `trace.fail(error)`. If the agent recovers, keep the child marked as failed and let the root complete successfully. See [Record failures in the trace contract](/reference/trace-contract#record-failures) for the exact recording rules and SDK examples. ## Generations For each LLM call, record the **full ordered message list sent to that call**: system prompt, prior conversation turns, tool results, and the current user message, not only the latest user string. That list is what the model saw; the root trace input is only the current user turn. **Bad**: little more than a name: ```text theme={null} answer ← generation (name only) ``` **Better**: model and latest user string only: ```text theme={null} answer ← model + current message + output ``` **Best**: model, complete message history for this call, and the response: ```text theme={null} answer ← model + full messages + output + duration ``` ```json theme={null} [ { "role": "system", "content": "You are a support agent." }, { "role": "user", "content": "Hi, I ordered last week" }, { "role": "assistant", "content": "Happy to help — what's the order id?" }, { "role": "user", "content": "Where is my order #1843?" }, { "role": "tool", "content": "{\"status\":\"shipped\",\"eta\":\"Fri\"}" } ] ``` In the SDK, pass that full list as both `input` and `llmInputMessages` / `llm_input_messages`. See [Generations](/tracing/instrumentation/generations) and the [trace contract](/reference/trace-contract#generations-and-message-history). ## Tool calls **Bad**: tools ran but were never recorded: ```text theme={null} support-agent ├── draft-reply └── final-answer ← search_docs ran, but no tool span ``` **Better**: tool span exists, but opaque: ```text theme={null} search_docs ← tool (name only) ``` **Best**: typed tool under the correct parent, with args and result: ```text theme={null} retrieve-context └── search_docs ← tool (args, result, duration) ``` Record each completed tool with `trace.recordTool({ name, input, output })` / `trace.record_tool(name=..., input=..., output=...)`, or use live tool handles when you want measured duration. See [Tool calls](/tracing/instrumentation/tool-calls). ## Happy-path example Use live handles so start → end establishes timing. Put the **current user message** on the root, and the **full model message list** on the generation. ```typescript theme={null} await lemma.trace( { name: "support-agent", input: userMessage, threadId, userId, }, async (trace) => { const retrieve = trace.startSpan({ name: "retrieve-context", input: { query: userMessage }, }); const search = retrieve.startTool({ name: "search_docs", input: { query: userMessage }, }); const docs = await searchDocs(userMessage); search.end({ output: docs }); retrieve.end({ output: { count: docs.length } }); const messages = [ { role: "system", content: "You are a support agent." }, ...priorMessages, { role: "user", content: userMessage }, { role: "tool", content: JSON.stringify(docs), }, ]; const generation = trace.startGeneration({ name: "answer", model: "gpt-4o", input: messages, llmInputMessages: messages, }); try { const response = await callModel(messages); generation.end({ output: response.text }); return response.text; } catch (error) { generation.end({ error }); throw error; } }, ); ``` ```python theme={null} def run(trace): retrieve = trace.start_span( name="retrieve-context", input={"query": user_message}, ) search = trace.start_tool( name="search_docs", input={"query": user_message}, parent_id=retrieve.id, ) docs = search_docs(user_message) search.end(output=docs) retrieve.end(output={"count": len(docs)}) messages = [ {"role": "system", "content": "You are a support agent."}, *prior_messages, {"role": "user", "content": user_message}, {"role": "tool", "content": json.dumps(docs)}, ] generation = trace.start_generation( name="answer", model="gpt-4o", input=messages, llm_input_messages=messages, ) try: response = call_model(messages) generation.end(output=response.text) return response.text except Exception as error: generation.end(error=error) raise lemma.trace( "support-agent", run, input=user_message, thread_id=thread_id, user_id=user_id, ) ``` ## Next steps The exact shape and fields Lemma reads. Build a complete trace one piece at a time. Capture model, full messages, and completion. Diagnose missing spans, flat trees, and blank I/O. # Runnable examples Source: https://docs.uselemma.ai/guides/examples The same Lemma docs chat agent, instrumented once per first-party stack The [examples directory](https://github.com/uselemma/lemma/tree/main/examples) is a copy-paste guide: every folder is the same chat agent, and the only thing that changes is how Lemma tracing is wired. The agent answers questions about Lemma by calling two tools against the live docs (`https://docs.uselemma.ai/llms.txt` and each page’s `.md` URL). Run a folder, ask a question, then open [Traces](/platform/traces) and confirm one root with nested tools and generations. Package-level snippets in `packages/ts/tracing/examples` and `packages/py/tracing/examples` are SDK API samples. Use this directory when you want a complete agent. ## Pick a stack Pass the framework adapter (callback, processor, exporter, or telemetry helper). Use `lemma.trace()` only when you are not on one of those frameworks. ### TypeScript | Stack | Directory | | - | - | | No framework | [`examples/direct-sdk`](https://github.com/uselemma/lemma/tree/main/examples/direct-sdk) | | Vercel AI SDK | [`examples/vercel-ai`](https://github.com/uselemma/lemma/tree/main/examples/vercel-ai) | | OpenAI Agents SDK | [`examples/openai-agents`](https://github.com/uselemma/lemma/tree/main/examples/openai-agents) | | LangChain | [`examples/langchain`](https://github.com/uselemma/lemma/tree/main/examples/langchain) | | LangGraph | [`examples/langgraph`](https://github.com/uselemma/lemma/tree/main/examples/langgraph) | | Mastra | [`examples/mastra`](https://github.com/uselemma/lemma/tree/main/examples/mastra) | ### Python | Stack | Directory | | - | - | | No framework | [`examples/python/direct-sdk`](https://github.com/uselemma/lemma/tree/main/examples/python/direct-sdk) | | OpenAI Agents SDK | [`examples/python/openai-agents`](https://github.com/uselemma/lemma/tree/main/examples/python/openai-agents) | | LangChain | [`examples/python/langchain`](https://github.com/uselemma/lemma/tree/main/examples/python/langchain) | | LangGraph | [`examples/python/langgraph`](https://github.com/uselemma/lemma/tree/main/examples/python/langgraph) | Vercel AI SDK and Mastra are TypeScript-only. ## Run an example Copy [`examples/.env.example`](https://github.com/uselemma/lemma/blob/main/examples/.env.example) to `examples/.env` and set `LEMMA_API_KEY`, `LEMMA_PROJECT_ID`, and `OPENAI_API_KEY`. TypeScript, from the repo root after `pnpm install`: ```bash theme={null} pnpm --filter @lemma/example-direct-sdk start "How do I instrument a Vercel AI agent?" ``` Python, after `uv sync`: ```bash theme={null} uv run --project examples/python/direct-sdk python examples/python/direct-sdk/main.py \ "How do I instrument a Python agent with Lemma?" ``` Omit the question argument to open a REPL. Each turn is its own root trace; turns in the same session share `threadId` / `thread_id`. See [Instrumenting multi-turn agents](/guides/instrumenting-multi-turn-agents). ## What a ready trace looks like ```text theme={null} lemma-docs-agent <- root (current question, final answer, thread id) ├── list_docs <- tool ├── read_doc <- tool (may repeat) └── answer <- generation(s) ``` If that shape is missing, enable `LEMMA_DEBUG=1` and follow [Common issues](/tracing/troubleshooting/common-issues). # Instrumenting multi-turn agents Source: https://docs.uselemma.ai/guides/instrumenting-multi-turn-agents Give each conversation turn its own trace and link the turns with a shared thread id A multi-turn agent has a back-and-forth with one user: the user asks something, the agent answers, the user follows up, and the agent answers again with the earlier turns in mind. The instinct is to record the whole conversation as one long trace. Don’t. Each turn is a separate agent execution, so each turn is its own trace, and the turns are linked by a shared `threadId` / `thread_id`. This page explains how to model that. For the exact fields and normalization, see the [trace contract](/reference/trace-contract). For the ideal per-trace shape, see [Building high-quality traces](/guides/building-high-quality-traces). **One agent execution = one trace still holds.** A conversation turn is one execution, so it is one trace. The `threadId` groups the turns; it does not merge them into a single trace. ## The core rule When one user has a conversation with your agent, each turn (one user message and the agent’s response) is a complete execution. Wrap each turn in its own `lemma.trace()` and give every turn in the same conversation the same thread id. Lemma groups traces that share a `threadId` / `thread_id` into one conversation. ```text theme={null} thread: conversation-42 ├── trace support-agent turn 1 input: "Where is my order #1843?" │ └── answer generation (system + current user) └── trace support-agent turn 2 input: "Can you change the shipping address?" ├── lookup_order tool └── answer generation (system + prior turns + tool result + current user) ``` Two turns, two traces, one thread. Everything a single turn does: LLM calls, tools, retrieval, app logic: still nests inside that turn’s trace, exactly as it would for a one-shot agent. ## What goes where The split that trips people up is between the **root input** and the **generation’s message history**. * **Root `input`** is the **current user turn** only: the single message that started this execution. * **The generation** carries the **full ordered message list sent to that model call**: system prompt, prior conversation turns, tool results, and the current user message. That list is what the model actually saw. Put the full history on the generation, not on the root. The root stays a clean record of “what did the user ask this turn”; the generation stays a faithful record of “what did the model see for this call.” Pass that full list as both `input` and `llmInputMessages` / `llm_input_messages`. See [Generations and message history](/reference/trace-contract#generations-and-message-history) and [Generations](/tracing/instrumentation/generations). ## Attach a user id Set `userId` / `user_id` on each turn’s trace so Lemma can slice a conversation, and every conversation, by the person on the other side. Set it on every turn in the thread, since each turn is a separate trace. | Context | SDK option | Client field | | - | - | - | | Thread / conversation | `threadId` / `thread_id` | `trace.thread_id` | | User | `userId` / `user_id` | `trace.user_id` | See [Threads & context](/tracing/instrumentation/context) for the full reference. ## Multi-process and async turns Turns do not always run in the same process. A request handler can answer turn 1 while a queue worker answers turn 2, or turns can run asynchronously. The rule is unchanged: start a new top-level trace per turn. Carry the thread id and user id across the boundary so the new process can reattach the turn to the conversation. Persist the thread id wherever you persist the conversation (session store, database row, queue message), then read it back when the next turn runs. This section covers successive user turns. If one turn spans a host and a sandbox, keep a single root. See [Cross-process turns](/tracing/instrumentation/cross-process-turns). ```typescript theme={null} // Producer: hand the thread id to the job that will answer the turn. await queue.add("turn", { threadId: conversationId, userId: user.id, message: userMessage, }); // Worker: start a fresh top-level trace for this turn, reusing the thread id. await lemma.trace( { name: "support-agent", input: job.message, threadId: job.threadId, userId: job.userId, }, async (trace) => runTurn(job.message, trace), ); ``` ```python theme={null} # Producer: hand the thread id to the job that will answer the turn. queue.add("turn", { "thread_id": conversation_id, "user_id": user.id, "message": user_message, }) # Worker: start a fresh top-level trace for this turn, reusing the thread id. def run(trace): return run_turn(job["message"], trace) lemma.trace( "support-agent", run, input=job["message"], thread_id=job["thread_id"], user_id=job["user_id"], ) ``` Do not reuse one trace id across processes to “continue” a conversation. Repeated deliveries for the same `trace.id` are append-only, so a second turn under the same id would graft onto the first turn’s tree instead of becoming its own turn. Use a new trace per turn and the same `threadId`. ## Record one turn on a host and sandbox A host plus a sandbox such as E2B is still one user turn, so it is one trace: ```text theme={null} thread: conversation-42 └── trace agent-turn turn 2 input: "Can you change the shipping address?" ├── e2b-sandbox span (host) │ ├── search_docs tool (sandbox journal) │ └── answer generation └── (host output) ``` See [Cross-process turns](/tracing/instrumentation/cross-process-turns) for `startTurn` / `attachTurn` / `apply`. ## Example: two turns of one thread Both turns share `threadId` / `thread_id` and `userId` / `user_id`. Turn 1 is a plain answer. Turn 2 calls a tool and sends the growing message history (system prompt, the turn 1 exchange, the tool result, and the current user message) to the model. ```typescript theme={null} const threadId = conversationId; const userId = user.id; // Turn 1 await lemma.trace( { name: "support-agent", input: "Where is my order #1843?", threadId, userId, }, async (trace) => { const messages = [ { role: "system", content: "You are a support agent." }, { role: "user", content: "Where is my order #1843?" }, ]; const response = await callModel(messages); trace.recordGeneration({ name: "answer", model: "gpt-4o", input: messages, llmInputMessages: messages, output: response.text, }); return response.text; }, ); // Turn 2 — new trace, same thread await lemma.trace( { name: "support-agent", input: "Can you change the shipping address?", threadId, userId, }, async (trace) => { const lookup = trace.startTool({ name: "lookup_order", input: { orderId: "1843" }, }); const order = await lookupOrder("1843"); lookup.end({ output: order }); const messages = [ { role: "system", content: "You are a support agent." }, { role: "user", content: "Where is my order #1843?" }, { role: "assistant", content: "It ships tomorrow and arrives Friday." }, { role: "user", content: "Can you change the shipping address?" }, { role: "tool", content: JSON.stringify(order) }, ]; const generation = trace.startGeneration({ name: "answer", model: "gpt-4o", input: messages, llmInputMessages: messages, }); const response = await callModel(messages); generation.end({ output: response.text }); return response.text; }, ); ``` ```python theme={null} thread_id = conversation_id user_id = user.id # Turn 1 def turn_one(trace): messages = [ {"role": "system", "content": "You are a support agent."}, {"role": "user", "content": "Where is my order #1843?"}, ] response = call_model(messages) trace.record_generation( name="answer", model="gpt-4o", input=messages, llm_input_messages=messages, output=response.text, ) return response.text lemma.trace( "support-agent", turn_one, input="Where is my order #1843?", thread_id=thread_id, user_id=user_id, ) # Turn 2 — new trace, same thread def turn_two(trace): lookup = trace.start_tool( name="lookup_order", input={"order_id": "1843"}, ) order = lookup_order("1843") lookup.end(output=order) messages = [ {"role": "system", "content": "You are a support agent."}, {"role": "user", "content": "Where is my order #1843?"}, {"role": "assistant", "content": "It ships tomorrow and arrives Friday."}, {"role": "user", "content": "Can you change the shipping address?"}, {"role": "tool", "content": json.dumps(order)}, ] generation = trace.start_generation( name="answer", model="gpt-4o", input=messages, llm_input_messages=messages, ) response = call_model(messages) generation.end(output=response.text) return response.text lemma.trace( "support-agent", turn_two, input="Can you change the shipping address?", thread_id=thread_id, user_id=user_id, ) ``` ## What good looks like A well-instrumented multi-turn agent records: * One trace per turn, each wrapped in its own `lemma.trace()` (or `startTurn` / `start_turn` when the turn spans a host and a sandbox). * The same `threadId` / `thread_id` on every turn of the conversation, so Lemma groups them. * `userId` / `user_id` set on every turn for per-user slicing. * Root `input` is the current user message only; the final answer (or error) is the root output. * Each turn’s generation carries the full message history for that call, including prior turns and tool results, as `input` and `llmInputMessages` / `llm_input_messages`. * The thread id (and user id) travels with the work across queues, workers, and async boundaries. * A host and sandbox that are one user turn share one root via `startTurn` / `attachTurn` / `apply`. ## Multi-turn pitfalls These patterns break conversation grouping or hide what the model saw: * **Cramming every turn into one trace.** A conversation is not one execution. One giant trace loses per-turn timing, inputs, and outputs, and grows unbounded. * **Losing the thread id across processes.** If a worker or async turn starts a trace without the conversation’s `threadId`, that turn detaches from the conversation. * **Host and sandbox as two traces.** If one user turn spans processes, use `startTurn` / `attachTurn` / `apply`. See [Cross-process turns](/tracing/instrumentation/cross-process-turns). * **Putting the full history only on the root.** The root `input` is the current turn. Prior turns and tool results belong on the generation’s message list: otherwise you can’t see what the model actually received. * **Reusing one trace id to “continue” a conversation.** Ingest is append-only per `trace.id`; reuse grafts turns together instead of grouping separate traces by thread. * **Incremental-ingesting one turn.** Send each turn as one complete trace when it finishes; do not stream sparse partial deliveries under the same `trace.id` expecting later calls to patch the root or re-run analysis. Retries of the same complete payload are safe. ## Related pages Group conversations and attach users. The exact shape and fields Lemma reads. Bad → better → best examples for the ideal instrumentation path. Capture model, full messages, and completion. `startTurn` / `attachTurn` / `apply` when one turn runs on a host and a sandbox. # From Langfuse to Lemma Source: https://docs.uselemma.ai/guides/migrations/langfuse Keep your existing Langfuse instrumentation and add Lemma SDK tracing alongside it to produce the Lemma trace contract If you already trace with Langfuse, you do not have to tear anything out to adopt Lemma. The Lemma SDK is **additive**: it runs alongside your existing observability and delivers its own trace shape to Lemma. What it does *not* do is read your Langfuse observations and turn them into Lemma traces for you. Adding the Lemma SDK next to Langfuse is fully supported. Langfuse instrumentation **on its own** is not sufficient for Lemma, because it does not produce the [trace contract](/reference/trace-contract) Lemma reads. Do not route Lemma work through Langfuse. ## Why existing instrumentation is not enough Lemma is an opinionated sink. It reads a specific trace shape to power input/output display, model visibility, timing, tool visibility, threads, and automated issue detection. See the [trace contract](/reference/trace-contract) for how Lemma reads your traces. Langfuse is a flexible observability layer. It will happily record traces, observations, and timing, but it does not guarantee the things Lemma depends on: * **One root trace per agent execution** with the current user input and the final output or error. * **Typed children**: generations, tools, and generic spans, not undifferentiated observations. * **Per-generation message history**: the full ordered message list sent to each LLM call. Because those semantics are not guaranteed by your existing setup, the fix is to emit the Lemma trace contract explicitly with the Lemma SDK. Keep Langfuse for whatever you already use it for; add Lemma for the Lemma product experience. ## Concept mapping The mental model transfers cleanly. What changes is that Lemma needs its children **typed** and its root **complete**. For the canonical field semantics and the client-vs-normalization mapping, see the [trace contract](/reference/trace-contract). | Langfuse concept | Lemma primitive | What Lemma needs | | - | - | - | | Langfuse trace | Lemma [trace record](/reference/trace-contract#trace-record) (root) | One agent execution = one trace, with `input` and final `output` or `error` | | Langfuse observation (generic) | Lemma [child span](/reference/trace-contract#child-spans) | A typed entry in `spans[]` | | Langfuse generation | Lemma [generation](/tracing/instrumentation/generations) (`type: "generation"`) | `model` plus the full per-call message history | | Langfuse span wrapping a tool | Lemma [tool](/tracing/instrumentation/tool-calls) (`type: "tool"`) | Tool `name`, arguments, and result or error | | Langfuse span for retrieval, ranking, or app logic | Lemma [span](/tracing/instrumentation/spans) (`type: "span"`) | `input` and `output` (or `error`) | | Parent / child observation nesting | `parent_id` nesting | Children under the step that caused them | | Observation start / end timestamps | `started_at` / `ended_at` / `duration_ms` | Real measured bounds | | Session id | [Thread](/tracing/instrumentation/context) (`threadId` / `thread_id`) | Groups multi-turn conversation traces | | User id | [User](/tracing/instrumentation/context) (`userId` / `user_id`) | Per-user slicing | `type` is a discriminator. Setting a `model` on an untyped span does not make it a generation: you must set `type: "generation"`. The SDK typed helpers (`recordGeneration` / `recordTool` / `recordSpan`) do this for you. ## What transfers and what does not **Transfers**: the structural intuition you already have: * Nesting: parent/child observation trees map directly onto Lemma’s `parent_id` nesting. * Timing: start/end timestamps map onto Lemma’s `started_at` / `ended_at` / `duration_ms`. * One logical run: the idea that a single agent execution is one unit maps onto one Lemma trace. **Does not transfer automatically**: Lemma needs these produced explicitly: * A dedicated **Lemma root trace** per execution (`lemma.trace(...)`). * **Typed** generations and tools, rather than generic observations. * **Root input and output** (or error) on the trace record. * **Per-generation message history**: the full ordered message list, not only the latest user string. For the bad → better → best version of each of these, see [Building high-quality traces](/guides/building-high-quality-traces). ## Run Lemma side by side Leave your Langfuse instrumentation exactly where it is. Wrap the same agent execution in `lemma.trace()` and record the LLM calls and tools you already know about as typed Lemma children. This produces a conforming Lemma trace without disturbing your existing pipeline. Install and configure the SDK first. See [Quickstart](/tracing/instrumentation/setup). ```typescript theme={null} import { Lemma } from "@uselemma/tracing"; const lemma = new Lemma(); // Your existing Langfuse instrumentation stays in place. const answer = await lemma.trace( { name: "support-agent", input: userMessage, threadId, userId, }, async (trace) => { const docs = await searchDocs(userMessage); trace.recordTool({ name: "search_docs", input: { query: userMessage }, output: docs, }); const messages = [ { role: "system", content: "You are a support agent." }, ...priorMessages, { role: "user", content: userMessage }, { role: "tool", content: JSON.stringify(docs) }, ]; const response = await callModel(messages); trace.recordGeneration({ name: "answer", model: "gpt-4o", input: messages, output: response.text, llmInputMessages: messages, }); return response.text; }, ); ``` ```python theme={null} import json from uselemma_tracing import Lemma lemma = Lemma() # Your existing Langfuse instrumentation stays in place. def run(trace): docs = search_docs(user_message) trace.record_tool( name="search_docs", input={"query": user_message}, output=docs, ) messages = [ {"role": "system", "content": "You are a support agent."}, *prior_messages, {"role": "user", "content": user_message}, {"role": "tool", "content": json.dumps(docs)}, ] response = call_model(messages) trace.record_generation( name="answer", model="gpt-4o", input=messages, output=response.text, llm_input_messages=messages, ) return response.text answer = lemma.trace( "support-agent", run, input=user_message, thread_id=thread_id, user_id=user_id, ) ``` Pass the **full ordered message list** for each call as both `input` and `llmInputMessages` / `llm_input_messages`. The root trace `input` is only the current user turn. See [Generations](/tracing/instrumentation/generations) and [Tool calls](/tracing/instrumentation/tool-calls) for live handles, errors, and nesting under a measured parent span. If your run is coordinated across streaming callbacks or helpers rather than one function, use a TypeScript trace handle and end it from the terminal callback. See [Agent traces](/tracing/instrumentation/traces). ## Migration pitfalls Record complete inputs and outputs only when safe. Redact secrets, credentials, and sensitive user data before tracing them. | Pitfall | Fix | | - | - | | Assuming Langfuse observations already satisfy Lemma | They usually do not carry the Lemma contract. Emit it explicitly with the Lemma SDK. | | Routing Lemma work through Langfuse | Do not. Add Lemma SDK tracing directly alongside Langfuse; keep Langfuse only for what still needs it. | | No Lemma root trace | Wrap each agent execution in `lemma.trace()` so children have one root to attach to. | | Untyped children | Record LLM calls with `recordGeneration` and tools with `recordTool` so they render and filter as generations and tools. | | Missing message history | Put the full ordered message list on each generation’s `input` and `llmInputMessages` / `llm_input_messages`, not only the latest user string. | ## Related pages Install the SDK and send your first complete agent trace. The exact shape and fields Lemma reads. Bad → better → best examples for the ideal path. Add Lemma alongside an existing OTel pipeline. # From OpenTelemetry to Lemma Source: https://docs.uselemma.ai/guides/migrations/opentelemetry Keep your existing OpenTelemetry instrumentation and add Lemma SDK tracing alongside it to produce the Lemma trace contract If you already trace with OpenTelemetry, you do not have to tear anything out to adopt Lemma. The Lemma SDK is **additive**: it runs alongside your existing observability and delivers its own trace shape to Lemma. What it does *not* do is read your OTel spans and turn them into Lemma traces for you. Adding the Lemma SDK next to OpenTelemetry is fully supported. OTel instrumentation **on its own** is not sufficient for Lemma, because it does not produce the [trace contract](/reference/trace-contract) Lemma reads. ## Why existing instrumentation is not enough Lemma is an opinionated sink. It reads a specific trace shape to power input/output display, model visibility, timing, tool visibility, threads, and automated issue detection. See the [trace contract](/reference/trace-contract) for how Lemma reads your traces. Generic OpenTelemetry is a flexible observability layer. It will happily record spans, attributes, and timing, but it does not guarantee the things Lemma depends on: * **One root trace per agent execution** with the current user input and the final output or error. * **Typed children**: generations, tools, and generic spans, not undifferentiated spans. * **Per-generation message history**: the full ordered message list sent to each LLM call. Because those semantics are not guaranteed by your existing setup, the fix is to emit the Lemma trace contract explicitly with the Lemma SDK. Keep OTel for whatever you already use it for; add Lemma for the Lemma product experience. ## Concept mapping The mental model transfers cleanly. What changes is that Lemma needs its children **typed** and its root **complete**. For the canonical field semantics and the client-vs-normalization mapping, see the [trace contract](/reference/trace-contract). | OpenTelemetry concept | Lemma primitive | What Lemma needs | | - | - | - | | OTel root span | Lemma [trace record](/reference/trace-contract#trace-record) (root) | One agent execution = one trace, with `input` and final `output` or `error` | | Generic child span | Lemma [child span](/reference/trace-contract#child-spans) | A typed entry in `spans[]` | | LLM span | Lemma [generation](/tracing/instrumentation/generations) (`type: "generation"`) | `model` plus the full per-call message history | | Tool span | Lemma [tool](/tracing/instrumentation/tool-calls) (`type: "tool"`) | Tool `name`, arguments, and result or error | | Retrieval, ranking, or app-logic span | Lemma [span](/tracing/instrumentation/spans) (`type: "span"`) | `input` and `output` (or `error`) | | Parent / child span nesting | `parent_id` nesting | Children under the step that caused them | | Span start / end timestamps | `started_at` / `ended_at` / `duration_ms` | Real measured bounds | `type` is a discriminator. Setting a `model` on an untyped span does not make it a generation: you must set `type: "generation"`. The SDK typed helpers (`recordGeneration` / `recordTool` / `recordSpan`) do this for you. ## What transfers and what does not **Transfers**: the structural intuition you already have: * Nesting: parent/child span trees map directly onto Lemma’s `parent_id` nesting. * Timing: start/end timestamps map onto Lemma’s `started_at` / `ended_at` / `duration_ms`. * One logical run: the idea that a single agent execution is one unit maps onto one Lemma trace. **Does not transfer automatically**: Lemma needs these produced explicitly: * A dedicated **Lemma root trace** per execution (`lemma.trace(...)`). * **Typed** generations and tools, rather than generic spans. * **Root input and output** (or error) on the trace record. * **Per-generation message history**: the full ordered message list, not only the latest user string. For the bad → better → best version of each of these, see [Building high-quality traces](/guides/building-high-quality-traces). ## Run Lemma side by side Leave your OpenTelemetry instrumentation exactly where it is. Wrap the same agent execution in `lemma.trace()` and record the LLM calls and tools you already know about as typed Lemma children. This produces a conforming Lemma trace without disturbing your existing pipeline. Install and configure the SDK first. See [Quickstart](/tracing/instrumentation/setup). ```typescript theme={null} import { Lemma } from "@uselemma/tracing"; const lemma = new Lemma(); // Your existing OpenTelemetry instrumentation stays in place. const answer = await lemma.trace( { name: "support-agent", input: userMessage, threadId, userId, }, async (trace) => { const docs = await searchDocs(userMessage); trace.recordTool({ name: "search_docs", input: { query: userMessage }, output: docs, }); const messages = [ { role: "system", content: "You are a support agent." }, ...priorMessages, { role: "user", content: userMessage }, { role: "tool", content: JSON.stringify(docs) }, ]; const response = await callModel(messages); trace.recordGeneration({ name: "answer", model: "gpt-4o", input: messages, output: response.text, llmInputMessages: messages, }); return response.text; }, ); ``` ```python theme={null} import json from uselemma_tracing import Lemma lemma = Lemma() # Your existing OpenTelemetry instrumentation stays in place. def run(trace): docs = search_docs(user_message) trace.record_tool( name="search_docs", input={"query": user_message}, output=docs, ) messages = [ {"role": "system", "content": "You are a support agent."}, *prior_messages, {"role": "user", "content": user_message}, {"role": "tool", "content": json.dumps(docs)}, ] response = call_model(messages) trace.record_generation( name="answer", model="gpt-4o", input=messages, output=response.text, llm_input_messages=messages, ) return response.text answer = lemma.trace( "support-agent", run, input=user_message, thread_id=thread_id, user_id=user_id, ) ``` Pass the **full ordered message list** for each call as both `input` and `llmInputMessages` / `llm_input_messages`. The root trace `input` is only the current user turn. See [Generations](/tracing/instrumentation/generations) and [Tool calls](/tracing/instrumentation/tool-calls) for live handles, errors, and nesting under a measured parent span. If your run is coordinated across streaming callbacks or helpers rather than one function, use a TypeScript trace handle and end it from the terminal callback. See [Agent traces](/tracing/instrumentation/traces). ## Keep your OpenTelemetry setup The Lemma SDK delivers its trace JSON over its own HTTP path; it does not register as an exporter or span processor in your OpenTelemetry pipeline. That means your existing OTel exporters can stay untouched: the two run independently and neither depends on the other. The one place the two pipelines meet today is the **Vercel AI SDK**, which is built on OpenTelemetry. There you add `vercelAI()` to the AI SDK’s telemetry `integrations` alongside any exporters you already use, and it produces the Lemma trace from AI SDK lifecycle events. See [Vercel AI SDK](/integrations/vercel-ai). For frameworks with a first-class Lemma integration, prefer the integration over hand-rolling the mapping: * [OpenAI Agents SDK](/integrations/openai-agents) * [Vercel AI SDK](/integrations/vercel-ai) * [LangChain](/integrations/langchain) * [LangGraph](/integrations/langgraph) Outside those framework integrations, treat OTel and Lemma as separate additive layers: keep OTel for your existing telemetry and use the Lemma SDK directly for the Lemma trace contract, as shown above. ## Migration pitfalls Record complete inputs and outputs only when safe. Redact secrets, credentials, and sensitive user data before tracing them. | Pitfall | Fix | | - | - | | Assuming OTel spans already satisfy Lemma | They usually do not carry the Lemma contract. Emit it explicitly with the Lemma SDK. | | No Lemma root trace | Wrap each agent execution in `lemma.trace()` so children have one root to attach to. | | Untyped children | Record LLM calls with `recordGeneration` and tools with `recordTool` so they render and filter as generations and tools. | | Missing message history | Put the full ordered message list on each generation’s `input` and `llmInputMessages` / `llm_input_messages`, not only the latest user string. | ## Related pages Install the SDK and send your first complete agent trace. The exact shape and fields Lemma reads. Bad → better → best examples for the ideal path. Framework integrations that produce the contract for you. # From Raindrop to Lemma Source: https://docs.uselemma.ai/guides/migrations/raindrop Map Raindrop’s event- and signal-centric model onto Lemma’s trace contract, and re-instrument your agent to emit conforming Lemma traces [Raindrop](https://www.raindrop.ai) is an AI-native observability platform for production agents. Its model is **event-centric**: you emit an *AI event* per interaction, group related events into a conversation, and Raindrop surfaces quality *signals*, individual failures (*stumbles*), and recurring failures (*issues*) on top of that stream. Lemma models the same reality differently. It is **trace-centric**: one agent execution is one trace, a nested tree of typed spans, and Lemma runs automated issue detection on complete, correctly typed traces. Migrating is mostly a re-instrumentation: instead of emitting an event (and optional spans) per interaction, you wrap each execution in `lemma.trace()` and record its LLM calls and tools as typed children. **One agent execution = one trace.** In Raindrop you emit one event per interaction and group by `convoId`; in Lemma you emit one trace per execution and group multi-turn conversations by `threadId`. See the [trace contract](/reference/trace-contract). ## How the two models differ Raindrop starts from the *event*, a single tracked AI interaction created with `trackAi()` (single-shot) or `begin()` / `finish()` (the interaction API), and layers tracing spans, signals, and feedback onto it. Conversations are formed by sharing a `convoId` across events. Lemma starts from the *trace*, one end-to-end execution, and requires its structure to be explicit: * **One root trace per agent execution** carrying the current user input and the final output or error. * **Typed children**: generations, tools, and generic spans, not undifferentiated records. * **Per-generation message history**: the full ordered message list sent to each LLM call. Because Lemma is an opinionated sink that reads a [specific trace shape](/reference/trace-contract), the migration is to produce that shape explicitly with the Lemma SDK. See the [trace contract](/reference/trace-contract) for how Lemma reads your traces. ## Concept mapping The structural intuition transfers cleanly. What changes is that Lemma needs its children **typed** and its root **complete**. | Raindrop concept | Closest Lemma primitive | What Lemma needs | | - | - | - | | AI event / interaction (`trackAi`, `begin` + `finish`) | Lemma [trace record](/reference/trace-contract#trace-record) (root) | One agent execution = one trace, with `input` and final `output` or `error` | | Event name (`event`) | Trace `name` | A stable agent / workflow name for grouping and filtering | | Event `input` / `output` | Trace `input` / `output` | Current user turn on the root; final answer or error on the root | | `convoId` | [Thread](/tracing/instrumentation/context) (`threadId` / `thread_id`) | Groups multi-turn conversation traces | | `userId` | [User](/tracing/instrumentation/context) (`userId` / `user_id`) | Per-user slicing | | `model` on the event | [Generation](/tracing/instrumentation/generations) `model` (`type: "generation"`) | Model lives on a typed generation span, plus the full per-call message history | | Tracing span (`withSpan`) | [Span](/tracing/instrumentation/spans) (`type: "span"`) | `input` and `output` (or `error`) | | Tool tracking (`withTool`, `trackTool`, `startToolSpan`) | [Tool](/tracing/instrumentation/tool-calls) (`type: "tool"`) | Tool `name`, arguments, and result or error | | Auto-instrumented LLM spans | [Generation](/tracing/instrumentation/generations) (`type: "generation"`) | `model` plus the full ordered message list | | Span / tool nesting | `parent_id` nesting | Children under the step that caused them | | Event `properties`, feature flags | Trace or span [`metadata`](/reference/trace-contract#trace-record) | Arbitrary JSON attached to the root or a child | | Stumbles / issues | Lemma [issue detection](/reference/trace-contract#issue-detection) | Complete, correctly typed trees improve automated issue extraction | `type` is a discriminator. Setting a `model` on an untyped span does not make it a generation: you must set `type: "generation"`. The SDK typed helpers (`recordGeneration` / `recordTool` / `recordSpan` and their Python equivalents) do this for you. For per-interaction context such as feature flags or arbitrary properties, attach it as trace or span [`metadata`](/reference/trace-contract#trace-record). Do not hand-build attribute names. Prefer native SDK props and first-class fields; use raw `metadata` only for context that has no dedicated field. See [Client fields vs ingest normalization](/reference/trace-contract#client-fields-vs-ingest-normalization). ## Migration path Install the SDK and set `LEMMA_API_KEY` and `LEMMA_PROJECT_ID`. See [Quickstart](/tracing/instrumentation/setup). ```bash theme={null} npm install @uselemma/tracing ``` ```bash theme={null} pip install uselemma-tracing ``` Replace the Raindrop `begin()` / `finish()` (or `trackAi()`) that bracketed one interaction with a single `lemma.trace()` around the whole execution. Put the current user turn on `input` and let the callback return the final answer. Emit each LLM call with `recordGeneration` / `record_generation` and each tool call with `recordTool` / `record_tool`. This is the Lemma equivalent of Raindrop’s auto-instrumented LLM spans, `withTool`, and `trackTool`. Map `convoId` to `threadId` / `thread_id` and `userId` to `userId` / `user_id` on the trace so multi-turn conversations group correctly. See [Threads & context](/tracing/instrumentation/context). Turn on [debug mode](/tracing/troubleshooting/debug-mode) (`LEMMA_DEBUG=1`) and check the `sending trace` log’s `spanCount` (`span_count` in Python) to confirm your generations and tools were recorded. Cross-check fields against the [trace contract](/reference/trace-contract). ## Re-instrumenting an interaction The example below is the Lemma equivalent of a Raindrop `begin()` → tool tracking → `finish()` flow: one trace, a typed tool, and a generation carrying the full message list. Install and configure the SDK first. See [Quickstart](/tracing/instrumentation/setup). ```typescript theme={null} import { Lemma } from "@uselemma/tracing"; const lemma = new Lemma(); // Raindrop: raindrop.begin({ event, userId, convoId, input, model }) // Lemma: one trace per execution. const answer = await lemma.trace( { name: "support-agent", // Raindrop `event` input: userMessage, // Raindrop event `input` threadId: conversationId, // Raindrop `convoId` userId: user.id, // Raindrop `userId` }, async (trace) => { const docs = await searchDocs(userMessage); trace.recordTool({ name: "search_docs", // Raindrop `withTool` / `trackTool` input: { query: userMessage }, output: docs, }); const messages = [ { role: "system", content: "You are a support agent." }, ...priorMessages, { role: "user", content: userMessage }, { role: "tool", content: JSON.stringify(docs) }, ]; const response = await callModel(messages); trace.recordGeneration({ name: "answer", model: "gpt-4o", // Raindrop event `model` input: messages, output: response.text, llmInputMessages: messages, }); return response.text; // Raindrop finish({ output }) }, ); ``` ```python theme={null} import json from uselemma_tracing import Lemma lemma = Lemma() # Raindrop: raindrop.begin(event=..., user_id=..., convo_id=..., input=...) # Lemma: one trace per execution. def run(trace): docs = search_docs(user_message) trace.record_tool( name="search_docs", # Raindrop withTool / trackTool input={"query": user_message}, output=docs, ) messages = [ {"role": "system", "content": "You are a support agent."}, *prior_messages, {"role": "user", "content": user_message}, {"role": "tool", "content": json.dumps(docs)}, ] response = call_model(messages) trace.record_generation( name="answer", model="gpt-4o", # Raindrop event model input=messages, output=response.text, llm_input_messages=messages, ) return response.text # Raindrop finish(output=...) answer = lemma.trace( "support-agent", # Raindrop event run, input=user_message, # Raindrop event input thread_id=conversation_id, # Raindrop convo_id user_id=user.id, # Raindrop user_id ) ``` Pass the **full ordered message list** for each call as both `input` and `llmInputMessages` / `llm_input_messages`. The root trace `input` is only the current user turn. For live handles that measure real duration, error recording, and nesting a tool under a measured parent span, see [Generations](/tracing/instrumentation/generations) and [Tool calls](/tracing/instrumentation/tool-calls). If your run spans streaming callbacks or helpers rather than one function, use a TypeScript trace handle and end it from the terminal callback. See [Agent traces](/tracing/instrumentation/traces). ## Migration pitfalls Record complete inputs and outputs only when safe. Redact secrets, credentials, and sensitive user data before tracing them. | Pitfall | Fix | | - | - | | Emitting one Lemma trace per LLM call, mirroring per-event tracking | Wrap the whole execution in one `lemma.trace()`; record each LLM call as a `generation` child. | | Using `convoId` semantics as the trace identity | `convoId` maps to `threadId` / `thread_id`, which groups turns: it is not the per-execution trace id. | | Putting `model` on the root instead of a generation | Model belongs on a typed `generation` span; a `model` on an untyped span does not become a generation. | | Missing message history | Put the full ordered message list on each generation’s `input` and `llmInputMessages` / `llm_input_messages`, not only the latest user string. | | Untyped children | Record LLM calls with `recordGeneration` and tools with `recordTool` so they render and filter as generations and tools. | ## Related pages Install the SDK and send your first complete agent trace. The exact shape and fields Lemma reads. Bad → better → best examples for the ideal path. Build a complete Lemma trace one piece at a time. # Querying analytics Source: https://docs.uselemma.ai/guides/query-analytics Ask a project for one aggregate over REST or MCP, with worked examples Using a coding agent? Install the Lemma analytics skill, and the agent reads the catalog, builds the query, and reports the result with its window and cost caveats, over the [Lemma MCP server](/connections/mcp) or REST: ```bash theme={null} npx skills add uselemma/lemma --skill "lemma-analytics" ``` Then ask it a question, such as “Which tools failed most in the last 7 days, and what did each model cost?” A project analytics query returns one aggregate: a count, a percentile, a sum, or an estimated cost, optionally grouped by time or by a field. You send it over REST or the Lemma [Model Context Protocol (MCP)](/connections/mcp) server. This page explains how a query is built, then shows [examples](#examples) you can copy. Every field and rule is on [Analytics query API](/reference/analytics-query). Use a query when one number or one breakdown answers the question. For a full [Analytics](/platform/analytics) panel, request a [preset view](#preset-views) instead. You need an organization API key whose scope is read or admin, plus the project id. An ingest-only key cannot call these routes. A key limited to one project works only for that project. Create the key under **API Keys** in the [Lemma dashboard](https://platform.uselemma.ai). ## Find events and fields in the catalog Every query reads one event, such as `trace.processed` or `tool.called`. The catalog lists each event with the fields you can filter and group by (`dimensions`), the fields you can aggregate (`measures`, each with its allowed `aggs`), and any `cost_measures`. It also lists `filter_ops`, `buckets`, and `limits`. ```bash theme={null} curl -s \ "https://api.uselemma.ai/projects/your_project_id/analytics/catalog" \ -H "Authorization: Bearer your_organization_api_key" ``` On MCP, call `describe_project_analytics` with `project_id`. A field that isn’t in the catalog can’t be queried. ## Send a query `POST` the query body to `/projects/{project_id}/analytics/query`: ```bash theme={null} curl -s -X POST \ "https://api.uselemma.ai/projects/your_project_id/analytics/query" \ -H "Authorization: Bearer your_organization_api_key" \ -H "Content-Type: application/json" \ -d @query.json ``` On MCP, call `query_project_analytics` with `project_id` plus the same body. MCP uses camelCase for three keys: `endExclusive`, `groupBy`, and `orderBy`. Field names such as `agent_name` stay snake\_case in both. ## Build the query body A body names an event in `from`, a window in `start` and `end_exclusive`, and at least one measure. The window is UTC, on whole seconds, with the end exclusive. Optional keys narrow or shape the result: * [Measures](/reference/analytics-query#measures): what to compute, each with an alias in `as`, and an optional per-measure `filter` * [Filters](/reference/analytics-query#filters): conditions every counted row must match * [Groups and order](/reference/analytics-query#groups-and-order): split by a time bucket or a field, then sort * [Join](/reference/analytics-query#join): add measures from one other event [Query request](/reference/analytics-query#query-request) lists every key and limit. ## Read the response A response has `rows`, one object per group keyed by your aliases, plus `truncated` and the queried `window`. When `truncated` is true, further rows exist, so narrow the filters or raise `limit`. REST and MCP return the same snake\_case shape. See [Query response](/reference/analytics-query#query-response). An invalid query returns HTTP 400 with the reason. [Query errors](/reference/analytics-query#query-errors) lists each message and its cause. ## Examples Each example below is a complete body. Save it as `query.json` and send it with the request in [Send a query](#send-a-query). ### Daily volume and latency for one agent Count one agent’s traces per day, with the 95th percentile of root duration. On `trace.processed`, root duration is `root_duration_ms`: ```json theme={null} { "from": "trace.processed", "start": "2026-09-23T00:00:00.000Z", "end_exclusive": "2026-09-30T00:00:00.000Z", "filters": [ { "field": "agent_name", "op": "eq", "value": "support_agent" } ], "group_by": [{ "bucket": "day", "as": "day" }], "measures": [ { "as": "traces", "agg": "count" }, { "as": "p95_ms", "agg": "p95", "field": "root_duration_ms" } ] } ``` Each row is one day. `p95_ms` is in milliseconds. ```json theme={null} { "day": "2026-09-23", "traces": 120, "p95_ms": 840.5 } ``` ### Tools that fail most Rank tools by failed calls. On `tool.called`, `error` is `1` when the tool span failed and `0` otherwise, so its sum is the failure count: ```json theme={null} { "from": "tool.called", "start": "2026-09-23T00:00:00.000Z", "end_exclusive": "2026-09-30T00:00:00.000Z", "group_by": [{ "field": "tool", "as": "tool" }], "measures": [ { "as": "calls", "agg": "count" }, { "as": "failures", "agg": "sum", "field": "error" } ], "order_by": [{ "measure": "failures", "direction": "desc" }], "limit": 20 } ``` Divide `failures` by `calls` for the failure rate. This row is 4.5%: ```json theme={null} { "tool": "search_docs", "calls": 400, "failures": 18 } ``` ### Estimated cost by model Estimate generation cost per model. `generation_cost` needs a `model` group with the alias `model`: ```json theme={null} { "from": "generation.completed", "start": "2026-09-23T00:00:00.000Z", "end_exclusive": "2026-09-30T00:00:00.000Z", "group_by": [{ "field": "model", "as": "model" }], "measures": [{ "as": "cost", "agg": "generation_cost" }] } ``` The row reports the estimate in `estimated_cost_usd`, not under the `cost` alias: ```json theme={null} { "model": "gpt-4o", "estimated_cost_usd": 12.4, "priced": true } ``` `priced: false` means Lemma has no list price for that model, and `estimated_cost_usd` is `null`. ### Runs with errors by agent Count each agent’s traces, and the traces where any span failed. The second measure’s `filter` limits it to traces with `error_count` above zero: ```json theme={null} { "from": "trace.processed", "start": "2026-09-23T00:00:00.000Z", "end_exclusive": "2026-09-30T00:00:00.000Z", "group_by": [{ "field": "agent_name", "as": "agent_name" }], "measures": [ { "as": "traces", "agg": "count" }, { "as": "runs_with_errors", "agg": "count", "filter": { "field": "error_count", "op": "gt", "value": 0 } } ], "order_by": [{ "measure": "runs_with_errors", "direction": "desc" }] } ``` ```json theme={null} { "agent_name": "support_agent", "traces": 120, "runs_with_errors": 9 } ``` ### Traces and tool use per agent Join tool calls to traces on `agent_name`. `uniqExact` on `tool` counts the distinct tools each agent called: ```json theme={null} { "from": "trace.processed", "start": "2026-09-23T00:00:00.000Z", "end_exclusive": "2026-09-30T00:00:00.000Z", "group_by": [{ "field": "agent_name", "as": "agent_name" }], "measures": [{ "as": "traces", "agg": "count" }], "join": { "from": "tool.called", "on": "agent_name", "measures": [ { "as": "tool_calls", "agg": "count" }, { "as": "tools_used", "agg": "uniqExact", "field": "tool" } ] } } ``` ```json theme={null} { "agent_name": "support_agent", "traces": 120, "tool_calls": 640, "tools_used": 7 } ``` ## Cost estimates `trace_cost` and `generation_cost` are estimates from recorded token counts and public list prices, not an invoice. [Cost fields](/reference/analytics-query#cost-fields) lists what each adds to a row, including `cost_lower_bound` when the estimate can be low. ## Preset views `GET /projects/{project_id}/analytics` returns one dashboard panel. Pass one `view`. MCP has one tool for each view. View names, MCP tools, parameters, and responses are on [Analytics query API](/reference/analytics-query#preset-views). ## Related pages Every event, field, rule, error, and preset view. Dashboard charts for volume, latency, errors, tools, models, and cost. Events, fields, filters, and limits for a project query. The REST operation for one aggregate. SDK fields the aggregates are computed from. # Turning issues into fixes Source: https://docs.uselemma.ai/guides/turning-issues-into-fixes Triage a production issue, inspect the evidence, ship a change, and confirm it does not recur Lemma groups repeated [trace](/platform/concepts#trace) evidence into an [issue](/platform/concepts#issue). This how-to takes that finding to a code change you can prove. Triage first. Then find the mechanism. Then change the code and verify. Do not start a patch until you can point at the supporting traces that show the defect. You need a project with [ready traces](/platform/traces) and at least one issue. [Inspect](/platform/inspect) and a coding agent on the [Lemma Model Context Protocol (MCP) server](/connections/mcp) are optional. Start from the Issues table in the Lemma dashboard. On an issue page, ask why it fired and how to fix it. Lemma’s detection is probabilistic. An issue title is a hypothesis backed by traces, not a confirmed bug. Open the evidence before you treat the title as ground truth. See [Issues](/platform/issues) and [Product boundaries](/platform/concepts#product-boundaries). ## Triage, investigate, ship, and verify Decide whether the finding is a real defect. Do not write code in this step. Start from an open issue in the Issues table, a [Slack](/connections/slack) alert, or an Issue Brief. Issues table grouped by agent, showing signals, supporting-trace counts, assignee, status, and last seen Do the following: 1. Open **Issues** in the project in the [Lemma dashboard](https://platform.uselemma.ai). 2. Filter by status, agent, assignee, or search if you are choosing what to work next. 3. Open the issue and read why Lemma flagged it: rationale, category, priority, first seen, last seen, and supporting-trace count. 4. Dismiss it, or accept it and work it. The issue is `dismissed` with a reason, or `in_progress` because you accepted it as a defect. Assignment inside Lemma does not create a Linear ticket. Create that link later only if your team files tickets. See [Linear](/connections/linear). Dismiss when the behavior is working as intended, synthetic traffic, or a false grouping. Do not dismiss a real defect because you plan to fix it. That status is `resolved` after the change ships. Start from the runs Lemma grouped, not from the issue title. You need an accepted issue with at least one supporting trace. Do the following: 1. Open more than one supporting trace from the issue, not only the newest one. 2. Read **Conversation** for what the person using the agent saw. 3. Read **Execution tree** for which generation, tool call, or [span](/tracing/instrumentation/concepts#span) produced that outcome. 4. Compare the assistant text to the tool results and span outputs in the same run. A mismatch between those two panels is the defect. Trace detail view with the conversation and linked issues on the left, and the execution tree with span timings on the right Why did this issue fire, which scenarios triggered it, and what do the supporting traces share? How should I fix the underlying defect? Export the issue as markdown (`GET /issues/{id}/markdown`, or the UI equivalent). Or connect a coding agent to the [Lemma MCP server](/connections/mcp) and pull the same evidence in the editor. Use the Lemma MCP to inspect this issue. Cite the supporting traces, name the failing span, and do not propose a patch until you can describe the mechanism. You can say: in these traces, span X did Y, and the visible result was Z. If the traces do not support the title, return to triage and dismiss with that evidence. Find the mechanism. Do not restate the issue title. Use the span-level mismatch from the previous step. Do the following: 1. Map the failing span to a function, prompt, tool wrapper, or policy in your repo. Use the span name, tool name, and timestamps on the execution tree. 2. Decide where the fix belongs: prompt or policy, tool or retrieval code, or a guard after the model responds. One defect can need more than one of those layers. 3. Write the intended change in one sentence: “When `lookup_order` returns `processing`, the assistant must not claim the order shipped.” You have a named code path and a claim you can falsify. If you cannot name the path, keep inspecting traces. Inspect can propose a fix. Treat that proposal as a draft against the traces. Do not skip the execution tree. Use your team’s development workflow. Lemma does not apply the patch. Use the one-sentence claim from the previous step, plus the exported issue or MCP evidence. Do the following: 1. Make the smallest change that addresses the mechanism in those traces. 2. Keep the original user prompt, tool arguments, and tool result from a supporting trace as the fixture you will replay. 3. If your team files tickets, [create or link a Linear ticket](/connections/linear) from the issue. Connecting Linear never files one on its own. Point the agent at the issue markdown or the Lemma MCP tools. Require it to cite the supporting traces in the pull request. Pull this Lemma issue from the MCP. Make the smallest change that addresses the failure mechanism in the supporting traces. Open a pull request that links the issue id and quotes the cited span evidence. You have a reviewable change. Its description links the Lemma issue and quotes the evidence. Set the issue to `in_progress` if it is not already. Do not resolve the issue because a pull request exists. Resolve after you verify. A passing unit test is not enough. Replay the scenario the traces recorded. Use the fixture from a supporting trace (user message, tool arguments, tool result) and a build that includes your change. Do the following: 1. Replay the same user turn against the same tool result, in staging or a local harness. 2. Confirm the new conversation no longer produces the bad assistant text. 3. Confirm the execution tree still records the tool or span you relied on. A “fix” that skips the lookup hides the defect. 4. Send that run to Lemma and open the new [ready trace](/platform/traces). 5. Confirm Lemma does not group the new run into the same issue. You have at least one new ready trace for the same scenario that does not match the original failure. If it still matches, the mechanism is not fixed. Return to naming the root cause. Resolve only after production, or a production-like replay, stops reproducing the pattern. You have a verified change, shipped or ready to ship, and the original issue. Do the following: 1. Ship through your usual release path. 2. Keep the issue `in_progress` until new production traces for that agent and scenario do not match the original failure. 3. Resolve the issue and note the fix, including a pull request link when you have one. 4. Watch the issue. Matching evidence can reopen a `resolved` issue. It never reopens a `dismissed` one. 5. If it reopens, treat that as failed verification. Inspect the new supporting traces. The issue is `resolved`, or it reopened and you are investigating again with fresh traces. Slack and [webhooks](/connections/webhooks) follow this lifecycle (`issue.resolved`, `issue.reopened`). Recurrence does not create a new Linear ticket. Dismiss and resolve are different terminal states. Dismiss means the finding is not a real problem. Resolve means the defect is fixed. Pick resolve after a verified change. ## Worked example: shipped claim on a processing order This example uses the `support-agent` from [Building high-quality traces](/guides/building-high-quality-traces). Substitute your agent name and tool names. Issue: “Agent tells the customer an order shipped while `lookup_order` still returns processing.” You open 3 of 12 supporting traces. Each customer asks “Where is my order #1843?” **Conversation** shows “It shipped yesterday and arrives Friday.” **Execution tree** shows `lookup_order` returning `{ "status": "processing" }` with no tracking number. The generation ran after that tool result. The defect is the answer ignoring the tool output, not a failed HTTP call. Add a rule: if `lookup_order.status` is not `shipped`, the assistant reports the current status and does not name a ship date or tracking number. A post-generation check can reject a “shipped” claim when the last lookup disagrees, then retry or fall back to a template. The pull request quotes `{ "status": "processing" }` next to the invented “shipped yesterday” line. Replay “Where is my order #1843?” with `lookup_order` stubbed to `processing`. The new conversation says the order is still processing. The execution tree still includes `lookup_order`. The new ready trace does not join the original issue. After that build is in production, you resolve with the pull request link. If a later trace claims shipped against a `processing` lookup, the issue reopens. Inspect that new tree. ## Related pages Status, priority, and how to open supporting traces. Conversation and execution-tree views for one run. Ask why an issue fired and how to fix it. Pull the same evidence into a coding agent. Create or link a ticket after you accept the issue. The trace shape this investigation depends on. # LangChain Source: https://docs.uselemma.ai/integrations/langchain Lemma callback tracing for LangChain chains, models, and tools Use Lemma’s LangChain integration when your app runs chains, chat models, retrievers, tools, or agents through LangChain callbacks. Each root execution owns one complete Lemma trace: current-turn input, final answer or root error, promoted `thread_id` / `user_id`, typed nested generations/tools/spans, and real wall-clock timing. Standalone LLM/tool/retriever callbacks finalize their own owned trace. Missing-parent events never overwrite concurrent trace state. You do not need to wrap LangChain calls in `lemma.trace()`. Pass the callback handler through LangChain’s `callbacks` option, then `flush()` to send the trace. In a long-lived process, open traces that stay idle past `open_trace_ttl` (default 2 hours) are finalized and sent so handler state cannot grow without bound. Last activity is the latest of the root open time, child ends, and in-flight run starts, so a long agent loop that still emits callbacks is not evicted. The sweep runs from every `on_*_start` callback at most once per `eviction_interval` (default 5 minutes). Lemma sends the partial owned trace, the same payload `flush()` would, rather than dropping abandoned runs. Pass `open_trace_ttl=None` to disable. A complete docs-chat agent using this integration lives in [`examples/langchain`](https://github.com/uselemma/lemma/tree/main/examples/langchain) (TypeScript) and [`examples/python/langchain`](https://github.com/uselemma/lemma/tree/main/examples/python/langchain) (Python). See [Runnable examples](/guides/examples). ## TypeScript Install LangChain and the Lemma SDK: ```bash theme={null} npm install @uselemma/tracing @langchain/core @langchain/openai ``` Pass `langChain()` as a callback handler: ```typescript theme={null} import { ChatOpenAI } from "@langchain/openai"; import { HumanMessage } from "@langchain/core/messages"; import { langChain } from "@uselemma/tracing"; const lemmaHandler = langChain({ apiKey: process.env.LEMMA_API_KEY, projectId: process.env.LEMMA_PROJECT_ID, release: process.env.LEMMA_RELEASE, agentName: "support-agent", threadIdKey: "conversation_id", userIdKey: "user_id", }); const model = new ChatOpenAI({ model: "gpt-4o", callbacks: [lemmaHandler], }); export async function callAgent(userMessage: string, conversationId: string) { const response = await model.invoke([new HumanMessage(userMessage)], { metadata: { conversation_id: conversationId }, }); await lemmaHandler.flush(); return response.content; } ``` ## Python Install the optional extra: ```bash theme={null} pip install "uselemma-tracing[langchain]" langchain-openai ``` Pass `langchain()` as a callback handler: ```python theme={null} import os from langchain_openai import ChatOpenAI from uselemma_tracing import langchain lemma_handler = langchain( api_key=os.environ["LEMMA_API_KEY"], project_id=os.environ["LEMMA_PROJECT_ID"], agent_name="support-agent", thread_id_key="conversation_id", user_id_key="user_id", ) model = ChatOpenAI(model="gpt-4o", callbacks=[lemma_handler]) def call_agent(user_message: str, conversation_id: str): response = model.invoke( user_message, config={"metadata": {"conversation_id": conversation_id}}, ) return response.content ``` ## What Lemma records | LangChain callback | Lemma record | | - | - | | Root chain / standalone LLM / tool / retriever | One owned Lemma trace with current-turn input, final output or root error, `thread_id` / `user_id`, and wall times | | LLM or chat model start/end | Generation with normalized `{ role, content }` history, structured assistant output (incl. tool calls), derived provider/model, timing, parent ID | | Tool start/end | Tool call with name, input, output **or** soft/hard error (never both), timing, parent ID | | Retriever start/end | Span with query input, output, timing, and parent ID | | Nested chain start/end | Child span preserving LangChain parent run IDs (orphan-safe fallback to root) | Promote identity from configurable metadata/tag keys (`threadIdKey` / `thread_id_key`, `userIdKey` / `user_id_key`). Defaults also check `conversation_id`, `session_id`, and `resourceId`. Prompts, tool inputs, outputs, model output text, and error messages are always recorded. Redact secrets and sensitive user data before they reach the chain. ## Debug this integration Enable debug mode while developing to confirm that spans arrive live and stay nested under the expected parent: ```typescript theme={null} import { enableDebugMode } from "@uselemma/tracing"; enableDebugMode(); ``` ```python theme={null} from uselemma_tracing import enable_debug_mode enable_debug_mode() ``` # LangGraph Source: https://docs.uselemma.ai/integrations/langgraph Lemma callback tracing for LangGraph graph runs LangGraph uses LangChain’s callback system, so Lemma’s LangGraph integration is an explicit LangChain callback adapter with LangGraph defaults (default trace name `langgraph-agent`). It creates one owned Lemma trace per graph invocation, records graph nodes as nested child spans, model calls as generations, tools as tool calls, extracts current-turn input from graph message state, and preserves parent run IDs with orphan-safe fallback. You do not need to wrap graph calls in `lemma.trace()`. Pass the callback handler through the graph invoke config. A complete docs-chat agent using this integration lives in [`examples/langgraph`](https://github.com/uselemma/lemma/tree/main/examples/langgraph) (TypeScript) and [`examples/python/langgraph`](https://github.com/uselemma/lemma/tree/main/examples/python/langgraph) (Python). See [Runnable examples](/guides/examples). ## TypeScript Install LangGraph and the Lemma SDK: ```bash theme={null} npm install @uselemma/tracing @langchain/langgraph ``` Pass `langGraph()` in the graph invoke config: ```typescript theme={null} import { StateGraph, START, END } from "@langchain/langgraph"; import { langGraph } from "@uselemma/tracing"; type GraphState = { input: string; output?: string; }; const graph = new StateGraph() .addNode("answer", async (state) => ({ output: `You said: ${state.input}`, })) .addEdge(START, "answer") .addEdge("answer", END) .compile(); export async function callGraph(userMessage: string) { const lemmaHandler = langGraph({ apiKey: process.env.LEMMA_API_KEY, projectId: process.env.LEMMA_PROJECT_ID, release: process.env.LEMMA_RELEASE, agentName: "support-graph", }); const result = await graph.invoke( { input: userMessage }, { callbacks: [lemmaHandler], }, ); await lemmaHandler.flush(); return result.output; } ``` ## Python Install the optional extra: ```bash theme={null} pip install "uselemma-tracing[langgraph]" langgraph ``` Pass `langgraph()` in the graph invoke config: ```python theme={null} import os from typing_extensions import TypedDict from langgraph.graph import END, START, StateGraph from uselemma_tracing import langgraph class GraphState(TypedDict): input: str output: str def answer(state: GraphState): return {"output": f"You said: {state['input']}"} graph = ( StateGraph(GraphState) .add_node("answer", answer) .add_edge(START, "answer") .add_edge("answer", END) .compile() ) def call_graph(user_message: str): result = graph.invoke( {"input": user_message}, { "callbacks": [ langgraph( api_key=os.environ["LEMMA_API_KEY"], project_id=os.environ["LEMMA_PROJECT_ID"], agent_name="support-graph", ) ] }, ) return result["output"] ``` ## What Lemma records | LangGraph callback | Lemma record | | - | - | | Root graph run | One owned Lemma trace (default name `langgraph-agent`) with current-turn input from state, final output or root error, `thread_id` / `user_id`, wall times | | Graph node | Child span with node input, output, timing, and parent ID | | LLM or chat model | Generation with normalized messages, structured assistant output, derived provider/model, timing, parent ID | | Tool call | Tool span with input, output **or** soft/hard error, timing, and parent ID | | Retriever | Span with query input, output, timing, and parent ID | `langGraph()` / `langgraph()` is an explicit LangChain-callback adapter. LangGraph emits LangChain callback events; Lemma does not use a separate LangGraph tracer. Prompts, tool inputs, outputs, model output text, and error messages are always recorded. Redact secrets and sensitive user data before they reach the graph. ## Debug this integration Enable debug mode while developing to confirm that spans arrive live and parent IDs are attached: ```typescript theme={null} import { enableDebugMode } from "@uselemma/tracing"; enableDebugMode(); ``` ```python theme={null} from uselemma_tracing import enable_debug_mode enable_debug_mode() ``` # Mastra Source: https://docs.uselemma.ai/integrations/mastra Lemma exporter for Mastra agents and workflows Use Lemma’s Mastra integration when your app runs agents or workflows through Mastra observability. The integration implements a Mastra `ObservabilityExporter` that creates one Lemma trace for each Mastra `agent_run` / `workflow_run`, maps model generations to Lemma generations, maps tool calls to tools, and preserves nesting (with orphan-safe parent fallback when Mastra hides internal spans). You do not need to wrap Mastra runs in `lemma.trace()`. Register `LemmaMastraExporter` once in Mastra’s `Observability` config, then run agents normally. A complete docs-chat agent using this integration lives in [`examples/mastra`](https://github.com/uselemma/lemma/tree/main/examples/mastra). See [Runnable examples](/guides/examples). ## Install the SDK Install Mastra and the Lemma SDK: ```bash theme={null} npm install @uselemma/tracing @mastra/core @mastra/observability ``` ## Register the exporter Register the Lemma exporter in your Mastra observability config. By default, credentials are read from `LEMMA_API_KEY` and `LEMMA_PROJECT_ID`: ```typescript theme={null} import { Mastra } from "@mastra/core"; import { Observability } from "@mastra/observability"; import { LemmaMastraExporter } from "@uselemma/tracing"; export const mastra = new Mastra({ // agents / workflows... observability: new Observability({ configs: { default: { serviceName: "my-app", exporters: [new LemmaMastraExporter()], }, }, }), }); ``` You can also pass credentials and options directly, or use the `mastra()` factory alias: ```typescript theme={null} import { mastra as lemmaMastra } from "@uselemma/tracing"; exporters: [ lemmaMastra({ apiKey: process.env.LEMMA_API_KEY, projectId: process.env.LEMMA_PROJECT_ID, baseUrl: process.env.LEMMA_BASE_URL, release: process.env.LEMMA_RELEASE, agentName: "support-agent", }), ] ``` Then run your agent normally: ```typescript theme={null} const lemmaExporter = new LemmaMastraExporter(); const agent = mastra.getAgent("supportAgent"); export async function callAgent(userMessage: string) { const result = await agent.generate(userMessage); await lemmaExporter.flush(); return result.text; } ``` ## What Lemma records Each Mastra span maps to a Lemma record: | Mastra span | Lemma record | | - | - | | `agent_run` / `workflow_run` (root) | One Lemma trace with current-turn input, final output (or `fail` on error), `threadId`, and `userId` | | `model_generation` / `model_step` | Generation with model, provider, full message list as `input` + `llmInputMessages`, output, timing, and parent ID | | `tool_call` / `mcp_tool_call` / `client_tool_call` / `provider_tool_call` | Tool call with tool name, input, output or soft/`errorInfo` error, timing, and parent ID | | Other span types (`generic`, `processor_run`, `workflow_step`, …) | Regular span with input, output, timing, and parent ID | The trace name is the Mastra agent or workflow id (`entityId`), falling back to its display name (`entityName`), then to the id parsed out of run span names like `agent run: 'support-agent'` / `workflow run: 'nightly-report'`. Set `agentName` on the exporter to override it. Mastra may set root `input` to `{ messages: [...] }`. Lemma extracts the latest user message as the trace root input so each turn stays readable; generations still carry the full message list Mastra sent to the model. Every child span, and the root, keeps Mastra’s real `startTime` / `endTime`. If a child’s `parentSpanId` points at a span Mastra never exported, Lemma attaches that child directly under the trace root so parent references never dangle. ## Threads / multi-turn Each Mastra `agent.generate()` (or workflow run) is one conversation turn and becomes exactly one Lemma trace. Link turns in the same conversation with a shared `threadId`, and set `userId` so Lemma can group by person. See [Instrumenting multi-turn agents](/guides/instrumenting-multi-turn-agents) and [Building high-quality traces](/guides/building-high-quality-traces). Lemma resolves `threadId` and `userId` from the root span’s `metadata` or `requestContext`, checking `threadId` and `userId` (then Mastra memory’s `resourceId`). Pass them via Mastra memory or `tracingOptions.metadata`: ```typescript theme={null} const result = await agent.generate(userMessage, { memory: { thread: conversationId, resource: user.id, }, // Or set explicitly on the root span: tracingOptions: { metadata: { threadId: conversationId, userId: user.id, }, }, }); ``` Override the lookup keys when your app uses different names: ```typescript theme={null} new LemmaMastraExporter({ threadIdKey: "conversationId", userIdKey: "customerId", }) ``` Root `input` is the current user turn (extracted from Mastra’s `{ messages }` payload when present). Generations carry the full message list sent to that model call (system, prior turns, tool results, and the current user message) as both `input` and `llmInputMessages`. ## What gets recorded The integration always records prompts, tool inputs, tool outputs, model output, and error messages: a trace cannot show what a run produced or why it failed without them. Redact secrets and sensitive user data before they reach the agent. ## Debug this integration Use debug mode to confirm spans are arriving nested under the expected parent: ```typescript theme={null} import { enableDebugMode } from "@uselemma/tracing"; enableDebugMode(); ``` Look for `span recorded` logs. Tool and generation spans should include the expected `parentId` when Mastra exported the parent, or sit directly under the trace when the parent was an internal/hidden span. # OpenAI Agents SDK Source: https://docs.uselemma.ai/integrations/openai-agents Lemma processor for the OpenAI Agents SDK Use Lemma’s OpenAI Agents integration when your app uses the OpenAI Agents SDK trace processor system. The integration creates one Lemma trace for each OpenAI Agents trace, maps model calls to generations, maps function calls to tools, and preserves parent IDs so child spans stay nested correctly. You do not need to wrap agent runs in `lemma.trace()`. Register the processor once, then run OpenAI Agents normally. A complete docs-chat agent using this integration lives in [`examples/openai-agents`](https://github.com/uselemma/lemma/tree/main/examples/openai-agents) (TypeScript) and [`examples/python/openai-agents`](https://github.com/uselemma/lemma/tree/main/examples/python/openai-agents) (Python). See [Runnable examples](/guides/examples). ## TypeScript Install both packages: ```bash theme={null} npm install @uselemma/tracing @openai/agents ``` Register the Lemma processor before running agents. By default, credentials are read from `LEMMA_API_KEY` and `LEMMA_PROJECT_ID`: ```typescript theme={null} import { addTraceProcessor } from "@openai/agents"; import { openAIAgents } from "@uselemma/tracing"; addTraceProcessor(openAIAgents()); ``` You can also pass credentials and identity options directly: ```typescript theme={null} addTraceProcessor( openAIAgents({ apiKey: process.env.LEMMA_API_KEY, projectId: process.env.LEMMA_PROJECT_ID, baseUrl: process.env.LEMMA_BASE_URL, release: process.env.LEMMA_RELEASE, metadata: { service: "support" }, threadIdKey: "threadId", userIdKey: "userId", }), ); ``` Then run your agent normally: ```typescript theme={null} import { Agent, addTraceProcessor, run } from "@openai/agents"; import { openAIAgents } from "@uselemma/tracing"; const processor = openAIAgents(); addTraceProcessor(processor); const agent = new Agent({ name: "support-agent", instructions: "Answer customer questions clearly and concisely.", }); export async function callAgent(userMessage: string) { const result = await run(agent, userMessage); await processor.forceFlush(); return result.finalOutput; } ``` ## Python Install the Python extra: ```bash theme={null} pip install "uselemma-tracing[openai-agents]" openai-agents ``` Register the Lemma processor before running agents. By default, credentials are read from `LEMMA_API_KEY` and `LEMMA_PROJECT_ID`: ```python theme={null} from uselemma_tracing import instrument_openai_agents instrument_openai_agents() ``` You can also pass credentials and identity options directly: ```python theme={null} import os from uselemma_tracing import instrument_openai_agents instrument_openai_agents( api_key=os.environ["LEMMA_API_KEY"], project_id=os.environ["LEMMA_PROJECT_ID"], base_url=os.getenv("LEMMA_BASE_URL", "https://api.uselemma.ai"), metadata={"service": "support"}, thread_id_key="thread_id", user_id_key="user_id", ) ``` Then run your agent normally: ```python theme={null} from agents import Agent, Runner agent = Agent( name="support-agent", instructions="Answer customer questions clearly and concisely.", ) async def call_agent(user_message: str): result = await Runner.run(agent, user_message) return result.final_output ``` ## Root input, output, user, and thread Both TypeScript and Python processors promote the same root fields on the Lemma trace: | Field | Source | | - | - | | Root input | First generation/response messages: last user message content | | Root output | Last successful generation/response output | | Root error | Hard errors on terminal span types (`agent`, `task`, `guardrail`, `custom`), not recovered soft tool errors | | `threadId` / `thread_id` | OpenAI Agents `group_id`, then configurable metadata key | | `userId` / `user_id` | Metadata (`userId`, `user_id`, or `resourceId` by default; override with `userIdKey` / `user_id_key`) | Soft tool failures (`isError` / `is_error` / Mastra `error: true`) mark the tool span as `ERROR` without failing the root when a later generation succeeds. ## Flush and shutdown Call `forceFlush()` / `force_flush()` (or `shutdown()`) to send any still-open traces exactly once. A later `onTraceEnd` / `on_trace_end` for a flushed trace is a no-op, so spans are never duplicated in the append-only ingest store. ```typescript theme={null} const processor = openAIAgents(); addTraceProcessor(processor); // ... await processor.forceFlush(); ``` ```python theme={null} from uselemma_tracing import openai_agents processor = openai_agents() # register processor, run agents... processor.force_flush() ``` ## What Lemma records | OpenAI Agents event | Lemma record | | - | - | | Trace start/end | One Lemma trace named from the OpenAI Agents trace, with root I/O and timing from span bounds | | Generation or response span | Generation with model, input, output, timing, and OpenAI span attributes | | Function span | Tool call with tool name, input, output or error, timing, and parent ID | | Other span types | Regular span with OpenAI trace/span fields preserved in attributes | OpenAI Agents trace `group_id` is recorded as the Lemma thread id, so runs in the same conversation stay grouped. ## What gets recorded The integration always records prompts, tool inputs, tool outputs, model output, and error messages. A trace cannot show what a run produced or why it failed without them. Redact secrets and sensitive user data before they reach the agent. ## Debug this integration Use debug mode to confirm spans are arriving live and nested under the expected parent: ```typescript theme={null} import { enableDebugMode } from "@uselemma/tracing"; enableDebugMode(); ``` ```python theme={null} from uselemma_tracing import enable_debug_mode enable_debug_mode() ``` Look for `span started` and `span ended` logs. Tool/function spans should include the generation or agent span ID as `parentId` in TypeScript or `parent_id` in Python. # Vercel AI SDK Source: https://docs.uselemma.ai/integrations/vercel-ai Record Vercel AI SDK model calls and tool executions with Lemma Use `vercelAI()` when your app calls `generateText`, `streamText`, or an AI SDK agent. The integration supports AI SDK v7 and v6. It creates one Lemma trace for each AI SDK run, puts the current user turn on the root, promotes `threadId` / `userId` from telemetry metadata, and records model calls and tool executions as child spans. Create a **new** `vercelAI()` integration for each AI SDK operation. One integration object owns one in-flight run; concurrent reuse fails fast because AI SDK terminal events do not carry a reliable run ID. Sequential reuse after a completed run is supported. A complete docs-chat agent using this integration lives in [`examples/vercel-ai`](https://github.com/uselemma/lemma/tree/main/examples/vercel-ai). See [Runnable examples](/guides/examples). ## Install the SDK ```bash theme={null} npm install @uselemma/tracing ai zod ``` ## AI SDK v7 Pass a fresh Lemma telemetry integration to the AI SDK `telemetry.integrations` option. Use `telemetry.functionId` for the agent name and put conversation identity in `telemetry.metadata`. ```typescript theme={null} import { generateText, tool } from "ai"; import { z } from "zod"; import { vercelAI } from "@uselemma/tracing"; const lemmaTelemetry = vercelAI({ apiKey: process.env.LEMMA_API_KEY, projectId: process.env.LEMMA_PROJECT_ID, release: process.env.LEMMA_RELEASE, }); try { const result = await generateText({ model, prompt: userMessage, tools: { searchDocs: tool({ inputSchema: z.object({ query: z.string(), }), execute: async ({ query }) => searchDocs(query), }), }, telemetry: { functionId: "support-agent", metadata: { threadId: conversationId, userId: user.id, }, integrations: [lemmaTelemetry], }, }); await lemmaTelemetry.flush(); return result.text; } catch (error) { await lemmaTelemetry.fail(error); throw error; } ``` ## AI SDK v6 For AI SDK v6, pass the same helper through `experimental_telemetry.integrations`. Use `experimental_telemetry.functionId` for the agent name. ```typescript theme={null} import { generateText, tool } from "ai"; import { z } from "zod"; import { vercelAI } from "@uselemma/tracing"; const lemmaTelemetry = vercelAI({ apiKey: process.env.LEMMA_API_KEY, projectId: process.env.LEMMA_PROJECT_ID, release: process.env.LEMMA_RELEASE, }); try { const result = await generateText({ model, prompt: userMessage, tools: { searchDocs: tool({ inputSchema: z.object({ query: z.string(), }), execute: async ({ query }) => searchDocs(query), }), }, experimental_telemetry: { functionId: "support-agent", metadata: { threadId: conversationId, userId: user.id, }, integrations: [lemmaTelemetry], }, }); await lemmaTelemetry.flush(); return result.text; } catch (error) { await lemmaTelemetry.fail(error); throw error; } ``` ## Streaming Use the same per-operation integration with `streamText`. Call `flush()` after the stream finishes to send the trace. Call `fail(error)` if the call throws before a terminal callback. ```typescript theme={null} import { streamText } from "ai"; import { vercelAI } from "@uselemma/tracing"; const lemmaTelemetry = vercelAI({ apiKey: process.env.LEMMA_API_KEY, projectId: process.env.LEMMA_PROJECT_ID, release: process.env.LEMMA_RELEASE, }); const result = streamText({ model, prompt: userMessage, telemetry: { functionId: "support-agent", metadata: { threadId: conversationId, userId: user.id, }, integrations: [lemmaTelemetry], }, }); for await (const _part of result.fullStream) { // stream to your response } await lemmaTelemetry.flush(); ``` ## What Lemma records | AI SDK event | Lemma record | | - | - | | Run start / finish | Root trace with current-turn input, final answer or root error, `thread_id` / `user_id`, and real wall-clock bounds | | Model call / step | Generation with model, provider, normalized messages, structured assistant output, and duration | | Tool execution | Tool call with name, arguments, result or soft/hard error (never both success output and error), and duration | AI SDK v7 includes model and tool durations. AI SDK v6 includes tool execution durations; model-call durations are inferred from start and finish timestamps when the AI SDK omits an explicit duration. `system` / `instructions` is prepended onto `messages` or `prompt` so generations include the playbook that was sent for that step. Use `telemetry.functionId` / `experimental_telemetry.functionId` for the agent name, or set it on the integration with `vercelAI({ agentName: "support-agent" })`. Override metadata keys when needed: ```typescript theme={null} vercelAI({ threadIdKey: "conversationId", userIdKey: "customerId", }) ``` For advanced cases, you can still attach to an existing trace by passing `vercelAI({ trace })` or by calling AI SDK inside a `lemma.trace()` callback. If you use the callback form of `lemma.trace()`, the callback owns trace closure. If you pass a trace handle to `vercelAI({ trace })`, the integration closes it automatically at the end of the AI SDK operation. Prompts, tool inputs, tool outputs, model output, and error messages are always recorded. Redact secrets and sensitive user data before they reach the AI SDK call. ## Abort, flush, and shutdown | Method | When to use | | - | - | | `fail(error)` | The AI SDK call threw or was aborted before a terminal telemetry callback | | `flush()` | Send the integration’s completed traces and await ingest delivery | | `shutdown()` | Flush and clear state in workers / serverless teardown | ```typescript theme={null} const lemmaTelemetry = vercelAI(); try { const result = await generateText({ model, prompt: userMessage, telemetry: { functionId: "support-agent", integrations: [lemmaTelemetry], }, }); await lemmaTelemetry.flush(); return result.text; } catch (error) { await lemmaTelemetry.fail(error); throw error; } finally { await lemmaTelemetry.shutdown(); } ``` # Analytics Source: https://docs.uselemma.ai/platform/analytics Project-level volume, latency, errors, tools, models, and estimated cost Analytics charts volume, latency, errors, tools, models, and estimated cost for every trace in a project over a chosen window. Use it to spot a trend, then jump to the traces or issues behind it. Open it from a project in the [Lemma dashboard](https://platform.uselemma.ai). Analytics is feature-gated. If you don’t see **Analytics** in the project sidebar, it isn’t enabled for your workspace. The project **Dashboard** (home and setup) is a different page. See [Product boundaries](/platform/concepts#product-boundaries). Analytics dashboard showing headline stats, an agent scorecard, trace latency over time, model comparison, cost by model, and error rate widgets ## Timeframes and granularity Pick a window: all time (clamped to the Analytics horizon), this week, this month, this quarter, this year, or a custom range. Lemma buckets the charts by day, week, or month, defaulting to whatever keeps them readable: day for a week or month window, week for a quarter, month for a year or all time. Analytics only looks back 90 days on a rolling basis, so older data won’t show even if you pick all time. Individual traces on the [Traces](/platform/traces) list are retained for 14 days, which means a chart point can represent a day whose raw traces have already aged out of that list. Use Analytics to see the trend and Traces to open the executions you can still read. ## Project and agent scope Analytics is project-scoped, and the agent scorecard lists every agent that appeared in the window. Agent names come from the traces themselves, not from a registry you maintain elsewhere. A missing agent means it didn’t emit ready traces in this project and window, not that someone deleted it from a catalog. ## What the surface shows Headline stats for the window include trace count, errors, p50 and p95 end-to-end latency, and estimated cost, each with a period-over-period delta when the prior window still fits inside the 90-day horizon. Below the headline stats, you can add, remove, and reorder widgets. Lemma only offers a widget once its required telemetry is present: * **Agent scorecard**: volume, latency, error rate, tools, tokens, and cost per agent * **Trace latency over time**: p50 and p95 latency plotted across the window * **Model comparison**: generation volume, latency, error rate, tokens, and cost by model * **Cost by model**: stacked estimated cost over time * **Tool call volume**: stacked calls by tool * **Tool comparison**: calls, latency, error rate, and share of traces by tool * **Error rate**: root-span error rate over time * **Tool error concentration**: tools ranked by failing proportion, linking into Traces Cost figures are estimated model cost calculated from recorded usage, not a billed invoice. Lemma doesn’t claim you were charged a specific amount on this page. Empty and low-data states are expected, not a bug. A new project, a quiet agent, or traces missing model, tool, or cost fields will hide or zero the widgets that depend on that data. Send complete traces (see [Building high-quality traces](/guides/building-high-quality-traces) and [Analytics telemetry](/reference/analytics-telemetry)) before assuming an empty chart means an outage. ## From a trend to evidence When a chart shows something worth investigating, open **[Traces](/platform/traces)** filtered to that window, and to the tool when the widget links there. If the failure looks recurring rather than a single spike, check **[Issues](/platform/issues)** instead. You can also ask **[Inspect](/platform/inspect)** window-aware questions directly on the Analytics page; it explains what the charts show rather than replacing them. For one aggregate from REST or MCP, see [Querying analytics](/guides/query-analytics). To have a coding agent run those queries, install the analytics skill described at the top of that guide. ## Related pages How traces, issues, Inspect, Artifacts, and Analytics fit together. Open the executions behind a chart point. Recurring failure patterns across many traces. What the SDK must emit for widgets to populate. Run one aggregate from REST or MCP. # Artifacts Source: https://docs.uselemma.ai/platform/artifacts Per-agent context, understanding documents, and decision-flow diagrams Artifacts are agent-scoped context that sharpen how Lemma interprets traces and issues for one agent. They aren’t instrumentation, and they don’t substitute for sending complete traces in the first place. Open **Artifacts** in a project in the [Lemma dashboard](https://platform.uselemma.ai). Every artifact is scoped to one Lemma project plus one agent name, and agent names come from observed traces. Context you write for `support-agent` says nothing about `billing-agent`, even in the same project. Artifacts page for an agent showing the decision-flow diagram, generated understanding document, and dismissal notes ## Context you write and context Lemma generates You supply **provided context**: text, or a supported text file, describing behavioral expectations, domain rules, architecture notes, or things the agent should never do. Don’t upload secrets, API keys, or customer personally identifiable information (PII) into it. Lemma generates the second layer, an **understanding document** and a **decision-flow diagram**, from observed traces plus whatever context you provided. Both are versioned, so review a new version after you regenerate rather than assuming the latest draft is automatically correct. ## Setting up an agent’s artifacts Send ready traces that identify the intended agent. See [Quickstart](/tracing/instrumentation/setup). Open **Artifacts** and select the agent, or the untagged bucket if your traces don’t carry an agent name yet. Add concise text or upload a supported text file. Favor short, testable rules over pasting in your entire prompt library. Review the generated document and diagram. When your context changes materially, regenerate if the UI offers it, then review the new version before trusting it. If an artifact is project-wide or applies to an unknown agent, describe it that way instead of assigning it to a named agent it doesn’t belong to. ## How this helps detection Extraction and Inspect use your context to judge whether a run violated your rules specifically, not only generic failure patterns. Sparse or contradictory notes make findings noisier, so precision here pays off. Artifacts can’t create traces, register an application entity, or replace issue status and Slack alerts; they only inform how Lemma reads what already arrived. Dismissal notes on this page log the runs you told Lemma weren’t real problems, with your reasoning attached. That feedback is part of the context Lemma reads for this agent going forward. ## Inspect on this page Because Inspect already knows the selected agent name, “this agent” resolves without you naming it. Useful prompts here: summarize what this agent does, walk through its decision flow, suggest what context to add, or list its recent issues. ## Related pages How artifacts relate to traces, issues, and agents. Projects, agents, and how dashboard surfaces fit together. Failure patterns that Artifacts context can sharpen. Ask what this agent does and what context to add. Complete traces still matter more than extra context. # Dashboard vocabulary Source: https://docs.uselemma.ai/platform/concepts Traces, issues, artifacts, and agents in the Lemma dashboard Once traces start arriving, four more nouns describe what you see in the dashboard. They build on the software development kit (SDK) vocabulary in [SDK concepts](/tracing/instrumentation/concepts): a trace is still the unit you send, and everything below describes what Lemma does with traces after ingest. ## Project A **project** is the top-level container for traces, issues, artifacts, and settings from your agent or agents. Each project has its own dashboard, API key, and project ID. Use separate projects to isolate environments (dev, staging, prod) or distinct products. ## Agent An **agent** isn’t something you register. It’s a name that appears on a trace, and Lemma uses that name to scope Artifacts, Analytics scorecards, and issue grouping to the right agent. A trace with no usable agent name still ingests, but nothing agent-scoped has anywhere to attach it. ## Trace A **trace** is one ready agent execution: the conversation, the execution tree of spans, and any errors it produced. Traces are what you search and open on the [Traces](/platform/traces) page, and they’re the raw evidence behind every issue. ## Issue An **issue** is a durable grouped failure pattern inferred from trace evidence. Say the issue was **observed in a trace**. Statuses are **open**, **in progress**, **resolved**, and **dismissed**. Lemma analyzes a trace after its conversation goes quiet, or immediately if the trace has no thread ID, then groups repeated evidence so the same bug accumulates supporting traces instead of spawning a new issue every time it recurs. Opening an issue shows you the pattern. Opening a supporting trace shows you the specific run. See [Issues](/platform/issues) for status, priority, and investigation. Detection and grouping are probabilistic. Treat an issue as a hypothesis backed by evidence, not a certainty, and open the supporting traces before you call it confirmed. Alerting follows this issue lifecycle; see [Product boundaries](#product-boundaries). ## Artifact **Artifacts** are agent-scoped. **Provided** context is text (or a supported text file) you supply about that agent. **Generated** artifacts are a versioned understanding document and a decision-flow diagram Lemma builds from traces plus that context. Agent names come from observed traces. Context for one named agent is not evidence about another. See [Artifacts](/platform/artifacts). ## Product boundaries Lemma evaluates **production usage** from real users. It does not offer offline evaluation. If someone asks about “online eval,” they mean evaluating actual production usage. Standalone monitors, incidents, and a standalone metrics product are not current surfaces. Failure alerting follows the issue lifecycle via [Slack](/connections/slack) and [webhooks](/connections/webhooks). ## Related pages How a trace moves through detection to become an issue. Status, priority, and investigating supporting traces. Agent-scoped context that sharpens detection. Trace, span, generation, tool call, and thread. # Inspect Source: https://docs.uselemma.ai/platform/inspect Ask Lemma about this project’s traces, issues, artifacts, and settings Inspect is Lemma’s in-product assistant. It answers from this project’s own traces, issues, artifacts, and settings, not from a generic model of your codebase. Ask it questions you’d otherwise dig through the dashboard to answer. When you need to scan, filter, or change status yourself, use [Traces](/platform/traces) or [Issues](/platform/issues) instead. Open the project’s **Inspect** page in the [Lemma dashboard](https://platform.uselemma.ai), or the Inspect widget available from other project pages. Inspect conversation showing a question about a latency regression and an answer grounded in trace evidence ## Where it runs Inspect appears in two places with the same durable conversation model: the full-page **Inspect** view for the project, and an **Inspect widget** that overlays other project pages. Inspect is page-aware. It reads context from the URL, and from an open trace drawer, so you can say “this issue” or “this trace” without pasting an ID: | You’re on | Inspect treats as | | - | - | | A specific issue | That issue | | A specific trace, or a trace drawer | That trace | | Traces list, no trace open | Project traces, list-level prompts | | Issues list, no issue open | Project issues, triage prompts | | Analytics | The Analytics window from the URL | | Artifacts | The selected agent | | Settings | Setup and integration how-tos | An open trace drawer takes priority over the issue URL underneath it, so Inspect talks about the trace you’re viewing, not the issue page behind it. ## What it can read, explain, or change Inspect reads and explains project evidence: traces, issues, Artifacts (the understanding document, the diagram, and whatever context you provided), Analytics windows, and how Slack, Linear, MCP, and webhooks are meant to be set up. It’s grounded in Lemma’s actual product surface. See [Product boundaries](/platform/concepts#product-boundaries) for what that surface does and doesn’t include. Inspect doesn’t replace the system of record. Issue status, Linear linking, and webhook configuration still live on Issues and Settings; Inspect can explain them but not change them on your behalf. It never returns secret values like API keys, Slack tokens, MCP credentials, or webhook signing secrets. It points you to the UI or your secret manager instead. It treats setup status as a snapshot: documentation or a configured URL isn’t proof that this specific project finished setup. Within a conversation, your permissions determine what you can do: edit a turn, regenerate a response, fork the conversation, share it with organization members, report a bad response, or rename and delete it. Sharing stays scoped to your organization, not a public unauthenticated URL. ## History, attachments, and new chats Conversations persist, so open **history** to resume one instead of starting over. Start a **new chat** when the topic changes rather than dragging Inspect through an unrelated thread. Attach the current page (a trace or an artifact, for example) so Inspect is looking at the same object you are. Shared conversations show up in the shared history for any member you granted access to. ## Suggested questions by page The UI swaps in page-specific prompts where it has them. Until then, these work everywhere: * **A trace**: why was this run flagged, and are there other traces with the same issue? * **An issue**: why did this fire, what scenarios triggered it, and how do I fix it? * **Issues list**: what should I look at first, and what’s the impact? * **Traces list**: where are users frustrated, confused, or asking for a human? * **Artifacts**: what does this agent do, what’s its decision flow, and what context should I add? * **Settings**: how do I connect Slack, alert on new issues, or see which integrations exist? ## Related pages How dashboard surfaces fit together after traces arrive. Search and open individual executions. Recurring failure patterns with evidence. Context Inspect reads for a named agent. Investigate the same project from a coding agent instead of the dashboard. # Issues Source: https://docs.uselemma.ai/platform/issues Recurring failure patterns Lemma detects from trace evidence An **issue** is a durable grouped failure pattern that Lemma infers from trace evidence. Say the issue was observed in a trace. Open **Issues** in a project in the [Lemma dashboard](https://platform.uselemma.ai) to triage them. See [Concepts](/platform/concepts) for how these nouns relate. Issues table grouped by agent, showing signals, supporting-trace counts, assignee, status, and last seen You don’t write assertions for this to work. Lemma analyzes traces and groups repeated evidence into issues on its own. Detection and grouping are probabilistic: always open supporting traces before you treat a title as a confirmed bug. Alerting follows this issue lifecycle through [Slack](/connections/slack) and [webhooks](/connections/webhooks). See [Product boundaries](/platform/concepts#product-boundaries) for what Lemma does not ship. ## Issues and supporting traces One issue can have many supporting traces. Opening an issue shows the rationale behind it, its category, priority signals, linked traces, and available actions. Opening a trace instead shows you whether Lemma observed one or more issues in that run. ## Issue status | Status | Meaning | | - | - | | `open` | Active; nobody is working it yet | | `in_progress` | Active; someone is working it | | `resolved` | Terminal: you believe the underlying defect is fixed | | `dismissed` | Terminal: not a real problem, or not worth tracking | `resolved` and `dismissed` are mutually exclusive. If two people act on the same issue at once, whichever transition commits first wins. Matching evidence can reopen a resolved issue, but never reopens a dismissed one. Reopening an issue manually doesn’t by itself change a linked Linear ticket; see [Linear](/connections/linear) for how those stay in sync. You can also flag an issue for review, assign it to a Lemma member, and add tags. Assignment inside Lemma doesn’t write to Linear unless you create a ticket and choose combined assignment. ## Priority and grouping Priority comes from recurrence and impact measured across your whole population of issues, things like volume and spikes, not from a threshold you configure on a monitor. Treat it as a triage hint rather than a guarantee. Grouping aims for one issue per underlying defect, but it isn’t perfect. If two issues look like the same bug, check their supporting traces rather than assuming Lemma missed an obvious merge. Grouping gets revised as more traces arrive. ## Investigating an issue Use this page for status, priority, grouping, and the connections below. For the procedure, see [Turning issues into fixes](/guides/turning-issues-into-fixes). Do not resolve from the issue title, an Inspect answer, or an open pull request. Dismiss here only when the finding is not a real defect. Add feedback so later extraction can learn. ## Workflow connections Each connection reports on the issue lifecycle in a different way: * **[Slack](/connections/slack)**: issue alerts and Issue Briefs sent to a project channel * **[Linear](/connections/linear)**: an optional one-to-one ticket link that you create, exact-link, or unlink explicitly * **[Lemma MCP server](/connections/mcp)**: lets coding agents list and act on issues with an organization API key * **[Webhooks](/connections/webhooks)**: `issue.created`, `issue.resolved`, `issue.dismissed`, and `issue.reopened` events Connecting Slack or Linear never auto-creates a ticket, and recurrence never creates a new one either. ## Related pages How traces and issues relate. Open the supporting executions for an issue. Ask why an issue was flagged and how to fix it. Triage, inspect evidence, ship a change, and confirm it does not recur. Agent context that can sharpen detection. Issue alerts and Issue Briefs in your workspace. # What happens after traces arrive Source: https://docs.uselemma.ai/platform/overview How Lemma turns a trace into an issue, and where projects and agents fit Lemma turns each agent execution into a [trace](/tracing/instrumentation/concepts#trace), finds the executions that went wrong, and groups repeated evidence into [issues](/platform/concepts#issue) you can triage. This page is the path after traces arrive: how a trace becomes an issue, how projects and agents relate, and which dashboard surface to open at each step. For software development kit (SDK) vocabulary, see [SDK concepts](/tracing/instrumentation/concepts). For dashboard vocabulary, see [Platform concepts](/platform/concepts). To send your first trace, see [Quickstart](/tracing/instrumentation/setup). See [Product boundaries](/platform/concepts#product-boundaries) for what Lemma evaluates and which surfaces are not shipped. ## How a trace becomes an issue 1. An instrumented agent emits a trace for one execution, or one conversation turn when you set a thread ID. 2. Lemma stores the spans and marks the trace **ready** when you can open it in the dashboard. An API key, an SDK dependency, or an ingest attempt alone does not prove observability. 3. Lemma analyzes completed traces for behavior that materially hurt the agent’s task. Detection is probabilistic: review the supporting traces before you treat a finding as ground truth. 4. Repeated evidence is grouped into an **issue**. Say the issue was observed in a trace. 5. You investigate in the dashboard or through [Inspect](/platform/inspect), Slack, Linear, or the Model Context Protocol (MCP). See [Turning issues into fixes](/guides/turning-issues-into-fixes). ## Projects, agents, and scope You create **projects**. A project holds traces, issues, artifacts, and project settings. Each project has its own dashboard. Use separate projects to isolate environments (dev, staging, prod) or distinct products. **Agents appear when named traces arrive.** Lemma does not ask you to register an application entity. The agent name on a trace is how Lemma groups Artifacts, Analytics scorecards, and issue scope. Traces without a usable agent name still ingest, but agent-scoped features have nothing to attach to. | Scope | What lives there | | - | - | | **Organization** | Slack workspace OAuth, Linear workspace OAuth, organization API keys | | **Project** | Alert channel and Issue Briefs, Linear team and project defaults, webhooks, traces, issues, artifacts | | **Person** | Linear identity mapping. Not applicable to API-key or other non-human principals | | **Application** | Not a Lemma entity. Agents appear when named traces arrive | ## Recommended workspace sequence 1. Select or create an organization and project. 2. Generate an API key and instrument the production agent. Keep `LEMMA_API_KEY` server-side: never commit it, and never put it in the browser. See [Quickstart](/tracing/instrumentation/setup). 3. Verify at least one real production trace becomes **ready**. Open [Traces](/platform/traces) and confirm the execution tree and conversation look complete. If the conversation is still active, wait: extraction runs after a thread goes quiet, or immediately when there’s no thread ID. 4. Add agent-specific [Artifacts](/platform/artifacts) context. 5. Connect the customer Slack workspace, bind a project alert channel, and optionally enable Issue Briefs. See [Slack](/connections/slack). 6. Configure [Linear](/connections/linear) if the team turns issues into tickets. Each member verifies their own Linear identity. 7. Connect external MCP servers only when Lemma needs their tools. See [Lemma MCP server](/connections/mcp#connect-an-external-mcp-server). 8. Connect the coding agent to [Lemma MCP](/connections/mcp) when it should investigate or act on Lemma evidence. 9. Add signed [webhooks](/connections/webhooks) when another system needs issue lifecycle events. 10. Set up the Lemma support Slack Connect channel independently if the plan includes it. It is not a product alert path. ## The dashboard surfaces Each surface answers a different question about the same traces: | Surface | What it’s for | | - | - | | **[Traces](/platform/traces)** | Search, filter, and open individual executions and threads. Distinct from the instrumentation [Agent traces](/tracing/instrumentation/traces) page, which explains how to create traces. | | **[Issues](/platform/issues)** | Recurring failure patterns with status and supporting traces. | | **[Inspect](/platform/inspect)** | An in-product assistant that reads the page you’re on (a trace, an issue, Analytics, Artifacts, settings) and can share conversations. | | **[Artifacts](/platform/artifacts)** | Per-agent context you provide, plus Lemma’s generated understanding document and decision-flow diagram. | | **[Analytics](/platform/analytics)** | Project-level volume, latency, errors, tools, models, and cost over a timeframe. Feature-gated; it may not appear in your workspace yet. | Start on Analytics or Issues to spot a trend. From there you open the supporting traces, ask Inspect what’s going on, and add Artifacts context if detection missed the mark. Alert or ticket the finding through [Connections](/connections/overview). Close the issue only after [Turning issues into fixes](/guides/turning-issues-into-fixes). ## Related pages Traces, issues, artifacts, and agents. Project-level volume, latency, errors, tools, models, and cost. Ask questions about the trace, issue, or page you’re on. Search and inspect ready executions in the dashboard. Recurring failure patterns with status and evidence. Triage, inspect evidence, ship a change, and confirm it does not recur. Per-agent context, understanding document, and decision-flow diagram. Slack, Linear, Lemma MCP, and webhooks. # Traces Source: https://docs.uselemma.ai/platform/traces Search, filter, and inspect ready traces in the Lemma dashboard This page covers searching, filtering, and inspecting traces in the dashboard once they’re ready. To create traces in the SDK, see [Agent traces](/tracing/instrumentation/traces). Open **Traces** in a project in the [Lemma dashboard](https://platform.uselemma.ai). This page is where you go once traces exist and you need to find a specific execution, not where you instrument one. Traces table showing rows with input previews, span counts, durations, and error marks ## What a row is Each row is one **ready** agent execution, meaning one conversation turn. Columns include time, agent, duration, span count, an input preview, and separate marks for errors and issues. Traces stay on this list for 14 days before they age out. Agent, release, user, and thread aren’t separate dashboards; they’re attributes on the trace that come from your SDK context. Filter or search on those fields directly, and see [Threads and context](/tracing/instrumentation/context) for how to set them. SDK `metadata` and `attributes` both land as searchable span attributes after ingest. ## Search and filters You can combine several filters at once: * Full-text or structured search, with saved searches * Time range, absolute or relative * Agent name * Duration, span count, token count, error state, and issue state * Attribute filters, for example `user.uid` if you recorded it Live updates can pause while you work a snapshot; refresh the page to pull in newly arrived traces. Batch **export** and **delete** apply to your current selection. Delete only removes the trace from the dashboard; it doesn’t un-send the original payload your agent already sent. An empty list usually means no ready traces exist in this project and window. Confirm ingest through the project get-started flow, or check ingest status for a specific trace ID, before assuming the dashboard is wrong. An API key or SDK install alone is not a ready trace. ## Opening a trace The detail view, whether a drawer or a dedicated page, has two panels that answer different questions. **Conversation** shows the user and assistant turns as a readable thread, and if traces share a thread ID, stitches multiple turns into one conversation. **Execution tree** shows the nested spans, generations, tool calls, and other work, each with timing, input and output, and any errors. Read the conversation to see what the person using the agent experienced, and the tree to find which tool or model call actually failed. Issues Lemma observed in that run appear on the trace so you can open them. Trace detail view with the conversation and linked issues on the left, and the execution tree with span timings on the right ## From traces to the rest of the product An issue mark means extraction grouped this run into a failure pattern; open **[Issues](/platform/issues)** to see its status and every other supporting trace. An error mark means a span or the root failed, which may or may not turn into an issue. From an open trace, ask **[Inspect](/platform/inspect)** to explain the flagged issue or find other traces like it. When you need the aggregate view across many traces rather than one execution, that’s **[Analytics](/platform/analytics)**. ## Related pages How traces become issues after they arrive. Grouped failure patterns across many traces. Ask questions about the trace you have open. How to create a root trace in the SDK. # Analytics query API Source: https://docs.uselemma.ai/reference/analytics-query Events, fields, request rules, responses, errors, and preset views for project analytics over REST and MCP Project analytics has three surfaces: the catalog lists what you can query, the query route runs one aggregate, and preset views return one precomputed [Analytics](/platform/analytics) panel. This page specifies each one. For a walkthrough with examples, see [Querying analytics](/guides/query-analytics). | Surface | REST | MCP tool | | - | - | - | | Catalog | `GET /projects/{project_id}/analytics/catalog` | `describe_project_analytics` | | Query | `POST /projects/{project_id}/analytics/query` | `query_project_analytics` | | Preset view | `GET /projects/{project_id}/analytics?view=...` | One tool per view, see [Preset views](#preset-views) | All routes need an organization API key with read or admin scope, sent as `Authorization: Bearer your_organization_api_key`. Analytics keeps 90 days of events. ## Catalog The catalog response has four keys. It’s the live source of truth for this page’s event and field lists: * `events`: each event’s `name`, `dimensions` (`field`, `type`), `measures` (`field`, `type`, `aggs`), and `cost_measures` * `filter_ops`: valid filter operators * `buckets`: valid time buckets * `limits`: `max_group_by`, `max_measures`, `max_filters`, `max_order_by`, `max_in_values`, `max_rows`, `max_window_days` Dimensions are string fields. Measures are numeric fields. Both can be filtered, and both accept `uniqExact`, though the catalog lists `aggs` only for measures. ## Events and fields Every event also has `actor`, `surface`, `user_id`, and `api_key_id`, which describe where the event was recorded. Missing string values are stored as `''`, not null. ### `trace.processed` One row per processed trace. Cost measure: `trace_cost`. | Field | Kind | Meaning | | - | - | - | | `agent_name` | Dimension | Agent that produced the trace | | `root_status_code` | Dimension | Root span status | | `root_duration_ms` | Measure | Whole-run latency in milliseconds | | `error_count` | Measure | Error spans in the trace | | `failing_tool_calls` | Measure | Failed tool spans in the trace | | `input_tokens`, `output_tokens` | Measure | Tokens across the trace | | `total_tokens` | Measure | `input_tokens + output_tokens`, `sum` only | | `estimated_cost_usd` | Measure | Stored per-trace estimate; use `trace_cost` for totals | Root latency on this event is `root_duration_ms`. The catalog also lists cost bookkeeping fields such as `estimated_cost_basis`, `cost_priced_tokens`, and `cost_unpriced_models`. This event has no queryable `trace_id`. ### `generation.completed` One row per model call. Cost measure: `generation_cost`. This event has no `agent_name`, so model use can’t be split by agent. | Field | Kind | Meaning | | - | - | - | | `model` | Dimension | Model name | | `trace_id`, `span_id` | Dimension | The call’s trace and span | | `duration_ms` | Measure | Model call latency in milliseconds | | `input_tokens`, `output_tokens` | Measure | Tokens for the call | | `tps` | Measure | Output tokens per second, null when unknown | | `error` | Measure | `1` if the call failed, else `0` | ### `tool.called` One row per tool span. | Field | Kind | Meaning | | - | - | - | | `tool` | Dimension | Tool name | | `agent_name` | Dimension | Agent of the trace the call ran in | | `trace_id`, `span_id` | Dimension | The call’s trace and span | | `duration_ms` | Measure | Tool call latency in milliseconds | | `output_tokens` | Measure | Output tokens recorded on the span | | `error` | Measure | `1` if the call failed, else `0` | ### `span.errored` One row per failed span of any type. Fields: `span_name`, `trace_id`, `span_id`, all dimensions. ### Issue lifecycle events `issue.created`, `issue.resolved`, `issue.dismissed`, `issue.reopened`, `issue.review_started`, `issue.flagged`, `issue.unflagged`, and `issue.slack_sent` carry only the fields every event has. Count them by time bucket. ## Query request A query body names one event, a window, and at least one measure. REST uses the snake\_case keys below. MCP uses `endExclusive`, `groupBy`, and `orderBy` for those three keys and takes `project_id` in the body. Unknown keys are rejected. | Key | Required | Rule | | - | - | - | | `from` | Yes | Event name | | `start` | Yes | ISO 8601, UTC, whole second, inclusive | | `end_exclusive` | Yes | ISO 8601, UTC, whole second, exclusive, after `start` | | `measures` | Yes | 1 to `max_measures` | | `filters` | No | Up to `max_filters`; a row must match all of them | | `group_by` | No | Up to `max_group_by` | | `order_by` | No | Up to `max_order_by` | | `limit` | No | 1 to `max_rows`; defaults to `max_rows` | | `join` | No | One other event | The window can’t exceed `max_window_days`. ### Measures A measure is `{ "as": "alias", "agg": "aggregation", "field": "field", "filter": {} }`: * `count` counts rows and takes no `field` * `uniqExact` counts distinct values of any field * `sum`, `avg`, `min`, `max`, `p50`, `p95`, and `p99` need a numeric field whose `aggs` lists the aggregation * `filter` is optional and holds one filter. The measure then aggregates only matching rows, while other measures see every row A cost measure uses `trace_cost` or `generation_cost` as `agg` and takes no `field` or `filter`. A query can have one cost measure, on the primary event. `generation_cost` also needs a `model` group whose alias is `model`. An alias starts with a lowercase letter, then uses up to 40 lowercase letters, digits, or underscores. Aliases must be unique across groups and measures. ### Filters A filter is `{ "field": "field", "op": "operator", "value": "value" }`: * `eq`, `neq`, `gt`, `gte`, `lt`, and `lte` take one string or number * `in` takes a non-empty array of up to `max_in_values` values * `is_null` and `is_not_null` take no `value` To match a missing string, such as an unset `agent_name`, filter `eq` with `""`. ### Groups and order A group is `{ "bucket": "day", "as": "day" }` or `{ "field": "tool", "as": "tool" }`. When you omit `as`, a bucket’s alias is `bucket` and a field’s alias is the field name. | Bucket | Row value | | - | - | | `minute`, `hour` | `2026-09-23 14:00:00` | | `day` | `2026-09-23` | | `week` | Date of the week’s Sunday | | `month` | Date of the month’s first day | An `order_by` entry is `{ "measure": "alias", "direction": "asc" }` or `"desc"`. It names a group alias or a primary measure alias. Without `order_by`, grouped rows are sorted by the groups. Groups with no rows are absent from the result. ### Join A join adds measures from one other event: `{ "from": "event", "on": "field", "filters": [], "measures": [] }`. Its rules: * `from` must differ from the primary event, and `on` must be a field on both events * Lemma groups the joined event by `on` and left-joins it to the primary rows on the primary group with that alias, so group the primary query by `on` and keep the default alias * `filters` applies only to the joined event * Primary rows without a match get null joined measures * Joined measures can’t be cost measures, reuse a primary alias, or appear in `order_by` Useful pairs: `trace.processed` with `tool.called` on `agent_name`, and `tool.called` with `generation.completed` or `span.errored` on `trace_id`. ## Query response A query returns `rows`, `truncated`, and `window`: * `rows`: one object per group, keyed by alias, or one object when there’s no `group_by`. Large integer results, such as counts and token sums, can be numeric strings * `truncated`: true when the row count reaches `limit`, or `max_rows` without a limit. More groups can exist * `window`: the `start` and `end_exclusive` that ran REST and MCP return the same snake\_case shape. ### Cost fields Cost rows don’t use the cost measure’s alias. A `generation_cost` row has `estimated_cost_usd` (null when unpriced) and `priced`. A `trace_cost` row has: * `estimated_cost_usd`, `priced_tokens`, `total_tokens`, and `priced_token_share` * `unpriced_models` and `pricing_fetched_at` * `cost_source`: `stored`, `mixed`, or `query_time` * `cost_lower_bound` and `cost_lower_bound_reasons`, from `unpriced_models`, `prices_unavailable`, `unbackfilled_traces`, `live_priced_breakdown_truncated`, and `model_rollup_truncated` * `model_breakdown_truncated` * `attribution`: `coverage` (`all`, `unwritten_only`, or `none`) and `values`, a map from model to estimated cost for traces priced at query time Cost figures are estimates from token counts and public list prices, not an invoice. A backfilled row keeps the price from that backfill. ## Query errors Lemma rejects an invalid query with HTTP 400 and a `detail` message: | `detail` | Cause | | - | - | | `Unknown field: name` | The field isn’t on that event | | `p95 is not valid for name` | The field is a string, or its `aggs` lacks the aggregation | | `count does not take a field` | `count` has a `field` | | `trace_cost is not valid on tool.called` | The cost measure belongs to another event | | `Only one cost measure is allowed` | The query has two cost measures | | `generation_cost requires grouping by model` | No `model` group | | `Invalid alias: name` | The alias breaks the alias pattern | | `Duplicate alias: name` | Two groups or measures share an alias | | `Unknown order field: name` | `order_by` names an unknown or joined alias | | `start and endExclusive must be second-aligned UTC` | A timestamp has milliseconds | | `Window exceeds 90 days` | The window is longer than `max_window_days` | | `in filters accept at most 20 values` | An `in` list is over `max_in_values` | | `Join must use a different event` | The join reads the primary event | An unknown project, or a key limited to another project, returns HTTP 404. A query that runs past 15s fails with a timeout. ## Preset views A preset view returns one precomputed dashboard payload, so its numbers match the [Analytics](/platform/analytics) page. Over REST, pass the view as `view`. On MCP, call the view’s tool. | View | MCP tool | Window | Returns | | - | - | - | - | | `window` | `get_project_analytics_window` | Required | Latency, totals, coverage, estimated cost | | `base` | `get_project_analytics_base` | Required | Series of traces, errors, tokens, cost, and cost by model | | `agent_scorecard` | `get_project_agent_scorecard` | Required | One row per agent | | `tool_aggregates` | `get_project_tool_aggregates` | Required | One row per tool, tool volume, error concentration | | `model_scorecard` | `get_project_model_scorecard` | `start`, optional end | One row per model | | `roi` | `get_project_roi` | None | Issue series and lifetime counts | Preset parameters are query parameters: * `start`: inclusive, ISO 8601, UTC, whole second * `endExclusive`: exclusive end, required where the table says so. REST also accepts `end_exclusive` * `end`: `model_scorecard` only, in place of `endExclusive` * `granularity`: `day` (default), `week`, or `month` The older per-view paths, such as `/projects/{project_id}/agent-scorecard`, are deprecated aliases for `?view=`. ### Preset responses Each view returns these keys: * `window`: `totals` (`traces`, `p50`, `p95`, `error_rate`, `failed_runs`, `root_status_known`, `root_absent`), `latency` (`granularity`, `points` of `date`, `p50`, `p95`), `coverage`, and `cost` * `base`: `traces`, `errors` (`span_errors`, `failed_runs`, `root_status_known`, `root_absent`, `traces` per date), `tokens`, `cost`, `cost_by_model`, and `cost_by_model_coverage` * `agent_scorecard`: `rows` with `agent`, `traces`, `p50`, `p95`, `error_rate`, `root_status_known`, `tools_used`, `tokens`, `cost`, and lower-bound flags * `tool_aggregates`: `comparison` rows (`tool`, `calls`, `errors`, `error_rate`, `p50`, `p95`, `traces`, `output_tokens`), `volume`, `failing_volume`, and `error_concentration` * `model_scorecard`: `rows` with `model`, `traces`, `tokens`, `input_tokens`, `output_tokens`, `avg_speed`, `priced`, and `estimated_cost_usd`, plus overall cost fields and `cost_attribution_coverage` * `roi`: `state` (`connect`, `waiting`, or `populated`), lifetime `counts`, and daily `base` series for `caught`, `resolved`, `dismissed`, `traces`, `errors`, `artifact_iterations`, and `resolution_days` `coverage` in the `window` view reports, per capability (`root_status`, `root_duration`, `agents`, `tokens`, `model`, `provider`, `cost_basis`, `tools`), whether the project sends it at all and how many window traces have it (`observed` of `eligible`). A missing capability explains an empty metric; see [Analytics telemetry](/reference/analytics-telemetry). A per-model `estimated_cost_usd` in `model_scorecard` can be null when part of the window was priced at write time. The overall figure stays complete; `cost_attribution_coverage` is `all` when the rows add up to it. ## Related pages Build and send a query, with worked examples. SDK fields the aggregates are computed from. Dashboard charts the preset views back. Connect a coding agent to these tools. # Analytics telemetry Source: https://docs.uselemma.ai/reference/analytics-telemetry Required SDK attributes for Analytics widgets, coverage semantics, and upgrade behavior Lemma Analytics widgets consume source telemetry from ingested traces. The SDK emits that telemetry; the backend computes windowed aggregates. The SDK must **not** derive user-facing totals or estimated cost. For the full ingest contract, see [Trace contract](/reference/trace-contract). This page lists the attributes Analytics depends on and how missing vs zero values are interpreted. ## Required signals | Analytics need | Source fields | | - | - | | Model identity | Generation `model`, attributes `llm.model_name` / `gen_ai.request.model` | | Provider identity | `llm.provider`, `gen_ai.system` (from `llmProvider` / `llmSystem`) | | Input / output tokens | Top-level `usage.input_tokens` / `usage.output_tokens`, plus `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` and `llm.token_count.prompt` / `llm.token_count.completion` | | Cached tokens | `usage.cache_read_input_tokens`, `usage.cache_creation_input_tokens` → `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_creation.input_tokens` | | Reasoning tokens | `usage.reasoning_output_tokens` → `gen_ai.usage.reasoning.output_tokens` | | Root latency | Trace `started_at` / `ended_at` / `duration_ms` | | Generation speed | Generation span `duration_ms` (kept distinct from root latency) | | Run / tool errors | Normalized `status: "ERROR"` and `error` on the root or tool span | | Tool identity | Tool span `name` / `tool_name`, nested under the same `trace.id` | | Coverage provenance | `lemma.sdk.language`, `lemma.sdk.integration` | ## Coverage semantics (omit vs zero) | Situation | Emit | | - | - | | Provider did not return usage | **Omit** `usage` and usage attributes entirely | | Provider returned zeros | **Emit** `0` | Analytics treats a missing field as “not instrumented / unknown” and an explicit `0` as a healthy zero. Inventing zeros in the SDK would under-report coverage and hide upgrade gaps. Estimated cost is a **backend** derivation from model/provider + token telemetry. Do not emit billed cost from the SDK. ## Provenance attributes Lemma extensions on every span: | Attribute | Values | | - | - | | `lemma.sdk.language` | `typescript` \| `python` | | `lemma.sdk.integration` | `manual` \| `vercel-ai` \| `langchain` \| `openai-agents` \| `mastra` | Use these to attribute sparse coverage to an integration or SDK language without hard-coding provider logic in the Analytics frontend. ## Framework integrations | Integration | Usage extraction | | - | - | | LangChain / LangGraph | `llmOutput.tokenUsage` / `token_usage`, message `usage_metadata` | | OpenAI Agents | `spanData.usage` or `response.usage` when present | | Mastra | `output.usage` / `attributes.usage` when input/output tokens are distinguishable (`totalTokens` alone is omitted) | | Vercel AI SDK | Extracts `usage` when present on finish/step-end events; AI SDK telemetry callbacks do not always include usage | Manual `recordGeneration` / `record_generation` always accepts usage when you have it from the provider response. ## Upgrade behavior (older SDK versions) | Older SDK behavior | Analytics impact | | - | - | | No `usage` object | Token / estimated-cost widgets show as not instrumented for those generations | | `llm.provider` missing | Provider breakdowns fall back to unknown / unsupported | | No `lemma.sdk.*` provenance | Coverage cannot be attributed to a specific integration | | Root or generation duration missing | Latency / generation-speed percentiles skip those points | Upgrading to an SDK that emits usage, provider, duration, and provenance enables Analytics to move from “not instrumented” to populated charts without re-ingesting historical traces. ## Related pages Project-level volume, latency, errors, tools, models, and cost. Full ingest payload and native contract props. How to record model calls with usage. Root latency vs generation duration. # Trace contract Source: https://docs.uselemma.ai/reference/trace-contract What clients send to Lemma, what ingest requires, and which fields unlock product behavior Lemma traces are JSON trees delivered to `POST /traces/ingest`. A **trace** is one agent execution: the root record carries the current input, final output or error, agent name, optional thread/user context, and timing. All other work is **child spans** in `trace.spans[]`, typed as `span`, `generation`, or `tool`, optionally nested via `parent_id`. The SDK builds this payload for you. This page defines what Lemma accepts, what each field means, how ingest normalizes it, and which fields unlock which product behavior. **One agent execution = one trace.** LLM calls, tool calls, retrieval, and app logic are child spans inside that trace, not separate top-level traces. ## Mental model | Concept | What it is | Client field | | - | - | - | | **Trace** | One end-to-end agent execution | The root `trace` object | | **Span** | A unit of work inside the trace | `spans[]` entry with `type: "span"` | | **Generation** | A single LLM call | `spans[]` entry with `type: "generation"` | | **Tool call** | A single tool invocation | `spans[]` entry with `type: "tool"` | The SDK sends a **trace record** plus child spans. At ingest, Lemma creates a synthetic root span (`{traceId}:root`) that carries root-level fields for storage and display. Producers should target the trace record; they do not create the synthetic root themselves. ## Requirement tiers | Tier | Meaning | Examples | | - | - | - | | **Ingest** | Required for a successful `201` response | Valid JSON, `project_id`, `trace.name`, named spans, unique span IDs | | **Semantic** | Needed for Lemma product features | Root input/output-or-error, typed generations and tools, model and payloads, nesting, thread/user | | **High quality** | Ideal instrumentation fidelity | Measured durations, complete span I/O, full per-call message history, causal nesting | Ingest accepts a name-only trace. Semantic fields are not HTTP-required, but without them Lemma can only render timing and a sparse tree. For the pedagogical ideal path, see [Building high-quality traces](/guides/building-high-quality-traces). ## Ingest payload ```json theme={null} { "project_id": "00000000-0000-0000-0000-000000000000", "trace": { "id": "trace-id", "name": "support-agent", "input": "Where is my order #1843?", "output": "It ships tomorrow and arrives Friday.", "thread_id": "conversation-42", "user_id": "user-7", "release": "1.8.3", "started_at": "2026-07-13T20:00:00.000Z", "ended_at": "2026-07-13T20:00:02.410Z", "duration_ms": 2410, "spans": [ { "id": "span-retrieve", "name": "retrieve-context", "type": "span", "input": { "query": "order 1843 status" }, "output": { "count": 2 }, "duration_ms": 412 }, { "id": "span-search", "parent_id": "span-retrieve", "name": "search_docs", "type": "tool", "input": { "query": "order 1843 status" }, "output": [{ "id": "doc-1" }], "duration_ms": 183 }, { "id": "span-answer", "name": "answer", "type": "generation", "model": "gpt-4o", "input": [ { "role": "system", "content": "You are a support agent." }, { "role": "user", "content": "Where is my order #1843?" } ], "output": "It ships tomorrow and arrives Friday.", "duration_ms": 1870, "attributes": { "llm.input_messages.0.message.role": "system", "llm.input_messages.0.message.content": "You are a support agent." } } ] } } ``` ### Envelope and validation | Field | Ingest rule | | - | - | | Body | Must be a JSON object | | `project_id` | Required UUID | | `trace` | Required object | | `trace.name` | Required non-empty string | | `trace.spans[].name` | Required non-empty string for each span | | Span `id` values | Must be unique within the payload when provided | | `status` | Exact `"OK"` or `"ERROR"` on the trace or a span. Ingest ignores any other value | | Authorization | Required; missing credentials return `401` | Successful ingest returns **201**. Invalid shape returns **400**. Missing `trace.id` or span `id` values are generated server-side. Missing or invalid `spans[].type` defaults to `"span"`. Missing `parent_id` attaches the child under the synthetic root. Ingest stores `"OK"` or `"ERROR"` on the trace and on each span. Ingest ignores every other value, including `"UNSET"`, `"ok"`, and `null`, and returns **201**. Ingest stores an omitted status as unset. A non-empty `error` forces `ERROR` on the trace or span that contains it, even when `status` is `"OK"`. When the tracing SDKs build the payload, they omit root `status` on success and send `"ERROR"` on failure. ### Delivery and retries Deliver **one complete trace** when the execution (agent turn) finishes. This is required; patching a trace over time is not currently supported. Treat `ingest` as a single delivery, not an incremental merge API. * **Required:** one complete delivery per execution: root input/output, thread/user, and all child spans in one payload, sent when the turn is done. * **Retries:** re-sending the same payload with the same span IDs is idempotent; spans whose IDs already exist are skipped. * **Not a merge API:** omitted root fields do **not** preserve prior values. There is no field-level upsert, and patching a trace across calls is not supported. * **Processing:** after an idle window Lemma processes the trace once; a later successful re-delivery does not restart `processTrace` / `extractSignals`. Late new span IDs may still append for display. * Do not reuse one `trace.id` across conversation turns. See [Instrumenting multi-turn agents](/guides/instrumenting-multi-turn-agents). * One user turn split across a host and a sandbox is still one delivery. Record a journal locally, `apply`, then `ingest()` once. See [Cross-process turns](/tracing/instrumentation/cross-process-turns). ## Trace record | Field | Role | | - | - | | `id` | Stable identity for the execution; generated if omitted | | `name` | Agent / workflow name used for grouping and filtering | | `input` | Current user turn or execution entry point | | `output` | Final answer on success | | `error` / `status` | Failure message, plus exact `"OK"` or `"ERROR"`. Ingest ignores any other status and returns **201**. A non-empty `error` forces `ERROR`. Ingest stores an omitted status as unset | | `started_at` / `ended_at` / `duration_ms` | Trace timing | | `thread_id` | Groups multi-turn conversation traces | | `user_id` | Per-user slicing | | `release` | Optional deployed app version from the SDK client (`release` / `LEMMA_RELEASE`). Trimmed; empty, over 200 characters, or values with newlines, tabs, or carriage returns are omitted (unreleased) | | `metadata` | Arbitrary JSON attached to the root | | `spans` | Child work | **Semantic guidance:** include root `input` and either `output` or `error`. Without them, the dashboard can show timing but not what the user asked or whether the run succeeded. Callback traces set `output` from the return value unless you override it with `trace.output(...)`. Uncaught exceptions mark the root as failed and still send the trace. ## Child spans Every child is a named object in `trace.spans[]`. | Field | Role | | - | - | | `id` | Stable child identity; generated if omitted | | `parent_id` | Parent span ID; omit to attach under the synthetic root | | `name` | Human-readable operation name | | `type` | `"span"`, `"generation"`, or `"tool"` | | `input` / `output` | Operation arguments and result | | `error` / `status` | Same `"OK"` and `"ERROR"` rule as the trace record. Prefer `error` over inventing an output | | `started_at` / `ended_at` / `duration_ms` | Timing | | `model` | Generation model name | | `usage` | Token counts when the provider supplied them (omit when unknown) | | `attributes` | Flattened contract attributes and escape-hatch keys | | `metadata` | Arbitrary JSON for the child | ### Typed children | Type | Use for | Semantic fields | | - | - | - | | `span` | Retrieval, ranking, planning, app logic | `input` + `output` (or `error`) | | `generation` | One LLM call | `model`, prompt/messages, completion (or `error`), optional `usage` | | `tool` | One tool invocation | tool `name`, arguments, result (or `error`) | Type is a discriminator. Setting `model` alone does not make a generation; set `type: "generation"`. ### Nesting Children nest through `parent_id`. In TypeScript, recording from a span handle sets the parent automatically. In Python, pass `parent_id` explicitly on `start_tool` / `start_generation` / `start_span` when the child belongs under another span. ```text theme={null} support-agent ← trace record ├── retrieve-context ← type: span │ └── search_docs ← type: tool, parent_id = retrieve-context └── answer ← type: generation ``` ## Generations and message history `trace.input` is the **current user turn**. A generation should record the **full ordered message list sent to that LLM call**, including system prompt, prior turns, tool results, and the current user message. Prefer both: * `input`: the prompt payload for that call * `llmInputMessages` / `llm_input_messages`: structured messages flattened into `llm.input_messages.{index}.message.*` `model` is the primary model identifier. Native generation props such as `llmProvider`, `llmInvocationParameters`, and `llmOutputMessages` populate OpenInference-style attributes. Pass `usage` (camelCase in TypeScript, snake\_case kwargs in Python) when the provider returns token counts: see [Token usage](#token-usage). ## Token usage Generations may include a top-level `usage` object. Lemma does **not** invent token counts. Coverage semantics: | Provider behavior | SDK behavior | | - | - | | No usage supplied | Omit `usage` entirely (and omit usage attributes) | | Usage includes zeros | Emit those zeros | Healthy zero ≠ missing instrumentation. Analytics uses absence vs explicit zero to decide whether a metric is unsupported or genuinely empty. ### Wire format (ingest JSON) ```json theme={null} { "usage": { "input_tokens": 1200, "output_tokens": 340, "cache_read_input_tokens": 800, "cache_creation_input_tokens": 50, "reasoning_output_tokens": 120 } } ``` All fields are optional. Only include fields the provider actually returned. ### TypeScript ```typescript theme={null} trace.recordGeneration({ name: "draft-reply", model: "gpt-4o", llmProvider: "openai", usage: { inputTokens: 1200, outputTokens: 340, cacheReadInputTokens: 800, cacheCreationInputTokens: 50, reasoningOutputTokens: 120, }, }); ``` ### Python ```python theme={null} trace.record_generation( name="draft-reply", model="gpt-4o", llm_provider="openai", input_tokens=1200, output_tokens=340, cache_read_input_tokens=800, cache_creation_input_tokens=50, reasoning_output_tokens=120, ) # or: usage={"input_tokens": 1200, "output_tokens": 340, ...} ``` The SDK also flattens usage into span attributes using OTel GenAI names (`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_creation.input_tokens`, `gen_ai.usage.reasoning.output_tokens`) plus OpenInference compatibility keys (`llm.token_count.prompt`, `llm.token_count.completion`). Do **not** derive user-facing aggregates or estimated cost in the SDK: the backend computes windowed Analytics quantities from this source telemetry. ## Tools Tool spans should carry: * `name`: stable tool identifier * `input`: arguments * `output` or `error`: result or failure * optional `toolParameters` / `tool_parameters` for the schema Ingest uses `tool_name` when present, otherwise the span `name`, and normalizes tool I/O into tool-call aliases for display. ## Record failures Record a failure on the exact span, generation, or tool where it happened: * Keep the **input** that led to the failure when it is safe to record. * Pass `error` instead of inventing an output. * End the failed child so timing is preserved. * Rethrow when the whole execution should fail; leave the root successful when the agent recovers. For callback traces, uncaught exceptions automatically fail the root. If you catch at the root boundary, call `trace.fail(error)`. For completed work, pass `error` to `recordSpan` / `recordGeneration` / `recordTool` (or the Python equivalents). For live handles, call `.end({ error })` / `.end(error=...)`. ```typescript theme={null} const tool = trace.startTool({ name: "lookup_order", input: { orderId }, }); try { const order = await lookupOrder(orderId); tool.end({ output: order }); } catch (error) { tool.end({ error }); throw error; } ``` ```python theme={null} tool = trace.start_tool( name="lookup_order", input={"order_id": order_id}, ) try: order = lookup_order(order_id) tool.end(output=order) except Exception as error: tool.end(error=error) raise ``` ## Measure duration Duration resolution order: 1. Explicit `duration_ms` / `durationMs` when provided 2. Otherwise `ended_at - started_at` when both timestamps exist 3. Otherwise, for children with missing duration, Lemma allocates remaining parent time Child duration inference: * Starts from the synthetic root and walks the parent → child tree * Explicit sibling durations claim time first * Siblings with `duration_ms == null` split the remaining parent duration equally * If explicit siblings already exceed the parent duration, omitted siblings receive `0` * The same rule applies recursively when a child becomes a parent Prefer live handles (`startSpan` / `startTool` / `startGeneration`) so `.end(...)` measures real elapsed time. One-shot `record*` helpers omit child duration when you do not pass it. ## Client fields vs ingest normalization Clients send first-class JSON fields. Ingest generates display and analysis aliases. | Client sends | Ingest normalizes to | | - | - | | `trace.input` | `input.value`, `ai.agent.input` | | `trace.output` | `output.value`, `ai.agent.output`, and `ai.response.text` for strings | | `trace.name` | `gen_ai.agent.name`, `ai.agent.name` | | `trace.thread_id` | `lemma.thread_id` | | `trace.user_id` | `user.id`, `enduser.id` | | `type: "generation"` | `kind: "generation"`, `openinference.span.kind: "llm"` | | `type: "tool"` | `kind: "tool"`, `openinference.span.kind: "tool"`, `tool.name` | | `model` | `gen_ai.request.model`, `ai.model.id`, `model_name` | | generation `input` / `output` | `input.value` / `output.value` | | tool `input` / `output` | `input.value`, `output.value`, `ai.toolCall.args`, `ai.toolCall.result` | | `error` | `error.message`. A non-empty value forces `ERROR` | | `status` | `OK` or `ERROR`; otherwise unset | | native props / `attributes` | Flattened OpenInference-style attribute keys | Do not hand-build the normalized aliases unless you are bypassing the SDK. Prefer native SDK props and first-class fields. ## SDK mapping | Contract field | SDK API | Client field | | - | - | - | | Trace input | `lemma.trace({ input })` | `trace.input` | | Trace output | Callback return value or `trace.output(output)` | `trace.output` | | Trace duration | Measured automatically, `durationMs`, or `trace.end({ durationMs })` | `trace.duration_ms` | | Agent name | `lemma.trace({ name })` | `trace.name` | | Thread id | `threadId` | `trace.thread_id` | | User id | `userId` | `trace.user_id` | | Generation | `trace.recordGeneration(...)` / `startGeneration(...)` | `type: "generation"` | | Generation model | `model` | `model` | | Generation usage | `usage: { inputTokens, ... }` | `usage` (snake\_case on the wire) | | Generation content | `input`, `output`, `llmInputMessages` | `input`, `output`, attributes | | Tool | `trace.recordTool(...)` / `startTool(...)` | `type: "tool"` | | Tool args / result | `input`, `output` | `input`, `output` | | Child duration | `durationMs` or live-handle `.end()` | `duration_ms` | | Nesting | Span-handle record/start methods or `parentId` | `parent_id` | | Error | thrown error, `trace.fail(error)`, or child `error` | `error`, `status` | | Contract field | SDK API | Client field | | - | - | - | | Trace input | `lemma.trace(..., input=...)` | `trace.input` | | Trace output | Callback return value or `trace.output(output)` | `trace.output` | | Trace duration | Measured automatically or `duration_ms` | `trace.duration_ms` | | Agent name | `lemma.trace("name", ...)` | `trace.name` | | Thread id | `thread_id` | `trace.thread_id` | | User id | `user_id` | `trace.user_id` | | Generation | `trace.record_generation(...)` / `start_generation(...)` | `type: "generation"` | | Generation model | `model` | `model` | | Generation usage | `usage={...}` or `input_tokens=` / `output_tokens=` kwargs | `usage` | | Generation content | `input`, `output`, `llm_input_messages` | `input`, `output`, attributes | | Tool | `trace.record_tool(...)` / `start_tool(...)` | `type: "tool"` | | Tool args / result | `input`, `output` | `input`, `output` | | Child duration | `duration_ms` or live-handle `.end()` | `duration_ms` | | Nesting | `parent_id` on `start_*` helpers | `parent_id` | | Error | raised exception, `trace.fail(error)`, or child `error` | `error`, `status` | ## Native contract props Use native props instead of hand-building flattened attribute names. In Python, use snake\_case. | SDK prop | Use on | Attribute key | | - | - | - | | `inputMimeType` | Span, generation, tool | `input.mime_type` | | `outputMimeType` | Span, generation, tool | `output.mime_type` | | SDK prop | Attribute key | | - | - | | `llmModelName` | `llm.model_name` | | `llmProvider` | `llm.provider` (also sets `gen_ai.system` when `llmSystem` is unset) | | `llmSystem` | `llm.system`, `gen_ai.system` | | `usage` | `gen_ai.usage.*`, `llm.token_count.prompt` / `completion` | | `llmInvocationParameters` | `llm.invocation_parameters` | | `llmInputMessages` | `llm.input_messages.{index}.message.*` | | `llmOutputMessages` | `llm.output_messages.{index}.message.*` | | `llmTools` | `llm.tools` | | `llmPromptTemplate` | `llm.prompt_template.template` | | `llmPromptTemplateVariables` | `llm.prompt_template.variables` | | `llmPromptTemplateVersion` | `llm.prompt_template.version` | | SDK prop | Attribute key | | - | - | | `toolDescription` | `tool.description` | | `toolParameters` | `tool.parameters` | | SDK prop | Attribute key | | - | - | | `embeddingModelName` | `embedding.model_name` | | `embeddingInvocationParameters` | `embedding.invocation_parameters` | | `embeddingEmbeddings` | `embedding.embeddings` | | SDK prop | Attribute key | | - | - | | `rerankerModelName` | `reranker.model_name` | | `rerankerInputDocuments` | `reranker.input_documents.{index}.document.*` | | `rerankerOutputDocuments` | `reranker.output_documents.{index}.document.*` | `model` also populates `llm.model_name` for generation records. Object and array attribute values are serialized where Lemma expects string-valued attributes. For fields without a native prop, pass raw `attributes`. ## Provenance attributes (Lemma extensions) Every emitted span includes SDK provenance so Analytics can attribute coverage gaps to an integration or language without provider-specific frontend logic: | Attribute | Values | Set by | | - | - | - | | `lemma.sdk.language` | `typescript` \| `python` | Core client | | `lemma.sdk.integration` | `manual` \| `vercel-ai` \| `langchain` \| `openai-agents` \| `mastra` | Core defaults to `manual`; integrations override | These are Lemma-specific extensions (not OpenTelemetry GenAI canonical names). ## Semantic requirements | Field | Why it matters | | - | - | | One shared `trace.id` for the whole execution | Otherwise one run scatters across unrelated traces | | Root `input` | Shows what the user asked | | Root `output` or `error` | Distinguishes success from failure | | Stable `name` | Groups and filters by agent/workflow | | Typed generations with model + content | Makes model calls inspectable | | Provider identity (`llm.provider` / `gen_ai.system`) | Model, token, and estimated-cost breakdowns | | Token `usage` when the provider supplies it | Token and estimated-cost Analytics; omit vs zero coverage | | Typed tools with args + result/error | Makes tool behavior inspectable | | Correct `parent_id` nesting | Preserves causality in the tree | | `thread_id` for multi-turn work | Groups conversation turns | | `user_id` when available | Enables per-user slicing | | Measured `duration_ms` on root and generations | Root latency vs generation speed as distinct quantities | ## Issue detection Complete, correctly typed trees improve classification and automated issue extraction. Root I/O, model fields, and tool payloads are **not** general ingest gates. Some environments may further restrict issue extraction to specific framework shapes; treat that as an environment policy, not as the base SDK contract. ## Redact before ingest Redact secrets, credentials, and sensitive user data before tracing them. Framework integrations always record inputs, outputs, and error messages: without them a trace cannot show what a run consumed, produced, or why it failed. ## Related pages Required attributes for Analytics widgets and upgrade behavior. Bad → better → best examples for the ideal instrumentation path. Build a complete Lemma trace one piece at a time. Capture model, full messages, and completion. Diagnose missing spans, flat trees, and blank I/O. # SDK vocabulary Source: https://docs.uselemma.ai/tracing/instrumentation/concepts Core SDK concepts and terminology in Lemma Five nouns describe everything the software development kit (SDK) records. They’re the same words used in the dashboard and across these docs. **One agent execution = one trace.** The root is the trace record; LLM calls, tool calls, and app logic are child spans inside it. ## How a trace is structured A single agent execution produces a tree. The root is the trace record; everything inside the execution nests under it as child spans.
support-agent
trace root
input · output · timing
draft-reply
generation · model · content
search\_docs
tool call · args · result
## Trace A **trace** is a single end-to-end execution of your agent, from the incoming request to the final response. It carries the input, the output (or error), timing, and the tree of spans inside it. Each trace has a unique trace ID. If every model or tool call shows up as its own trace, the child work ran without the root trace handle or trace IDs. Create the root trace first, then record child spans through that trace or an explicit parent span handle. See [Agent traces](/tracing/instrumentation/traces). ## Span A **span** is one unit of work inside a trace: retrieval, ranking, parsing, a database query, or any app operation. Spans nest, so a parent span can contain child spans, forming the execution tree. See [Spans](/tracing/instrumentation/spans). ## Generation A **generation** is a span that represents a single LLM call. Typing a span as a generation lets Lemma read its model, prompt, completion, and timing. See [Generations](/tracing/instrumentation/generations). ## Tool call A **tool call** is a span that represents a single tool invocation: its name, arguments, and result. Tool execution usually happens in your code, so it must be recorded explicitly to be visible. See [Tool calls](/tracing/instrumentation/tool-calls). ## Thread A **thread** groups the traces of a multi-turn conversation. Each turn is its own trace; giving related turns the same thread ID (`threadId` / `thread_id`) groups them into one conversation. See [Threads & context](/tracing/instrumentation/context). Every trace also belongs to a **project**, the container that holds your API key and project ID. See [Platform concepts](/platform/concepts) for how projects, agents, issues, and artifacts fit together on the dashboard side. ## Next steps Issues, artifacts, and agents once traces arrive. The exact shape Lemma reads. Install the SDK and send your first complete agent trace. Search and inspect ready executions. # Threads & context Source: https://docs.uselemma.ai/tracing/instrumentation/context Group multi-turn conversations, attach users, and add custom metadata Set `threadId`, `userId`, and custom metadata when you open a trace so Lemma can group turns, filter by person, and search extra attributes. ## Threads When one user has a back-and-forth with your agent, each turn is its own trace. Give related turns the same `threadId` so Lemma groups them into one conversation. ```typescript theme={null} await lemma.trace( { name: "support-agent", input: userMessage, threadId: conversationId, }, async (trace) => { return runAgent(userMessage, trace); }, ); ``` ```python theme={null} async def run(trace): return await run_agent(user_message, trace) await lemma.async_trace( "support-agent", run, input=user_message, thread_id=conversation_id, ) ``` ## Attach a user id Set `userId` / `user_id` on the trace so Lemma can filter conversations by person. ```typescript theme={null} await lemma.trace( { name: "support-agent", input: userMessage, threadId: conversationId, userId: user.id, }, async (trace) => { return runAgent(userMessage, trace); }, ); ``` ```python theme={null} async def run(trace): return await run_agent(user_message, trace) await lemma.async_trace( "support-agent", run, input=user_message, thread_id=conversation_id, user_id=user.id, ) ``` ## Attach custom metadata Attach any custom keys on the trace or a child span. Keys such as `firm.slug` and `plan` work; ingest copies each key onto that span as an attribute, so you can filter with `attr.:`. Use dotted string keys, not nested objects. `metadata: { "firm.slug": "acme" }` becomes the attribute `firm.slug`. `metadata: { firm: { slug: "acme" } }` becomes one object-valued `firm` attribute and will not filter cleanly. ```typescript theme={null} await lemma.trace( { name: "support-agent", input: userMessage, metadata: { "firm.slug": "acme", plan: "enterprise", }, }, async (trace) => runAgent(userMessage, trace), ); ``` ```python theme={null} lemma.trace( "support-agent", lambda trace: run_agent(user_message, trace), input=user_message, metadata={"firm.slug": "acme", "plan": "enterprise"}, ) ``` Filter those traces with `attr.firm.slug:acme`. To keep staging data out of production, use a separate Lemma project. The SDK ingest path always stores `service_name: "lemma-sdk"`; set a custom service name through the OTLP resource `service.name`. ## Context fields | Context | SDK option | Client field | Ingest normalizes to | | - | - | - | - | | Agent name | `name` | `trace.name` | `gen_ai.agent.name`, `ai.agent.name` | | Thread / conversation | `threadId` / `thread_id` | `trace.thread_id` | `lemma.thread_id` | | User | `userId` / `user_id` | `trace.user_id` | `user.id`, `enduser.id` | | Metadata | `metadata` | `trace.metadata` / `spans[].metadata` | Each key becomes a span attribute | See the [trace contract](/reference/trace-contract) for how each field is sent and normalized. ## Next steps What clients send, what ingest requires, and what Lemma normalizes. The ideal instrumentation path, with bad → better → best examples. # Cross-process turns Source: https://docs.uselemma.ai/tracing/instrumentation/cross-process-turns Record a host and sandbox as one Lemma trace When a user turn spans a host and a sandbox such as E2B, record one Lemma [trace](/tracing/instrumentation/traces). The host owns the root and the API key. The child records a journal (local span, tool, and generation records). You apply that journal on the host and `ingest()` once. A **context token** is the payload the host exports. It includes `traceId`, `parentSpanId`, `threadId`, `userId`, and `startedAt`. The helpers are `startTurn` / `attachTurn` / `apply` (Python `start_turn` / `attach_turn` / `apply`). A [thread id](/tracing/instrumentation/context) groups successive conversation turns. It does not join a host and a sandbox into one turn. For later turns in other processes, see [Instrumenting multi-turn agents](/guides/instrumenting-multi-turn-agents). ## Host, child, and coordinator The work splits across three roles: * **Host**: process that holds the Lemma client and API key * **Child**: sandbox or worker that records the journal * **Coordinator**: process that applies the journal and sends (the host, unless another process holds the journal) The host starts the root, exports the token, and calls `ingest()`. The child never constructs `Lemma` and never calls `/traces/ingest`. ## Record the host and child as one turn Export a token on the host, attach in the child, apply the journal, then end once. Call `startTurn` / `start_turn`, start a sandbox span, and export with that span as `parentSpanId`. In a sandbox, pass the token as env (for example `LEMMA_TURN`). Call `attachTurn` / `attach_turn` with the token. Record generations, tools, and spans on the local handle. Emit `local.records()` on the app’s existing event channel. Do not set `LEMMA_API_KEY` in the child. On the host, `apply` the journal, end the sandbox span, then `turn.end()` or `ingest()`. Do not call `/traces/ingest` from the child. Lemma ingest happens once at the end. This example runs all three steps in one process so you can copy it. Split `export` / `attachTurn` / `apply` across host and child in production. ```typescript {4-9,11,17,19-20} theme={null} import { Lemma, attachTurn, startTurn } from "@uselemma/tracing"; const lemma = new Lemma(); const turn = startTurn(lemma, { name: "agent-turn", input: userMessage, threadId: conversationId, }); const sandbox = turn.startSpan({ name: "e2b-sandbox" }); const local = attachTurn(turn.export({ parentSpanId: sandbox.id })); local.recordTool({ name: "lookup_order", input: { orderId: "1843" }, output: { status: "shipped" }, }); turn.apply(local.records()); sandbox.end(); await turn.end({ output: "It ships tomorrow." }); ``` ```python {4-9,11,17,19-20} theme={null} from uselemma_tracing import Lemma, attach_turn lemma = Lemma() turn = lemma.start_turn( "agent-turn", input=user_message, thread_id=conversation_id, ) sandbox = turn.start_span(name="e2b-sandbox") local = attach_turn(turn.export(parent_span_id=sandbox.id)) local.record_tool( name="lookup_order", input={"orderId": "1843"}, output={"status": "shipped"}, ) turn.apply(local.records()) sandbox.end() turn.end(output="It ships tomorrow.") ``` `apply` accepts any of: * **Full journal**: `local.records()` * **JSON string**: a serialized journal * **One record**: a single event * **Array of records**: streamed events `apply` only updates the in-memory tree on the host or coordinator. It does not call `/traces/ingest`. Stream journal events into `apply` as they arrive, or apply the full journal once. Send to Lemma once with `turn.end()` or `ingest()` after the sandbox finishes. ## Assemble a trace from a stored journal When you have a token and journal and no live turn handle, build a `TraceContext` and `ingest()` once. The journal can be a file or a queued payload: ```typescript theme={null} import { Lemma, assembleTurn } from "@uselemma/tracing"; const lemma = new Lemma(); const { context, startedAt } = assembleTurn(token, journal, { input: userMessage, output: answer, }); await lemma.ingest(context, { startedAt }); ``` ```python theme={null} from uselemma_tracing import Lemma, assemble_turn lemma = Lemma() context, started_at = assemble_turn( token, journal, input=user_message, output=answer ) lemma.ingest(context, started_at=started_at) ``` Re-applying the same journal is idempotent. Span IDs are stable, so a retried assemble plus ingest does not duplicate children. ## Recover when the sandbox exits uncleanly If the sandbox dies before a complete journal, apply whatever records arrived. End the sandbox span as `ERROR`, leave incomplete tools as they are, then `fail` and `end()` so one payload still goes out. ```typescript theme={null} turn.apply(journal); sandbox.end({ status: "ERROR", error: "sandbox killed" }); turn.fail("sandbox exited uncleanly"); await turn.end(); ``` ```python theme={null} turn.apply(journal) sandbox.end(status="ERROR", error="sandbox killed") turn.fail("sandbox exited uncleanly") turn.end() ``` `turn.end()` is **strict**: a 4xx/5xx from Lemma throws so you can retry the same payload. Automatic `lemma.trace()` delivery fails open. ## Helpers for each role | Role | TypeScript | Python | | - | - | - | | Start the host root | `startTurn(lemma, ...)` | `lemma.start_turn(...)` / `start_turn(lemma, ...)` | | Export the child token | `turn.export({ parentSpanId })` | `turn.export(parent_span_id=...)` | | Record in the child | `attachTurn(token)` | `attach_turn(token)` / `Lemma.attach(token)` | | Collect journal records | `local.records()` | `local.records()` | | Apply the journal | `turn.apply(journal)` / `applyTurnJournal(context, journal)` | `turn.apply(journal)` / `apply_turn_journal(context, journal)` | | Assemble without a live handle | `assembleTurn(token, journal)` | `assemble_turn(token, journal)` | | Send once | `await turn.end(...)` or `lemma.ingest(...)` | `turn.end(...)` or `lemma.ingest(...)` | Journal fields are camelCase in both languages (`traceId`, `parentSpanId`, `parentId`, `startedAt`, …). A TypeScript host can apply a Python child’s journal. A Python host can apply a TypeScript child’s journal. ## Related pages Callback traces, handles, and `ingest()` in one process. New root per user turn, same `threadId` for the conversation. `threadId` and `userId` on the root. Two roots, missing children, sparse ingest. # Generations Source: https://docs.uselemma.ai/tracing/instrumentation/generations Capture LLM calls with model, prompt, completion, and timing A **generation** is a single LLM call inside a trace. Record it with `trace.recordGeneration()` / `trace.record_generation()` after the model call completes. ## Record a generation ```typescript theme={null} await lemma.trace({ name: "support-agent", input: userMessage }, async (trace) => { const response = await callModel(messages); trace.recordGeneration({ name: "draft-reply", input: messages, output: response.text, model: "gpt-4o", llmProvider: "openai", llmInputMessages: messages, llmInvocationParameters: { temperature: 0.2 }, usage: { inputTokens: response.usage?.inputTokens, outputTokens: response.usage?.outputTokens, }, }); return response.text; }); ``` ```python theme={null} def run(trace): response = call_model(messages) trace.record_generation( name="draft-reply", input=messages, output=response.text, model="gpt-4o", llm_provider="openai", llm_input_messages=messages, llm_invocation_parameters={"temperature": 0.2}, input_tokens=getattr(response, "input_tokens", None), output_tokens=getattr(response, "output_tokens", None), ) return response.text lemma.trace("support-agent", run, input=user_message) ``` Pass the **full ordered message list** for that call as both `input` and `llmInputMessages` / `llm_input_messages`. The root trace input is only the current user turn; the generation should include system, prior turns, tool results, and the current user message. When the provider returns token counts, pass them as `usage` (TypeScript camelCase) or `input_tokens` / `output_tokens` / … kwargs (Python). **Omit** usage when the provider did not supply it: do not invent zeros. Explicit zeros are fine and mean “provider reported zero.” See [Token usage](/reference/trace-contract#token-usage) and [Analytics telemetry](/reference/analytics-telemetry). ## Measure a live generation Use `startGeneration()` / `start_generation()` when you want a handle before the model call finishes. End it only after the provider returns or fails. ```typescript theme={null} const generation = trace.startGeneration({ name: "draft-reply", input: messages, model: "gpt-4o", }); try { const response = await callModel(messages); generation.end({ output: response.text, durationMs: response.durationMs, }); return response.text; } catch (error) { generation.end({ error }); throw error; } ``` ```python theme={null} generation = trace.start_generation( name="draft-reply", input=messages, model="gpt-4o", ) try: response = call_model(messages) generation.end( output=response.text, duration_ms=response.duration_ms, ) return response.text except Exception as error: generation.end(error=error) raise ``` ## What the SDK records | Field | Client sends | Ingest normalizes to | | - | - | - | | Generation kind | `type: "generation"` | `openinference.span.kind: "llm"` | | Prompt / input | `input`, `llmInputMessages` | `input.value`, `llm.input_messages.*` | | Completion / output | `output` | `output.value` | | Model | `model` | `gen_ai.request.model`, `ai.model.id` | | Provider | `llmProvider` | `llm.provider`, `gen_ai.system` | | Token usage | `usage` / `input_tokens`… (omit when unknown) | `usage.*`, `gen_ai.usage.*`, `llm.token_count.*` | | Duration | `durationMs` / `duration_ms` | Generation speed (distinct from root latency) | | Contract props | Native props such as `llmInvocationParameters` and `llmPromptTemplate`; raw `attributes` for escape-hatch keys | Flattened OpenInference-style attributes | See [Native contract props](/reference/trace-contract#native-contract-props) and [Client fields vs ingest normalization](/reference/trace-contract#client-fields-vs-ingest-normalization). ## Record generation failures If a model call fails, record a failed generation before rethrowing: ```typescript theme={null} try { const response = await callModel(messages); trace.recordGeneration({ name: "draft-reply", input: messages, output: response.text }); return response.text; } catch (error) { trace.recordGeneration({ name: "draft-reply", input: messages, status: "ERROR", error, }); throw error; } ``` ```python theme={null} try: response = call_model(messages) trace.record_generation( name="draft-reply", input=messages, output=response.text, ) return response.text except Exception as error: trace.record_generation( name="draft-reply", input=messages, status="ERROR", error=error, ) raise ``` ## Next steps Record tool arguments and results. Trace retrieval, ranking, and app logic. # Instrument an agent Source: https://docs.uselemma.ai/tracing/instrumentation/instrument-an-agent Build a complete Lemma trace one piece at a time This page starts from an untraced support agent and adds Lemma software development kit (SDK) calls one piece at a time. Complete the [Quickstart](/tracing/instrumentation/setup) first so the SDK is installed and a Lemma client can read your credentials. The per-primitive pages go deeper on each piece: [Agent traces](/tracing/instrumentation/traces), [Generations](/tracing/instrumentation/generations), [Tool calls](/tracing/instrumentation/tool-calls), [Spans](/tracing/instrumentation/spans), and [Threads and context](/tracing/instrumentation/context). One agent execution is one `lemma.trace()` call. Record model calls, tool calls, and app work inside that trace. If that execution spans a host and a sandbox, follow [Cross-process turns](/tracing/instrumentation/cross-process-turns) instead of starting a second root. ## What you will build ```text theme={null} support-agent <- trace root (input, output, agent name, thread, user) ├── retrieve-context <- span (retrieval work) ├── search_docs <- tool call (query, docs) ├── lookup_order <- tool call (user ID, order) └── answer <- generation (model, prompt, completion) ``` ## Before you start The examples assume you already have an agent shaped like this: ```typescript theme={null} async function handleSupportRequest(req: { message: string; conversationId: string; userId: string; }): Promise { const docs = await searchDocs(req.message); const order = await lookupOrder(req.userId); const messages = buildPrompt(req.message, docs, order); const response = await callModel(messages); return response.text; } ``` ```python theme={null} def handle_support_request(message: str, conversation_id: str, user_id: str) -> str: docs = search_docs(message) order = lookup_order(user_id) messages = build_prompt(message, docs, order) response = call_model(messages) return response.text ``` ## Instrument the agent Start with the root boundary. The returned value becomes the trace output. ```typescript theme={null} import { Lemma } from "@uselemma/tracing"; const lemma = new Lemma(); return lemma.trace( { name: "support-agent", input: req.message, threadId: req.conversationId, userId: req.userId, }, async (trace) => { const docs = await searchDocs(req.message); const order = await lookupOrder(req.userId); const messages = buildPrompt(req.message, docs, order); const response = await callModel(messages); return response.text; }, ); ``` ```python theme={null} from uselemma_tracing import Lemma lemma = Lemma() def run(trace): docs = search_docs(message) order = lookup_order(user_id) messages = build_prompt(message, docs, order) response = call_model(messages) return response.text return lemma.trace( "support-agent", run, input=message, thread_id=conversation_id, user_id=user_id, ) ``` Use a span for app work that is not itself a model call or tool call. ```typescript theme={null} const retrieve = trace.startSpan({ name: "retrieve-context", input: { query: req.message }, }); const docs = await searchDocs(req.message); retrieve.end({ output: { count: docs.length }, durationMs: 120, }); ``` ```python theme={null} retrieve = trace.start_span( name="retrieve-context", input={"query": message}, ) docs = search_docs(message) retrieve.end( output={"count": len(docs)}, duration_ms=120, ) ``` Pass `durationMs` / `duration_ms` when you already measured the retrieval time. If you omit it, the SDK records timestamps and Lemma can allocate missing child durations from the parent. Record tools after they return so you can include both arguments and results. ```typescript theme={null} trace.recordTool({ name: "search_docs", input: { query: req.message }, output: docs, durationMs: 45, toolParameters: { query: "string" }, }); const order = await lookupOrder(req.userId); trace.recordTool({ name: "lookup_order", input: { userId: req.userId }, output: order, durationMs: 30, }); ``` ```python theme={null} trace.record_tool( name="search_docs", input={"query": message}, output=docs, duration_ms=45, tool_parameters={"query": "string"}, ) order = lookup_order(user_id) trace.record_tool( name="lookup_order", input={"user_id": user_id}, output=order, duration_ms=30, ) ``` Use stable, boring names like `search_docs`, `lookup_order`, and `rerank_results`. They make traces easier to scan. Generations are for LLM calls. Include the model, prompt/input, output, and timing when available. ```typescript theme={null} const messages = buildPrompt(req.message, docs, order); const response = await callModel(messages); trace.recordGeneration({ name: "answer", model: response.model, input: messages, output: response.text, durationMs: response.durationMs, }); return response.text; ``` ```python theme={null} messages = build_prompt(message, docs, order) response = call_model(messages) trace.record_generation( name="answer", model=response.model, input=messages, output=response.text, duration_ms=response.duration_ms, ) return response.text ``` Callback traces are measured automatically when the callback returns. For trace handles, record root duration when you call `trace.end(...)`. ```typescript theme={null} const trace = lemma.trace({ name: "support-agent", input: req.message }); const response = await runAgent(trace); await trace.end({ output: response.text, durationMs: totalDurationMs, }); ``` ```python theme={null} # Callback traces are measured automatically. # Pass duration_ms only when you already have an external measurement. return lemma.trace( "support-agent", run, input=message, duration_ms=total_duration_ms, ) ``` For open handles, leave duration blank until the work is actually finished. The SDK cannot know the final elapsed time until `span.end(...)`, `trace.end(...)`, or the callback returns. `lemma.trace()` records uncaught errors on the root trace. If you catch a tool or model error before re-raising, record that child failure too. ```typescript theme={null} try { const order = await lookupOrder(req.userId); trace.recordTool({ name: "lookup_order", input: { userId: req.userId }, output: order, }); } catch (error) { trace.recordTool({ name: "lookup_order", input: { userId: req.userId }, error, }); throw error; } ``` ```python theme={null} try: order = lookup_order(user_id) trace.record_tool( name="lookup_order", input={"user_id": user_id}, output=order, ) except Exception as error: trace.record_tool( name="lookup_order", input={"user_id": user_id}, error=error, ) raise ``` ## Go deeper Use these pages for field-level detail and framework-specific tracing. The exact fields and native SDK props Lemma reads. The same docs chat agent, instrumented once per stack. Record AI SDK model calls and tool executions automatically. Fix missing traces, empty outputs, and shape issues. # Send your first trace Source: https://docs.uselemma.ai/tracing/instrumentation/setup Install the Lemma SDK and send your first complete agent trace Send one complete agent trace to Lemma. The software development kit (SDK) sends JSON to the Lemma API. One agent execution is one [trace](/tracing/instrumentation/concepts). Record model calls, tool calls, retrieval, and app logic inside that trace. ## Set credentials Find these in [Lemma project settings](https://platform.uselemma.ai), or in the project get-started flow. ```bash theme={null} export LEMMA_API_KEY="lma_your_api_key_here" export LEMMA_PROJECT_ID="proj_your_project_id_here" export LEMMA_RELEASE="1.8.3" # optional ``` Keep `LEMMA_API_KEY` **server-side**. Never commit it. Never put it in browser or client-side code. The SDK sends to `https://api.uselemma.ai` by default. Pass `baseUrl` / `base_url` only for a custom endpoint. ## What a ready trace is Creating an API key, adding an SDK dependency, or seeing an ingest attempt is not proof of observability. Observability is proven when a real production-like path produces a **ready** trace you can open in the dashboard: inspectable, with a root, nested work, and input and output you recognize. A key, an SDK install, or a failed or empty ingest is not a ready trace. ## What you are building You send one support-agent trace with a nested generation and tool call. ```text theme={null} support-agent <- trace root (input, output) ├── draft-reply <- generation (model, content) └── search_docs <- tool call (args, result) ``` ## Instrument the agent Pick one path. Framework adapters live on the integration pages: [Vercel AI SDK](/integrations/vercel-ai), [OpenAI Agents](/integrations/openai-agents), [LangChain](/integrations/langchain), and [LangGraph](/integrations/langgraph). Use a coding agent to instrument the production agent. In the [Lemma dashboard](https://platform.uselemma.ai), select or create the organization and project, then open that project’s get-started flow. ```bash theme={null} npx skills add uselemma/lemma --skill "lemma-tracing" ``` Ask the coding agent to use that skill to instrument the production agent. Run a production-like request through the instrumented agent. Install the SDK and record one generation and one tool call inside a trace callback. ```bash theme={null} npm install @uselemma/tracing ``` ```bash theme={null} pip install uselemma-tracing ``` Create a client, open a trace, then record one generation and one tool call inside the trace callback. ```typescript theme={null} import { Lemma } from "@uselemma/tracing"; const lemma = new Lemma(); const answer = await lemma.trace( { name: "support-agent", input: userMessage, threadId: conversationId, userId: user.id, }, async (trace) => { const docs = await searchDocs(userMessage); trace.recordTool({ name: "search_docs", input: { query: userMessage }, output: docs, }); const response = await callModel(userMessage, docs); trace.recordGeneration({ name: "draft-reply", input: response.messages, output: response.text, model: "gpt-4o", }); return response.text; }, ); ``` ```python theme={null} from uselemma_tracing import Lemma lemma = Lemma() def run(trace): docs = search_docs(user_message) trace.record_tool( name="search_docs", input={"query": user_message}, output=docs, ) response = call_model(user_message, docs) trace.record_generation( name="draft-reply", input=response.messages, output=response.text, model="gpt-4o", ) return response.text answer = lemma.trace( "support-agent", run, input=user_message, thread_id=conversation_id, user_id=user.id, ) ``` ## Verify a ready trace Open **[Traces](/platform/traces)** in the [Lemma dashboard](https://platform.uselemma.ai). You should see: * One trace for the whole execution * The root trace with input and output * A nested generation with model metadata * A nested tool call with arguments and result After the trace looks complete, see [Overview](/platform/overview) for issues, Inspect, and workflow connections. ## Stamp a release Tag every ingest payload with the stable identifier your deployment process assigns to the running app, such as an app version, commit SHA, or container tag. Change it with each deployment. Pass `release` on the client, or set `LEMMA_RELEASE`. Most continuous integration systems expose the current commit SHA as an environment variable. Set `LEMMA_RELEASE` to that value during deployment. An explicit constructor value wins over the environment variable. Lemma omits empty, whitespace-only, longer-than-200-character, or newline, tab, or carriage-return values. It does not guess a git SHA. Adapters (`vercelAI`, `openAIAgents`, `langChain`, `langGraph`, `mastra`) accept the same `release` field and forward it to the client they construct. ```typescript theme={null} const lemma = new Lemma({ release: "1.8.3", }); ``` ```python theme={null} lemma = Lemma(release="1.8.3") ``` ## Next steps Use these pages after the first ready trace lands. Add spans, tools, generations, and errors to a real agent. The same docs chat agent, instrumented once per stack. Search and inspect the trace you sent. Issues, Inspect, Artifacts, and Analytics after ingest. # Spans Source: https://docs.uselemma.ai/tracing/instrumentation/spans Trace retrieval, ranking, and app logic as child spans A **span** is a unit of work inside a trace that is not an LLM call or a tool call: retrieval, reranking, planning, validation, parsing, a database query, or any app operation you want to see and time. ## Record a completed span ```typescript theme={null} const ranked = await rerank(candidates); trace.recordSpan({ name: "rerank-results", input: { candidates: candidates.length }, output: { kept: ranked.length }, }); ``` ```python theme={null} ranked = rerank(candidates) trace.record_span( name="rerank-results", input={"candidates": len(candidates)}, output={"kept": len(ranked)}, ) ``` ## Measure a live span Use `startSpan()` / `start_span()` when you want the SDK to measure duration: ```typescript theme={null} const span = trace.startSpan({ name: "retrieve-context", input: query, }); try { const docs = await retrieve(query); span.end({ output: docs, durationMs: 250 }); return docs; } catch (error) { span.end({ status: "ERROR", error }); throw error; } ``` ```python theme={null} span = trace.start_span( name="retrieve-context", input=query, ) try: docs = retrieve(query) span.end(output=docs, duration_ms=250) return docs except Exception as error: span.end(status="ERROR", error=error) raise ``` The shorter handle form is useful when you also want to attach child work to the span: ```typescript theme={null} const span = trace.startSpan("retrieve-context"); const docs = await searchDocs(query); span.recordTool({ name: "search_docs", input: { query }, output: docs, }); span.end({ output: { count: docs.length }, durationMs: 250 }); ``` ```python theme={null} span = trace.start_span(name="retrieve-context") docs = search_docs(query) tool = trace.start_tool( name="search_docs", input={"query": query}, parent_id=span.id, ) tool.end(output=docs) span.end(output={"count": len(docs)}, duration_ms=250) ``` Pass `durationMs` / `duration_ms` on `trace.recordSpan(...)` / `trace.record_span(...)`, `trace.recordGeneration(...)` / `trace.record_generation(...)`, `trace.recordTool(...)` / `trace.record_tool(...)`, or span `end(...)` when you already measured the operation. If you omit child duration, Lemma splits the parent span’s remaining unclaimed duration equally across siblings that also omitted duration. For handle spans, duration is only finalized when the handle ends. The same handle pattern exists for typed children: use `trace.startTool(...)` / `trace.start_tool(...)` for live tool calls and `trace.startGeneration(...)` / `trace.start_generation(...)` for live model calls. Pass contract fields as native props such as `embeddingModelName` and `rerankerOutputDocuments`. The full supported list is in [Native contract props](/reference/trace-contract#native-contract-props). Use `attributes` only when you need to send raw span attributes that do not yet have a native SDK prop. ## Attach by trace ID If a helper receives only IDs, create the span from the client. This detached helper API is TypeScript-specific: ```typescript theme={null} const span = lemma.startSpan({ traceId, parentSpanId, name: "rerank-results", }); span.end({ output: { kept: ranked.length } }); ``` `traceId` is required for detached spans. When the detached span has a parent, use `parentSpanId`; missing IDs warn and no-op. ## Nesting spans Spans, generations, and tools recorded during `lemma.trace()` become children of that trace. Calls made on a span handle become children of that span. ```text theme={null} support-agent <- trace root └── retrieve-context <- span ├── search_docs <- tool call └── draft-reply <- generation ``` ## Next steps Group conversations and attach users. The exact shape Lemma reads. # Tool calls Source: https://docs.uselemma.ai/tracing/instrumentation/tool-calls Record tool invocations as child spans with arguments and results A **tool call** is a single tool invocation inside a trace. Record it with `trace.recordTool()` / `trace.record_tool()` after the tool returns. ## Record a tool call ```typescript theme={null} await lemma.trace({ name: "support-agent", input: userMessage }, async (trace) => { const docs = await searchDocs(userMessage); trace.recordTool({ name: "search_docs", input: { query: userMessage }, output: docs, durationMs: 25, toolParameters: { query: "string" }, }); return answerFromDocs(docs); }); ``` ```python theme={null} def run(trace): docs = search_docs(user_message) trace.record_tool( name="search_docs", input={"query": user_message}, output=docs, duration_ms=25, tool_parameters={"query": "string"}, ) return answer_from_docs(docs) lemma.trace("support-agent", run, input=user_message) ``` If the tool belongs under a measured parent span, record it as a child of that span: ```typescript theme={null} const span = trace.startSpan("retrieve-context"); const docs = await searchDocs(userMessage); span.recordTool({ name: "search_docs", input: { query: userMessage }, output: docs, durationMs: 25, }); ``` ```python theme={null} span = trace.start_span(name="retrieve-context") docs = search_docs(user_message) tool = trace.start_tool( name="search_docs", input={"query": user_message}, parent_id=span.id, ) tool.end(output=docs, duration_ms=25) ``` If the helper only has IDs, attach the tool from the client. This detached helper API is TypeScript-specific: ```typescript theme={null} lemma.recordTool({ traceId, parentSpanId: spanId, name: "search_docs", input: { query }, output: docs, durationMs: 25, }); ``` `traceId` is required for detached tools. When the tool belongs under a span, `parentSpanId` is required; missing IDs warn and no-op. Pass `durationMs` when you already measured the tool call. If you omit it, Lemma splits the parent span’s remaining unclaimed duration equally across siblings that also omitted duration. ### Show a tool call as an assistant message When a successful tool sends text directly to the user, set `userFacingMessage` / `user_facing_message` to the exact text they received. Lemma displays it as an assistant message while preserving complete input and output in the span detail. ```typescript theme={null} const input = { message: "Your order arrives Friday." }; const result = await sendWhatsApp(input); trace.recordTool({ name: "send_whatsapp", input, output: result, userFacingMessage: input.message, }); ``` ```python theme={null} tool_input = {"message": "Your order arrives Friday."} result = send_whatsapp(tool_input) trace.record_tool( name="send_whatsapp", input=tool_input, output=result, user_facing_message=tool_input["message"], ) ``` Lemma does not inspect your tool’s input schema. Omit this field for internal tools or unsuccessful sends; their existing rendering remains unchanged. ## Measure a live tool call Use `startTool()` / `start_tool()` when you want a handle before the tool finishes. End it only after the tool returns or fails. ```typescript theme={null} const tool = trace.startTool({ name: "search_docs", input: { query }, }); try { const docs = await searchDocs(query); tool.end({ output: docs, durationMs: 25 }); return docs; } catch (error) { tool.end({ error }); throw error; } ``` ```python theme={null} tool = trace.start_tool( name="search_docs", input={"query": query}, ) try: docs = search_docs(query) tool.end(output=docs, duration_ms=25) return docs except Exception as error: tool.end(error=error) raise ``` ## What the SDK records | Field | Client sends | Ingest normalizes to | | - | - | - | | Tool kind | `type: "tool"` | `openinference.span.kind: "tool"` | | Tool name | span `name` | `tool.name` | | Arguments | `input` | `input.value`, `ai.toolCall.args` | | Result | `output` | `output.value`, `ai.toolCall.result` | | User-facing message | `userFacingMessage` / `user_facing_message` | `lemma.tool.kind: "user_message"`, `lemma.tool.message` | | Contract props | Native props such as `toolDescription` and `toolParameters`; raw `attributes` for escape-hatch keys | Flattened OpenInference-style attributes | See [Native contract props](/reference/trace-contract#native-contract-props) and [Client fields vs ingest normalization](/reference/trace-contract#client-fields-vs-ingest-normalization). ## Record failures ```typescript theme={null} try { const customer = await lookupCustomer(customerId); trace.recordTool({ name: "lookup_customer", input: { customerId }, output: customer }); return customer; } catch (error) { trace.recordTool({ name: "lookup_customer", input: { customerId }, status: "ERROR", error, }); throw error; } ``` ```python theme={null} try: customer = lookup_customer(customer_id) trace.record_tool( name="lookup_customer", input={"customer_id": customer_id}, output=customer, ) return customer except Exception as error: trace.record_tool( name="lookup_customer", input={"customer_id": customer_id}, status="ERROR", error=error, ) raise ``` Capture arguments and results only when safe. Redact secrets, credentials, and sensitive user data before passing them to traced tools. ## Next steps Trace retrieval, ranking, and app logic. Group conversations and attach users. # Agent traces Source: https://docs.uselemma.ai/tracing/instrumentation/traces Create a Lemma trace for one agent execution A **trace** is one end-to-end agent execution, from the user’s input to the final response. Everything your agent does inside that execution (LLM calls, tool calls, retrieval, app logic) is recorded as child work inside the trace. This page is how you **create** traces in the SDK. To search, filter, and inspect ready traces in the dashboard, see [Traces](/platform/traces). **One agent execution = one trace.** Use `lemma.trace()` as the boundary around the run. ## The root trace ```typescript theme={null} import { Lemma } from "@uselemma/tracing"; const lemma = new Lemma(); const answer = await lemma.trace( { name: "support-agent", input: userMessage, }, async (trace) => { const answer = await runAgent(userMessage, trace); return answer; }, ); ``` ```python theme={null} from uselemma_tracing import Lemma lemma = Lemma() async def run(trace): answer = await run_agent(user_message, trace) return answer answer = await lemma.async_trace( "support-agent", run, input=user_message, ) ``` The SDK records the trace input, the returned value as output, and the trace name as the agent name. ## Trace handles and span handles Use the callback form when one function owns the whole run. In TypeScript, use a trace handle when work is coordinated across several helpers and you want to pass IDs around explicitly. In Python, keep the root trace in `lemma.trace()` / `lemma.async_trace()` and use span, tool, or generation handles inside the callback. ```typescript theme={null} const trace = lemma.trace({ name: "support-agent", input: userMessage, threadId, userId, }); const span = trace.startSpan("retrieve-context"); const docs = await searchDocs(userMessage); span.recordTool({ name: "search_docs", input: { query: userMessage }, output: docs, }); span.end({ output: { count: docs.length } }); await trace.end({ output: "Here is what I found...", durationMs: 1234 }); ``` ```python theme={null} def run(trace): span = trace.start_span(name="retrieve-context") docs = search_docs(user_message) tool = trace.start_tool( name="search_docs", input={"query": user_message}, parent_id=span.id, ) tool.end(output=docs) span.end(output={"count": len(docs)}) trace.output("Here is what I found...") lemma.trace( "support-agent", run, input=user_message, thread_id=thread_id, user_id=user_id, duration_ms=1234, ) ``` The TypeScript trace handle has a stable `trace.id`. Call `trace.end({ output, durationMs })` once when the run is finished to send the complete trace. Callback traces in both SDKs measure total trace duration automatically; pass `durationMs` / `duration_ms` only when you already measured it. ## Root latency vs generation duration Analytics treats these as **distinct** quantities: | Quantity | Source | Meaning | | - | - | - | | **Root / run latency** | Trace `started_at` → `ended_at` / `duration_ms` | End-to-end agent execution time | | **Generation duration** | Generation span `duration_ms` (or its `started_at` → `ended_at`) | Time for a single model call | Keep generation timing as measured wall-clock (or provider-reported response time when that is what the integration records). Do not substitute root latency for generation speed, or sum child durations into the root when the root already has an accurate wall-clock measurement. Framework integrations already set both. For manual instrumentation, pass `durationMs` on each generation and let the root measure from `lemma.trace()` start to `end()` / callback return. ## Record by ID Helpers can attach work to a trace when they only have IDs from the caller. This detached helper API is TypeScript-specific. ```typescript theme={null} const trace = lemma.trace(); const span = lemma.startSpan({ traceId: trace.id, }); lemma.recordTool({ traceId: trace.id, parentSpanId: span.id, name: "tool call", }); await trace.end(); ``` Detached handle calls require `traceId`. If a detached span, generation, or tool belongs under a parent span, pass `parentSpanId`; calls that cannot attach safely warn and no-op. ## Send a trace you built yourself `lemma.trace()` assumes the client owns the trace lifecycle within a single process. When the producer lives elsewhere (a cross-process buffer, a queue worker, a batch backfill), build a `TraceContext` yourself and deliver it with `ingest()`. Give it a stable `id` so retries address the same trace. ```typescript theme={null} import { Lemma, TraceContext } from "@uselemma/tracing"; const lemma = new Lemma(); const context = new TraceContext({ id: turnId, // stable id for this execution (use for retries) name: "support-agent", input: prompt, threadId: conversationId, }); context.recordTool({ name: "search_docs", input: query, output: docs, durationMs: 25 }); context.recordGeneration({ name: "answer", model: "gpt-4o", output: finalAnswer }); context.output(finalAnswer); await lemma.ingest(context, { startedAt }); ``` ```python theme={null} from uselemma_tracing import Lemma, TraceContext lemma = Lemma() context = TraceContext( id=turn_id, # stable id for this execution (use for retries) name="support-agent", input=prompt, thread_id=conversation_id, ) context.record_tool(name="search_docs", input=query, output=docs, duration_ms=25) context.record_generation(name="answer", model="gpt-4o", output=final_answer) context.output(final_answer) lemma.ingest(context, started_at=started_at) ``` `ingest()` POSTs one payload. Deliver **one complete trace** when the execution (agent turn) finishes: root input/output, thread/user, and all child spans in one call. This is required: patching a trace over time is not currently supported. `ingest()` is not an incremental merge API: omitted root fields do not preserve prior values, and after Lemma processes the trace once, a later re-delivery does not re-run issue extraction (occasional late child spans may still append to the tree for display). Retries of the same complete payload are safe, because already-stored span IDs are skipped, so a failed send can be retried as-is. It raises on a non-2xx response and never mutates the trace’s status. Across queues and workers, assemble the full tree in a coordinator and `ingest()` once when the turn is done. Do not stream partial deliveries under one `trace.id`. When one user turn spans a host and a sandbox, use `startTurn` / `attachTurn` / `apply` (Python `start_turn` / `attach_turn`) so they become one root. See [Cross-process turns](/tracing/instrumentation/cross-process-turns). ## Override output or record errors Return values are captured automatically. Use `trace.output()` only when the recorded output should differ from the return value. ```typescript theme={null} await lemma.trace({ name: "support-agent", input: userMessage }, async (trace) => { try { const response = await runAgent(userMessage); trace.output(response.text); return response; } catch (error) { trace.fail(error); throw error; } }); ``` ```python theme={null} def run(trace): try: response = run_agent(user_message) trace.output(response.text) return response except Exception as error: trace.fail(error) raise lemma.trace("support-agent", run, input=user_message) ``` If the callback throws, the SDK records the trace as failed, sends it to Lemma, and rethrows the original error. Automatic delivery (`trace` / `async_trace` / `TraceHandle.end()`) fails open: a Lemma ingest 4xx/5xx or network error is logged in debug mode and dropped, so it cannot fail the caller’s application or replace the original error. A transport failure on a successful run is not recorded as an agent failure. Use `ingest()` when you need a failed send to raise so you can retry. ## Pass trace context to helpers Pass the trace context into nested helper functions that need to record child work: ```typescript theme={null} import type { TraceContext } from "@uselemma/tracing"; function recordToolResult(trace: TraceContext, result: unknown) { trace.recordTool({ name: "search_docs", output: result, }); } ``` ```python theme={null} from uselemma_tracing import TraceContext def record_tool_result(trace: TraceContext, result): trace.record_tool( name="search_docs", output=result, ) ``` Avoid relying on ambient trace state. A process can coordinate multiple traces at once, so helpers should receive the trace or span handle they need explicitly. ## Add the work inside * [Generations](/tracing/instrumentation/generations) for LLM calls (model, prompt, completion). * [Tool calls](/tracing/instrumentation/tool-calls) for tool invocations (name, args, result). * [Spans](/tracing/instrumentation/spans) for everything else (retrieval, ranking, app logic). ```text theme={null} support-agent <- trace root (input, output) ├── draft-reply <- generation ├── search_docs <- tool call └── final-answer <- generation ``` ## Next steps Capture LLM calls with model, prompt, completion, and timing. Record tool arguments and results. Search and inspect the traces you send. The exact shape Lemma reads. # Common issues Source: https://docs.uselemma.ai/tracing/troubleshooting/common-issues Troubleshoot missing traces and trace-quality issues: lost context, invisible tools, blank input/output, and unsupported shapes This page is organized by **symptom**. Most issues are trace-shape problems: the data arrives, but it does not match the [trace contract](/reference/trace-contract). For the ideal happy path and bad → better → best examples, see [Building high-quality traces](/guides/building-high-quality-traces). When behavior is unclear, first enable [Debug mode](/tracing/troubleshooting/debug-mode) in the same runtime that serves traffic. It shows whether the SDK is running, which trace it tries to send, how many child spans were included, and whether the ingest request succeeds. ## Audit traces that already land Use the Lemma diagnostics skill when traces are arriving and you want to check whether their shape matches the [high-quality traces guide](/guides/building-high-quality-traces). The skill audits your traces without editing your app. ```bash theme={null} npx skills add uselemma/lemma --skill "lemma-diagnostics" ``` Then prompt your coding agent: ```text theme={null} Are these traces high quality? Audit the shape of traces already in Lemma. ``` If no traces are arriving, start with [Quickstart](/tracing/instrumentation/setup) instead. ## No traces in Lemma **Cause:** credentials, project ID, base URL, or outbound network access is wrong. **Fix:** create one `Lemma` client with your API key and project ID, then send a tiny test trace. ```typescript theme={null} import { Lemma } from "@uselemma/tracing"; const lemma = new Lemma({ apiKey: process.env.LEMMA_API_KEY, projectId: process.env.LEMMA_PROJECT_ID, }); await lemma.trace({ name: "smoke-test", input: "hello" }, async () => "ok"); ``` ```python theme={null} import os from uselemma_tracing import Lemma lemma = Lemma( api_key=os.environ["LEMMA_API_KEY"], project_id=os.environ["LEMMA_PROJECT_ID"], ) lemma.trace("smoke-test", lambda trace: "ok", input="hello") ``` * Confirm `LEMMA_API_KEY` and `LEMMA_PROJECT_ID` come from the same project. * Confirm your runtime can reach `https://api.uselemma.ai`. * Pass `baseUrl` / `base_url` only when using a self-hosted or staging endpoint. **Debug with Debug mode:** enable `LEMMA_DEBUG=1` and run the smoke test again. If you do not see SDK logs, the traced code path is not running in that process. If you see an ingest failure, check the status code, credentials, project ID, and `baseUrl`. ## Every span appears as a separate trace **Cause:** child work is being recorded outside the `lemma.trace()` callback, so the SDK cannot attach it to the root trace. **Fix:** make `lemma.trace()` the boundary around the whole agent run, and record tools, generations, and spans inside that callback. ```typescript theme={null} const answer = await lemma.trace( { name: "support-agent", input: userMessage, threadId, userId }, async (trace) => { const docs = await searchDocs(userMessage); trace.recordTool({ name: "search_docs", input: { query: userMessage }, output: docs }); const response = await callModel(userMessage, docs); trace.recordGeneration({ name: "answer", model: response.model, input: response.messages, output: response.text, }); return response.text; }, ); ``` ```python theme={null} def run(trace): docs = search_docs(user_message) trace.record_tool( name="search_docs", input={"query": user_message}, output=docs, ) response = call_model(user_message, docs) trace.record_generation( name="answer", model=response.model, input=response.messages, output=response.text, ) return response.text answer = lemma.trace( "support-agent", run, input=user_message, thread_id=thread_id, user_id=user_id, ) ``` Distinguish two boundaries: * **Successive conversation turns** in different processes (request handler answers turn 1, a worker answers turn 2): start a **new** root for each turn and pass the same `threadId` / `userId`. See [Instrumenting multi-turn agents](/guides/instrumenting-multi-turn-agents). * **One user turn** on a host and a sandbox: export a context token, record a journal in the child with no API key, then `apply` and `ingest()` once. See [Cross-process turns](/tracing/instrumentation/cross-process-turns). **Debug with Debug mode:** compare logs from the request handler and any background worker. If the child work runs after the `lemma.trace()` callback has returned, it will not attach to the root trace automatically. Move the recording into the callback, pass trace/span IDs to detached `record*` helpers from the coordinator, or use `startTurn` / `attachTurn` when the child is a separate process of the **same** turn. ## Host and sandbox show up as two traces **Cause:** each process started its own `lemma.trace()` and shared `threadId` / `thread_id`. Lemma groups those as two conversation turns, not one execution. **Fix:** export a context token on the host. Call `attachTurn` / `attach_turn` in the child with no API key. Apply the journal on the host and `ingest()` once. Keep `LEMMA_API_KEY` off the sandbox. ```typescript theme={null} const turn = startTurn(lemma, { name: "agent-turn", input: userMessage, threadId, }); const sandbox = turn.startSpan({ name: "e2b-sandbox" }); turn.apply(childJournal); sandbox.end(); await turn.end({ output }); ``` ```python theme={null} turn = lemma.start_turn("agent-turn", input=user_message, thread_id=thread_id) sandbox = turn.start_span(name="e2b-sandbox") turn.apply(child_journal) sandbox.end() turn.end(output=answer) ``` See [Cross-process turns](/tracing/instrumentation/cross-process-turns). Set `threadId` on the host root so this turn groups with the rest of the conversation. **Debug with Debug mode:** the host should log one `sending trace` / `trace sent` with sandbox children included. The child process should not log ingest at all. ## Tool calls are missing **Cause:** provider instrumentation captures model calls but not your application tools, or the tool result is never recorded. **Fix:** call `trace.recordTool()` after each tool completes. ```typescript theme={null} const result = await lookupOrder(orderId); trace.recordTool({ name: "lookup_order", input: { orderId }, output: result, }); ``` ```python theme={null} result = lookup_order(order_id) trace.record_tool( name="lookup_order", input={"order_id": order_id}, output=result, ) ``` For failures, record the error before re-raising: ```typescript theme={null} try { return await chargeCard(payload); } catch (error) { trace.recordTool({ name: "charge_card", input: payload, error }); throw error; } ``` ```python theme={null} try: return charge_card(payload) except Exception as error: trace.record_tool(name="charge_card", input=payload, error=error) raise ``` **Debug with Debug mode:** check the `spanCount` (`span_count` in Python) on the `sending trace` log. If the count does not increase after you add `recordTool(...)`, the tool recording code path is not running or is running outside the trace. ## Inputs or outputs are blank **Cause:** the trace was created without `input`, and the callback did not return or set the final output. **Fix:** pass input to `lemma.trace()` and return the final answer. Use `trace.output()` only when the callback return value is not the user-visible output. ```typescript theme={null} await lemma.trace({ name: "support-agent", input: userMessage }, async (trace) => { const answer = await runAgent(userMessage); trace.output(answer); return answer; }); ``` ```python theme={null} def run(trace): answer = run_agent(user_message) trace.output(answer) return answer lemma.trace("support-agent", run, input=user_message) ``` **Debug with Debug mode:** confirm the trace logs `trace sent`, then open that trace in Lemma. If root output is still blank, the callback likely returned `undefined`, streamed the answer elsewhere, or exited before setting `trace.output(...)`. ## Root input/output is null after a second delivery **Cause:** the trace was delivered again with a sparse `ingest()` payload, for example a follow-up call that carries a late child span but omits root `input`/`output`. `ingest()` is not an incremental merge API, so omitted root fields do not preserve prior values, and analysis already ran once after the first complete delivery. **Fix:** send **one complete trace** when the execution finishes: root input/output, thread/user, and all child spans in a single `ingest()` call. Retries of that same complete payload are safe (already-stored span IDs are skipped). Across queues and workers, assemble the full tree in a coordinator and ingest once rather than streaming partial deliveries under one `trace.id`. For a host and sandbox that are one turn, apply the child journal on the host and ingest once. See [Cross-process turns](/tracing/instrumentation/cross-process-turns). **Debug with Debug mode:** check the `sending trace` log for each delivery. If a later send has a small `spanCount` (`span_count` in Python) and no root I/O, that sparse payload is overwriting the root fields. Consolidate it into the single terminal send. ## Model or content is missing **Cause:** the LLM call is recorded as a generic span, so Lemma has no model, prompt, or completion to read as a generation. **Fix:** record model calls with `trace.recordGeneration()`. ```typescript theme={null} trace.recordGeneration({ name: "draft-reply", model: "gpt-4o", input: messages, output: response.text, }); ``` ```python theme={null} trace.record_generation( name="draft-reply", model="gpt-4o", input=messages, output=response.text, ) ``` See [Generations](/tracing/instrumentation/generations). **Debug with Debug mode:** check whether `spanCount` (`span_count` in Python) increases when the model call runs. If it does but the dashboard still lacks model/content metadata, switch that call site from generic span recording to `recordGeneration()` / `record_generation()` and include `model`, `input`, and `output`. ## Agent name is blank **Cause:** no stable name is attached to the trace, so traces cannot be grouped or filtered by workflow. **Fix:** set `name` on every `lemma.trace()` call. ```typescript theme={null} await lemma.trace({ name: "support-agent", input: userMessage }, async (trace) => { return runAgent(userMessage, trace); }); ``` ```python theme={null} lemma.trace( "support-agent", lambda trace: run_agent(user_message, trace), input=user_message, ) ``` **Debug with Debug mode:** check the `name` on `trace started` or `sending trace`. It should be the stable workflow name you expect, not a generated fallback. ## Trace nesting is flat **Cause:** long-running subtasks are recorded as sibling observations when they should have their own parent span. **Fix:** use `trace.startSpan()` / `trace.start_span()` when a subtask has nested work. In TypeScript, record children from the span handle. In Python, pass the parent span handle’s `id` as `parent_id` when starting child work. ```typescript theme={null} const retrieve = trace.startSpan({ name: "retrieve-context", input: { query } }); try { const docs = await searchDocs(query); retrieve.recordTool({ name: "search_docs", input: { query }, output: docs }); retrieve.end({ output: { count: docs.length } }); return docs; } catch (error) { retrieve.end({ error }); throw error; } ``` ```python theme={null} retrieve = trace.start_span(name="retrieve-context", input={"query": query}) try: docs = search_docs(query) tool = trace.start_tool( name="search_docs", input={"query": query}, parent_id=retrieve.id, ) tool.end(output=docs) retrieve.end(output={"count": len(docs)}) return docs except Exception as error: retrieve.end(error=error) raise ``` ```text theme={null} support-agent ├─ retrieve-context │ └─ search_docs └─ answer ``` **Debug with Debug mode:** confirm `spanCount` (`span_count` in Python) includes both the parent span and the child tool, then inspect the trace in Lemma. If they appear as siblings, record the child from the parent span handle in TypeScript, or pass `parent_id=retrieve.id` when starting the child in Python, before ending the parent span. ## Serverless traces are delayed or missing **Cause:** the request exits before the SDK request finishes, or a background task records work after the trace callback has returned. **Fix:** `await lemma.trace(...)` inside the handler and keep trace recording inside the callback. The SDK sends the completed payload before `lemma.trace()` resolves. **Debug with Debug mode:** check whether the ingest log appears before the function returns. If it appears after the response is sent, or never appears, await the trace promise and avoid detached background recording unless you explicitly end the trace handle. ## Traces render but there are no issues **Cause:** the trace renders, but it is not in a shape Lemma recognizes for automated issue detection. **Fix:** match the contract: one root trace, input/output, typed generation children, typed tool children, error states, and thread/user context where available. * Use `lemma.trace()` for the root. * Use `trace.recordGeneration()` / `trace.record_generation()` for model calls. * Use `trace.recordTool()` / `trace.record_tool()` for tool calls. * Use `trace.recordSpan()` / `trace.record_span()` for completed app work or `trace.startSpan()` / `trace.start_span()` for live app work. * Pass `threadId` / `thread_id` and `userId` / `user_id` for multi-turn analysis. * If one turn spans a host and a sandbox, use `startTurn` / `attachTurn` / `apply`. See [Cross-process turns](/tracing/instrumentation/cross-process-turns). **Debug with Debug mode:** confirm the SDK logs `trace sent` with the expected trace name and child `spanCount` (`span_count` in Python), then compare the dashboard trace against the contract. # Debug mode Source: https://docs.uselemma.ai/tracing/troubleshooting/debug-mode Use SDK runtime logs to debug trace delivery and trace shape Debug mode prints SDK logs from the same runtime where your agent runs. Use it when a trace is missing, delayed, attached incorrectly, or missing child work. It helps answer four questions: 1. Did this code path create a Lemma trace? 2. Did the SDK send a payload to `POST /traces/ingest`? 3. Did the ingest request succeed or fail? 4. How many child spans did the SDK include? ## Enable debug mode Enable debug mode before creating the Lemma client. ```typescript theme={null} import { enableDebugMode, Lemma } from "@uselemma/tracing"; enableDebugMode(); const lemma = new Lemma({ apiKey: process.env.LEMMA_API_KEY, projectId: process.env.LEMMA_PROJECT_ID, }); ``` ```python theme={null} from uselemma_tracing import Lemma, enable_debug_mode enable_debug_mode() lemma = Lemma() ``` You can also enable it without code: ```bash theme={null} export LEMMA_DEBUG=1 ``` `LEMMA_DEBUG=true` is also accepted for backwards compatibility. After init, debug mode prints a one-time config banner: ```text theme={null} [LEMMA:client] initialized baseUrl: https://api.uselemma.ai projectId: apiKey: ... ingestPath: /traces/ingest expectedSuccessStatus: 201 ``` If `baseUrl` is not production or `projectId` is not a valid UUID, the banner includes `warnings`. Verify logs and `debugSmokeTest().ingestStatus` reflect whether **this specific trace** was enqueued / ingested / ready: not whether the project has any ready traces. ## Run a smoke test Use `debugSmokeTest()` for a one-call delivery checklist you can paste into support: ```typescript theme={null} const result = await lemma.debugSmokeTest(); console.log(result); ``` ```python theme={null} result = lemma.debug_smoke_test() print(result) ``` The result includes config warnings, ingest HTTP status, response headers (`cf-ray`, `server`), and per-trace `ingestStatus` (`enqueued` / `ingested` / `ready` / `not_found`). You can also run one minimal trace manually from the same runtime that serves production traffic: ```typescript theme={null} await lemma.trace({ name: "smoke-test", input: "hello" }, async () => "ok"); ``` ```python theme={null} lemma.trace("smoke-test", lambda trace: "ok", input="hello") ``` For a successful trace, look for this sequence: ```text theme={null} [LEMMA:client] initialized [LEMMA:client] trace started [LEMMA:client] sending trace [LEMMA:client] trace sent ``` `sending trace` now includes `projectId`, `bodyBytes`, and `requestedAt`. `trace sent` includes response headers (`cf-ray`, `server`, `date`) and warns when `status !== 201`. If the trace uses a handle, you should see `trace handle created`, then `sending trace` when the handle ends. ## Verify ingest enqueue After a successful 201, optionally poll the API to confirm **this specific trace** was enqueued (or already ingested/ready). This requires **both** debug mode and verify: ```bash theme={null} export LEMMA_DEBUG=1 export LEMMA_DEBUG_VERIFY=1 ``` `true` is also accepted for both flags. With both enabled, the SDK calls `GET /traces/ingest-status?project_id=...&otel_trace_id=...` after ingest (polls immediately, then every 1s for up to 15s) and logs one of: ```text theme={null} [LEMMA:verify] ingest accepted (201), trace enqueued (status=enqueued) [LEMMA:verify] ingest accepted (201), trace visible (ingested) (status=ingested) [LEMMA:verify] ingest accepted (201), trace visible (ready) (status=ready) [LEMMA:verify] ingest accepted (201), ingest-status=not_found after 15s — may be processing lag or downstream issue [LEMMA:verify] ingest accepted (201), ingest-status check failed (status/network) ``` `enqueued` means the OTLP worker accepted the payload and wrote it to the ingest queue: it does **not** require the dashboard to show the trace yet (useful during backlog). ## Confirming delivery reached Lemma | SDK log | Server-side (Dash0) | Meaning | | - | - | - | | `trace sent` status 201 | `otlp.ingest.received` → `enqueued` | Full success | | `trace sent` status 201 | No `otlp.ingest.*` logs | Not hitting Lemma production: wrong URL or proxy | | `trace sent` status 200 | none | Wrong endpoint (`/otel/v1/traces` or mock) | | `trace sent` status 201 | `received` but no `enqueued` | Worker-side failure after accept | | `trace sent` status 201 + verify `status=enqueued` | `enqueued` | Accepted into ingest queue (may still be backlog) | | `trace sent` status 201 + verify `not_found` | missing enqueue index / CH row | Downstream lag or failure | Support bundle to request from customers: * `projectId`, `traceId`, ingest `url`, HTTP `status` * Response headers (`cf-ray`, `server`, `date`) * Timestamp from `requestedAt` in the `sending trace` log ## Read the logs Use the first missing log line to narrow the problem. | What you see | What it means | What to check | | - | - | - | | No Lemma logs | Debug mode is not enabled in this process, or this code path is not running | Set `LEMMA_DEBUG=1` in the runtime that serves traffic | | `trace started`, but no `sending trace` | The callback did not finish, or the trace handle was never ended | Await `lemma.trace(...)`, call `trace.end(...)`, and check thrown errors | | `sending trace`, then `trace ingest failed` | The SDK reached the endpoint, but ingest rejected the request | Check status code, `hint` field, API key, project ID, `baseUrl`, and network policy | | `trace sent`, but the dashboard looks wrong | Delivery worked; the issue is likely trace shape | Compare `spanCount` (`span_count` in Python), tool/generation calls, and the [trace contract](/reference/trace-contract) | ## Debug common issues ### Missing traces Start with the smoke test. If it logs `trace sent`, credentials and networking are working; move to the real request path and confirm that path also logs `trace started`. If it logs `trace ingest failed`, check: * `LEMMA_API_KEY` and `LEMMA_PROJECT_ID` are from the same project. * `baseUrl` / `base_url` points at a server that implements `POST /traces/ingest`. * Your runtime can make outbound HTTPS requests. * The `hint` field on `trace ingest failed` (401 → API key, 403 → project mismatch, 429 → rate limit, 404 → wrong host). ### Missing tools or generations Look at the `spanCount` (`span_count` in Python) in the `sending trace` log. If the count is lower than expected, the SDK never recorded those children. Common fixes: * Put `recordTool()` / `recordGeneration()` or `record_tool()` / `record_generation()` inside the `lemma.trace()` callback. * For work under a parent span, record it on the span handle in TypeScript or pass `parent_id` from the parent handle in Python before calling `span.end(...)`. * For detached helpers, pass `traceId` and `parentSpanId` where needed. ### Flat or incorrectly nested traces If a parent span exists but its children appear as siblings, debug the order of calls: ```typescript theme={null} const retrieve = trace.startSpan({ name: "retrieve-context" }); retrieve.recordTool({ name: "search_docs", output: docs }); retrieve.end({ output: { count: docs.length } }); ``` ```python theme={null} retrieve = trace.start_span(name="retrieve-context") tool = trace.start_tool(name="search_docs", parent_id=retrieve.id) tool.end(output=docs) retrieve.end(output={"count": len(docs)}) ``` The child call should happen before `retrieve.end(...)`. In TypeScript, make it from the parent span handle. In Python, pass the parent handle’s `id` as `parent_id` when starting the child. ### Serverless or streaming handlers Debug mode should show `trace sent` before the function exits. If not, make sure the handler awaits the trace: ```typescript theme={null} return await lemma.trace({ name: "support-agent", input }, async (trace) => { return runAgent(input, trace); }); ``` ```python theme={null} return await lemma.async_trace( "support-agent", lambda trace: run_agent(input, trace), input=input, ) ``` For trace handles, call and await `trace.end(...)` from the terminal callback or finalization path.