Debug mode

Debug mode prints SDK logs from the same runtime where your agent runs. Use it when a trace is missing, delayed, attached incorrectly, or missing child work. It helps answer four questions:

Did this code path create a Lemma trace?
Did the SDK send a payload to POST /traces/ingest?
Did the ingest request succeed or fail?
How many child spans did the SDK include?

Enable it

Enable debug mode before creating the Lemma client.

TypeScript
Python

import { enableDebugMode, Lemma } from "@uselemma/tracing";

enableDebugMode();

const lemma = new Lemma({
  apiKey: process.env.LEMMA_API_KEY,
  projectId: process.env.LEMMA_PROJECT_ID,
});

from uselemma_tracing import Lemma, enable_debug_mode

enable_debug_mode()

lemma = Lemma()

You can also enable it without code:

export LEMMA_DEBUG=1

LEMMA_DEBUG=true is also accepted for backwards compatibility. After init, debug mode prints a one-time config banner:

[LEMMA:client] initialized
  baseUrl: https://api.uselemma.ai
  projectId: <your-project-id>
  apiKey: ...<last4>
  ingestPath: /traces/ingest
  expectedSuccessStatus: 201

If baseUrl is not production or projectId is not a valid UUID, the banner includes warnings. Verify logs and debugSmokeTest().ingestStatus reflect whether this specific trace was enqueued / ingested / ready — not whether the project has any ready traces.

Run a smoke test

Use debugSmokeTest() for a one-call delivery checklist you can paste into support:

TypeScript
Python

const result = await lemma.debugSmokeTest();
console.log(result);

result = lemma.debug_smoke_test()
print(result)

The result includes config warnings, ingest HTTP status, response headers (cf-ray, server), and per-trace ingestStatus (enqueued / ingested / ready / not_found). You can also run one minimal trace manually from the same runtime that serves production traffic:

TypeScript
Python

await lemma.trace({ name: "smoke-test", input: "hello" }, async () => "ok");

lemma.trace("smoke-test", lambda trace: "ok", input="hello")

For a successful trace, look for this sequence:

[LEMMA:client] initialized
[LEMMA:client] trace started
[LEMMA:client] sending trace
[LEMMA:client] trace sent

sending trace now includes projectId, bodyBytes, and requestedAt. trace sent includes response headers (cf-ray, server, date) and warns when status !== 201. If the trace uses a handle, you should see trace handle created, then sending trace when the handle ends.

Verify ingest enqueue

After a successful 201, optionally poll the API to confirm this specific trace was enqueued (or already ingested/ready). This requires both debug mode and verify:

export LEMMA_DEBUG=1
export LEMMA_DEBUG_VERIFY=1

true is also accepted for both flags. With both enabled, the SDK calls GET /traces/ingest-status?project_id=...&otel_trace_id=... after ingest (polls immediately, then every 1s for up to 15s) and logs one of:

[LEMMA:verify] ingest accepted (201), trace enqueued (status=enqueued)
[LEMMA:verify] ingest accepted (201), trace visible (ingested) (status=ingested)
[LEMMA:verify] ingest accepted (201), trace visible (ready) (status=ready)
[LEMMA:verify] ingest accepted (201), ingest-status=not_found after 15s — may be processing lag or downstream issue
[LEMMA:verify] ingest accepted (201), ingest-status check failed (status/network)

enqueued means the OTLP worker accepted the payload and wrote it to the ingest queue — it does not require the dashboard to show the trace yet (useful during backlog).

Confirming delivery reached Lemma

SDK log	Server-side (Dash0)	Meaning
`trace sent` status 201	`otlp.ingest.received` → `enqueued`	Full success
`trace sent` status 201	No `otlp.ingest.*` logs	Not hitting Lemma production — wrong URL or proxy
`trace sent` status 200	—	Wrong endpoint (`/otel/v1/traces` or mock)
`trace sent` status 201	`received` but no `enqueued`	Worker-side failure after accept
`trace sent` status 201 + verify `status=enqueued`	`enqueued`	Accepted into ingest queue (may still be backlog)
`trace sent` status 201 + verify `not_found`	missing enqueue index / CH row	Downstream lag or failure

Support bundle to request from customers:

projectId, traceId, ingest url, HTTP status
Response headers (cf-ray, server, date)
Timestamp from requestedAt in the sending trace log

Read the logs

Use the first missing log line to narrow the problem.

What you see	What it means	What to check
No Lemma logs	Debug mode is not enabled in this process, or this code path is not running	Set `LEMMA_DEBUG=1` in the runtime that serves traffic
`trace started`, but no `sending trace`	The callback did not finish, or the trace handle was never ended	Await `lemma.trace(...)`, call `trace.end(...)`, and check thrown errors
`sending trace`, then `trace ingest failed`	The SDK reached the endpoint, but ingest rejected the request	Check status code, `hint` field, API key, project ID, `baseUrl`, and network policy
`trace sent`, but the dashboard looks wrong	Delivery worked; the issue is likely trace shape	Compare `spanCount` (`span_count` in Python), tool/generation calls, and the trace contract

Debug common issues

Missing traces

Start with the smoke test. If it logs trace sent, credentials and networking are working; move to the real request path and confirm that path also logs trace started. If it logs trace ingest failed, check:

LEMMA_API_KEY and LEMMA_PROJECT_ID are from the same project.
baseUrl / base_url points at a server that implements POST /traces/ingest.
Your runtime can make outbound HTTPS requests.
The hint field on trace ingest failed (401 → API key, 403 → project mismatch, 429 → rate limit, 404 → wrong host).

Missing tools or generations

Look at the spanCount (span_count in Python) in the sending trace log. If the count is lower than expected, the SDK never recorded those children. Common fixes:

Put recordTool() / recordGeneration() or record_tool() / record_generation() inside the lemma.trace() callback.
For work under a parent span, record it on the span handle in TypeScript or pass parent_id from the parent handle in Python before calling span.end(...).
For detached helpers, pass traceId and parentSpanId where needed.

Flat or incorrectly nested traces

If a parent span exists but its children appear as siblings, debug the order of calls:

TypeScript
Python

const retrieve = trace.startSpan({ name: "retrieve-context" });
retrieve.recordTool({ name: "search_docs", output: docs });
retrieve.end({ output: { count: docs.length } });

retrieve = trace.start_span(name="retrieve-context")
tool = trace.start_tool(name="search_docs", parent_id=retrieve.id)
tool.end(output=docs)
retrieve.end(output={"count": len(docs)})

The child call should happen before retrieve.end(...). In TypeScript, make it from the parent span handle. In Python, pass the parent handle’s id as parent_id when starting the child.

Serverless or streaming handlers

Debug mode should show trace sent before the function exits. If not, make sure the handler awaits the trace:

TypeScript
Python

return await lemma.trace({ name: "support-agent", input }, async (trace) => {
  return runAgent(input, trace);
});

return await lemma.async_trace(
    "support-agent",
    lambda trace: run_agent(input, trace),
    input=input,
)

For trace handles, call and await trace.end(...) from the terminal callback or finalization path.

Getting Started

Instrumentation

Integrations

Connections

Enable it

Run a smoke test

Verify ingest enqueue

Confirming delivery reached Lemma

Read the logs

Debug common issues

Missing traces

Missing tools or generations

Flat or incorrectly nested traces

Serverless or streaming handlers

​Enable it

​Run a smoke test

​Verify ingest enqueue

​Confirming delivery reached Lemma

​Read the logs

​Debug common issues

​Missing traces

​Missing tools or generations

​Flat or incorrectly nested traces

​Serverless or streaming handlers

Enable it

Run a smoke test

Verify ingest enqueue

Confirming delivery reached Lemma

Read the logs

Debug common issues

Missing traces

Missing tools or generations

Flat or incorrectly nested traces

Serverless or streaming handlers