Building high-quality traces

A high-quality Lemma trace is not just “data arrived.” It is a complete, nested record of one agent execution: every span has real timing, every successful span has input and output, generations carry the full message list sent to the model, and tools show what they were asked and what they returned. This page is the pedagogical ideal path. For the exact ingest fields, validation rules, and client-vs-normalization mapping, see the trace contract. Use Common issues when something is missing or malformed.

One agent execution = one trace. Child LLM calls, tool calls, retrieval, and app logic belong inside that root — not as separate top-level traces.

The ideal trace

trace rootsupport-agent2.41s
input:  “Where is my order #1843?“
output: “It ships tomorrow and arrives Friday.”
spanretrieve-context412ms
input:  
output: 
toolsearch_docs183ms
args:   
result: 
toollookup_order96ms
args:   
result: 
generationanswer1.87s
model:    gpt-4o
messages: [system, prior turns, user, tool results]
output:   “It ships tomorrow and arrives Friday.”

What “ideal” means on each node:

Root — current user input, final answer (or error), measured duration
Spans / tools — real input and output, nested under the step that caused them, measured start → end
Generations — model name plus the full message list sent to that call (system, prior turns, tool results, current user), not just the latest user string

Quality checklist

Rule	Why it matters
One root per agent execution	Otherwise one run scatters across unrelated traces
Root has input and final output (or error)	You can read what the user asked and whether the run succeeded
Correct typed children	Generations, tools, and generic spans render and filter correctly
Every span has a real start and end	Duration, latency, and timelines stay trustworthy
Every successful span has input and output	Opaque spans hide what happened; failures keep input plus error
Every failure is recorded on the span where it occurred	You can see which operation failed and what input caused it
Nest under the work that caused the child	Flat trees hide retrieval, retries, and sub-steps
Generations record the full message list for that call	Prior turns and tool results are part of what the model saw
Tools record arguments and result (or error)	Tool behavior becomes inspectable

Record complete inputs and outputs only when safe. Redact secrets, credentials, and sensitive user data before tracing them.

Structure and nesting

Bad — each call is its own top-level trace:

draft-reply                ← separate trace
search_docs                ← separate trace
final-answer               ← separate trace

Better — one root, but everything is flat under it:

support-agent
├── plan
├── embed-query
├── search_docs
├── rerank
└── final-answer

Best — one root, children nested under the step that performed them:

support-agent
├── plan
│   └── embed-query
├── retrieve
│   ├── search_docs
│   └── rerank
└── final-answer

Wrap the run with lemma.trace() and record children from the trace or span handle that owns them. See Traces and Spans.

Input / output

Bad — named span with nothing useful:

retrieve-context           ← no input, no output

Better — partial payload:

retrieve-context           ← input only (or output only)

Best — every successful span carries both sides:

retrieve-context           ← input + output

Failures should still keep the input and record the error instead of an output. Opaque spans hide what happened; input and output make the tree inspectable.

Timing

Bad — no duration at all:

retrieve-context           ← duration missing

Better — duration inferred from the parent:

retrieve-context           ← duration inferred

Best — measured from real start → end:

retrieve-context           ← duration from start → end

Prefer live handles (startSpan / start_span, and the matching tool and generation helpers) so .end(...) establishes duration after the work completes. Pass durationMs / duration_ms only when you already measured the operation externally. Lemma can infer missing child durations from a parent, but measured bounds are better.

Errors

Record a failure on the exact span, generation, or tool where it happened. Keep the input, pass the error instead of an output, and end the span so timing remains accurate. Bad — the exception is swallowed and the failed operation is missing:

support-agent              ← lookup_order failed, but no failed span

Better — the root fails, but the source is unclear:

support-agent              ← error: "Order service unavailable"
└── lookup_order           ← no error recorded here

Best — the child records its input, error, and duration; the root also fails when the execution cannot recover:

support-agent              ← error: "Order service unavailable"
└── lookup_order           ← input + error + duration

With a live handle, call tool.end({ error }) / tool.end(error=error) and rethrow when the whole execution should fail. Callback traces automatically record uncaught exceptions on the root. If you catch an error at the root boundary, call trace.fail(error). If the agent recovers, keep the child marked as failed and let the root complete successfully. See Errors in the trace contract for the exact recording rules and SDK examples.

Generations

For each LLM call, record the full ordered message list sent to that call — system prompt, prior conversation turns, tool results, and the current user message — not only the latest user string. That list is what the model saw; the root trace input is only the current user turn. Bad — little more than a name:

answer                     ← generation (name only)

Better — model and latest user string only:

answer                     ← model + current message + output

Best — model, complete message history for this call, and the response:

answer                     ← model + full messages + output + duration

[
  { "role": "system", "content": "You are a support agent." },
  { "role": "user", "content": "Hi, I ordered last week" },
  { "role": "assistant", "content": "Happy to help — what's the order id?" },
  { "role": "user", "content": "Where is my order #1843?" },
  { "role": "tool", "content": "{\"status\":\"shipped\",\"eta\":\"Fri\"}" }
]

In the SDK, pass that full list as both input and llmInputMessages / llm_input_messages. See Generations and the trace contract.

Tool calls

Bad — tools ran but were never recorded:

support-agent
├── draft-reply
└── final-answer           ← search_docs ran, but no tool span

Better — tool span exists, but opaque:

search_docs                ← tool (name only)

Best — typed tool under the correct parent, with args and result:

retrieve-context
└── search_docs            ← tool (args, result, duration)

Record each completed tool with trace.recordTool({ name, input, output }) / trace.record_tool(name=..., input=..., output=...), or use live tool handles when you want measured duration. See Tool calls.

Happy-path example

Use live handles so start → end establishes timing. Put the current user message on the root, and the full model message list on the generation.

TypeScript
Python

await lemma.trace(
  {
    name: "support-agent",
    input: userMessage,
    threadId,
    userId,
  },
  async (trace) => {
    const retrieve = trace.startSpan({
      name: "retrieve-context",
      input: { query: userMessage },
    });

    const search = retrieve.startTool({
      name: "search_docs",
      input: { query: userMessage },
    });
    const docs = await searchDocs(userMessage);
    search.end({ output: docs });

    retrieve.end({ output: { count: docs.length } });

    const messages = [
      { role: "system", content: "You are a support agent." },
      ...priorMessages,
      { role: "user", content: userMessage },
      {
        role: "tool",
        content: JSON.stringify(docs),
      },
    ];

    const generation = trace.startGeneration({
      name: "answer",
      model: "gpt-4o",
      input: messages,
      llmInputMessages: messages,
    });

    try {
      const response = await callModel(messages);
      generation.end({ output: response.text });
      return response.text;
    } catch (error) {
      generation.end({ error });
      throw error;
    }
  },
);

def run(trace):
    retrieve = trace.start_span(
        name="retrieve-context",
        input={"query": user_message},
    )

    search = trace.start_tool(
        name="search_docs",
        input={"query": user_message},
        parent_id=retrieve.id,
    )
    docs = search_docs(user_message)
    search.end(output=docs)

    retrieve.end(output={"count": len(docs)})

    messages = [
        {"role": "system", "content": "You are a support agent."},
        *prior_messages,
        {"role": "user", "content": user_message},
        {"role": "tool", "content": json.dumps(docs)},
    ]

    generation = trace.start_generation(
        name="answer",
        model="gpt-4o",
        input=messages,
        llm_input_messages=messages,
    )

    try:
        response = call_model(messages)
        generation.end(output=response.text)
        return response.text
    except Exception as error:
        generation.end(error=error)
        raise

lemma.trace(
    "support-agent",
    run,
    input=user_message,
    thread_id=thread_id,
    user_id=user_id,
)

Next steps

Trace contract

The exact shape and fields Lemma reads.

Instrument an agent

Build a complete trace one piece at a time.

Generations

Capture model, full messages, and completion.

Common issues

Diagnose missing spans, flat trees, and blank I/O.

​The ideal trace

​Quality checklist

​Structure and nesting

​Input / output

​Timing

​Errors

​Generations

​Tool calls

​Happy-path example

​Next steps

Trace contract

Instrument an agent

Generations

Common issues

The ideal trace

Quality checklist

Structure and nesting

Input / output

Timing

Errors

Generations

Tool calls

Happy-path example

Next steps