> ## Documentation Index
> Fetch the complete documentation index at: https://docs.uselemma.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Building high-quality traces
> The ideal instrumentation path, with bad → better → best examples for structure, timing, nesting, generations, and tools
A high-quality Lemma trace is not just “data arrived.” It is a complete, nested record of one agent execution: every span has real timing, every successful span has input and output, generations carry the full message list sent to the model, and tools show what they were asked and what they returned.
This page is the pedagogical ideal path. For the exact ingest fields, validation rules, and client-vs-normalization mapping, see the [trace contract](/reference/trace-contract). Use [Common issues](/tracing/troubleshooting/common-issues) when something is missing or malformed.
**One agent execution = one trace.** Child LLM calls, tool calls, retrieval, and app logic belong inside that root — not as separate top-level traces.
## The ideal trace
What “ideal” means on each node:
* **Root** — current user input, final answer (or error), measured duration
* **Spans / tools** — real input and output, nested under the step that caused them, measured start → end
* **Generations** — model name plus the **full message list** sent to that call (system, prior turns, tool results, current user), not just the latest user string
## Quality checklist
| Rule | Why it matters |
| ------------------------------------------------------- | ----------------------------------------------------------------- |
| One root per agent execution | Otherwise one run scatters across unrelated traces |
| Root has input and final output (or error) | You can read what the user asked and whether the run succeeded |
| Correct typed children | Generations, tools, and generic spans render and filter correctly |
| Every span has a real start and end | Duration, latency, and timelines stay trustworthy |
| Every successful span has input and output | Opaque spans hide what happened; failures keep input plus error |
| Every failure is recorded on the span where it occurred | You can see which operation failed and what input caused it |
| Nest under the work that caused the child | Flat trees hide retrieval, retries, and sub-steps |
| Generations record the full message list for that call | Prior turns and tool results are part of what the model saw |
| Tools record arguments and result (or error) | Tool behavior becomes inspectable |
Record complete inputs and outputs only when safe. Redact secrets, credentials, and sensitive user data before tracing them.
## Structure and nesting
**Bad** — each call is its own top-level trace:
```text theme={null}
draft-reply ← separate trace
search_docs ← separate trace
final-answer ← separate trace
```
**Better** — one root, but everything is flat under it:
```text theme={null}
support-agent
├── plan
├── embed-query
├── search_docs
├── rerank
└── final-answer
```
**Best** — one root, children nested under the step that performed them:
```text theme={null}
support-agent
├── plan
│ └── embed-query
├── retrieve
│ ├── search_docs
│ └── rerank
└── final-answer
```
Wrap the run with `lemma.trace()` and record children from the trace or span handle that owns them. See [Traces](/tracing/instrumentation/traces) and [Spans](/tracing/instrumentation/spans).
## Input / output
**Bad** — named span with nothing useful:
```text theme={null}
retrieve-context ← no input, no output
```
**Better** — partial payload:
```text theme={null}
retrieve-context ← input only (or output only)
```
**Best** — every successful span carries both sides:
```text theme={null}
retrieve-context ← input + output
```
Failures should still keep the input and record the error instead of an output. Opaque spans hide what happened; input and output make the tree inspectable.
## Timing
**Bad** — no duration at all:
```text theme={null}
retrieve-context ← duration missing
```
**Better** — duration inferred from the parent:
```text theme={null}
retrieve-context ← duration inferred
```
**Best** — measured from real start → end:
```text theme={null}
retrieve-context ← duration from start → end
```
Prefer live handles (`startSpan` / `start_span`, and the matching tool and generation helpers) so `.end(...)` establishes duration after the work completes. Pass `durationMs` / `duration_ms` only when you already measured the operation externally. Lemma can infer missing child durations from a parent, but measured bounds are better.
## Errors
Record a failure on the exact span, generation, or tool where it happened. Keep the input, pass the error instead of an output, and end the span so timing remains accurate.
**Bad** — the exception is swallowed and the failed operation is missing:
```text theme={null}
support-agent ← lookup_order failed, but no failed span
```
**Better** — the root fails, but the source is unclear:
```text theme={null}
support-agent ← error: "Order service unavailable"
└── lookup_order ← no error recorded here
```
**Best** — the child records its input, error, and duration; the root also fails when the execution cannot recover:
```text theme={null}
support-agent ← error: "Order service unavailable"
└── lookup_order ← input + error + duration
```
With a live handle, call `tool.end({ error })` / `tool.end(error=error)` and rethrow when the whole execution should fail. Callback traces automatically record uncaught exceptions on the root. If you catch an error at the root boundary, call `trace.fail(error)`. If the agent recovers, keep the child marked as failed and let the root complete successfully.
See [Errors in the trace contract](/reference/trace-contract#errors) for the exact recording rules and SDK examples.
## Generations
For each LLM call, record the **full ordered message list sent to that call** — system prompt, prior conversation turns, tool results, and the current user message — not only the latest user string. That list is what the model saw; the root trace input is only the current user turn.
**Bad** — little more than a name:
```text theme={null}
answer ← generation (name only)
```
**Better** — model and latest user string only:
```text theme={null}
answer ← model + current message + output
```
**Best** — model, complete message history for this call, and the response:
```text theme={null}
answer ← model + full messages + output + duration
```
```json theme={null}
[
{ "role": "system", "content": "You are a support agent." },
{ "role": "user", "content": "Hi, I ordered last week" },
{ "role": "assistant", "content": "Happy to help — what's the order id?" },
{ "role": "user", "content": "Where is my order #1843?" },
{ "role": "tool", "content": "{\"status\":\"shipped\",\"eta\":\"Fri\"}" }
]
```
In the SDK, pass that full list as both `input` and `llmInputMessages` / `llm_input_messages`. See [Generations](/tracing/instrumentation/generations) and the [trace contract](/reference/trace-contract#generations-and-message-history).
## Tool calls
**Bad** — tools ran but were never recorded:
```text theme={null}
support-agent
├── draft-reply
└── final-answer ← search_docs ran, but no tool span
```
**Better** — tool span exists, but opaque:
```text theme={null}
search_docs ← tool (name only)
```
**Best** — typed tool under the correct parent, with args and result:
```text theme={null}
retrieve-context
└── search_docs ← tool (args, result, duration)
```
Record each completed tool with `trace.recordTool({ name, input, output })` / `trace.record_tool(name=..., input=..., output=...)`, or use live tool handles when you want measured duration. See [Tool calls](/tracing/instrumentation/tool-calls).
## Happy-path example
Use live handles so start → end establishes timing. Put the **current user message** on the root, and the **full model message list** on the generation.
```typescript theme={null}
await lemma.trace(
{
name: "support-agent",
input: userMessage,
threadId,
userId,
},
async (trace) => {
const retrieve = trace.startSpan({
name: "retrieve-context",
input: { query: userMessage },
});
const search = retrieve.startTool({
name: "search_docs",
input: { query: userMessage },
});
const docs = await searchDocs(userMessage);
search.end({ output: docs });
retrieve.end({ output: { count: docs.length } });
const messages = [
{ role: "system", content: "You are a support agent." },
...priorMessages,
{ role: "user", content: userMessage },
{
role: "tool",
content: JSON.stringify(docs),
},
];
const generation = trace.startGeneration({
name: "answer",
model: "gpt-4o",
input: messages,
llmInputMessages: messages,
});
try {
const response = await callModel(messages);
generation.end({ output: response.text });
return response.text;
} catch (error) {
generation.end({ error });
throw error;
}
},
);
```
```python theme={null}
def run(trace):
retrieve = trace.start_span(
name="retrieve-context",
input={"query": user_message},
)
search = trace.start_tool(
name="search_docs",
input={"query": user_message},
parent_id=retrieve.id,
)
docs = search_docs(user_message)
search.end(output=docs)
retrieve.end(output={"count": len(docs)})
messages = [
{"role": "system", "content": "You are a support agent."},
*prior_messages,
{"role": "user", "content": user_message},
{"role": "tool", "content": json.dumps(docs)},
]
generation = trace.start_generation(
name="answer",
model="gpt-4o",
input=messages,
llm_input_messages=messages,
)
try:
response = call_model(messages)
generation.end(output=response.text)
return response.text
except Exception as error:
generation.end(error=error)
raise
lemma.trace(
"support-agent",
run,
input=user_message,
thread_id=thread_id,
user_id=user_id,
)
```
## Next steps
The exact shape and fields Lemma reads.
Build a complete trace one piece at a time.
Capture model, full messages, and completion.
Diagnose missing spans, flat trees, and blank I/O.