Why We Record Our AI Assistants' Work on Our Own Machine

An AI assistant can give me a finished answer without making the route to that answer clear. As an owner, I want to know what it read, which tools it used, where it failed and what resources the work consumed.

We built our observability setup around those questions. Observability means recording enough of a system's activity to investigate what happened inside it. We use Langfuse on our own machine to inspect our assistants' work.

The reason for keeping that record local was concrete: traces can contain client material. A trace is a record of an execution, including the inputs and outputs captured along the way. It can hold the text of a document, not merely a note that a document was opened.

The system has already forced us to correct assumptions about coverage, accounting and privacy. It has also made clear that recording every step is an operating intention we have to verify, not a claim a dashboard can prove by itself.

What we actually record

For Claude Code, our tracing plugin runs when a turn ends and when a session ends. It captures the prompt, assistant messages, token counts, cost information, tool calls and their inputs, outputs and timing. Tokens are the units of text a model processes and generates.

It also captures the thinking blocks exposed in the transcript, nested subagent activity, and tags identifying the skills used. A subagent is another assistant given part of the work. Those records show the available execution history; they are not a complete window into a model's internal reasoning.

Images pasted into the conversation or returned by tools can also be uploaded as media. Conversations are grouped into sessions, so we can follow the work across turns rather than inspect disconnected events.

That is our Claude Code path. Codex has a different path: a scheduled import reads its session transcripts and sends turn and model-usage records without the conversation text. Calling both of these complete textual recordings would hide a material difference.

Reading the records back is the purpose of the setup. We want to investigate which sessions used the most resources, where a skill failed, or what happened in runs that used a particular set of instructions. Collection without that ability would leave us with storage to maintain and little help making decisions.

Why this record belongs on our machine

Our decision to self-host came from the contents of the traces. Tool outputs can include client documents and working drafts. Sending a trace to an external service would create another copy of that material outside our own machine.

We chose local Langfuse rather than its cloud service for that reason. The event allowance on the cloud option we considered was another concern, but control over the captured material was the decisive issue.

This is a statement about where we send observability data. It does not mean that using a local trace store makes every AI operation local, or that no data can reach the model provider (in Bulgarian). Those are separate data paths and need separate decisions.

The stack runs in Docker, which packages the supporting services into isolated applications called containers. The web interface, background processing, databases and media storage run together locally. We removed unnecessary exposed database ports so those services communicate inside the stack.

Owning that setup means owning its operation. It needs to be running when records are sent, and its storage needs attention. We accepted that responsibility because the alternative changed where client-bearing records would live.

Local did not automatically mean private

Our review found that the web interface had been reachable from the local network with account registration open. That created a route for somebody on the same network to register and read traces.

We changed the interface to listen only on the machine itself and disabled registration. The recorded check confirmed local access and that the network address was inaccessible. Registration was checked through the configured flag rather than by creating a real account.

That distinction matters. We should be precise about both the fix and the test. Saying that we host locally would not have explained the exposure, and saying that everything was verified would overstate the registration check.

There is another issue beyond access: whether we should capture sensitive content at all. A private trace store is still an additional copy. Limiting the amount of text captured does not remove sensitive information from the part that remains.

Our dossier proposes scanning for sensitive identifiers and keys, then deciding which legal projects should be excluded. The available records do not confirm that the scan or project exclusions were implemented. The confirmed protection is the restriction on network access; it is not a restriction on which client content gets captured. The same applies to image capture, which can preserve a document even when attention is focused on the text fields.

We also documented a configuration trap: clearing the local server address while leaving credentials available could make the plugin use its cloud fallback. Disabling capture required checking the credential settings and then verifying that no trace arrived. An empty destination field was not a safe off switch.

What the records taught us about missing records

The end of a session exposed a problem in our capture mechanism. The session-end hook needed longer than the default timeout allowed, and the final turn could be dropped. We extended that timeout after identifying the cancelled hook.

Without checking the mechanism, a session could look complete while its last activity was absent. That makes a retrospective less trustworthy, especially when the missing part contains the final check or result.

We also established what happens when the local stack is down. The sender retries briefly, but there is no local queue retaining the trace for later delivery. The record is lost. The assistant's work continues.

We accepted that tradeoff rather than maintain an additional queueing mechanism. It means these traces are useful operational evidence, with known gaps. We should not present them as a lossless audit record.

The open questions also identify coverage that still needed verification for unattended work started through the Agent SDK, the library used to run assistants from code. A setting inherited by terminal sessions does not prove that a scheduled process sends the same records.

For me, this is the practical lesson: when a team says everything is logged, ask how it checked the end of the session, the background jobs and the period when the collector was unavailable (in Bulgarian).

Our cost accounting needed correction too

The Codex integration taught us that more telemetry does not necessarily create a clearer record. Telemetry is data emitted about a program's activity. In our local experiment, a request for a reply consisting of a single word produced about 1000 spans, or individual recorded operations.

Usage data and the model name were attached to different operations. That made the raw feed an awkward starting point for model-and-cost reporting. We chose the transcript import, where those details were easier to associate.

The first import then revealed a different error. A child agent's transcript began with copied parent history. Counting that history again duplicated 345 turns. A reviewer caught the problem, the duplicate records were removed, and the importer was changed to count turns by their identifiers across files.

We also had to avoid counting reasoning tokens again when they were already included in output tokens. Otherwise the same work would appear more expensive merely because we had sent overlapping fields.

The subscription distinction is especially relevant to owners. The model-based price shown for Codex is useful for comparison, but our subscription is not billed by that displayed token total. Treating the number as an invoice would give us the wrong financial interpretation even after the counting was fixed.

Quality needs more than a cheaper session

We wanted evidence about work quality as well as resource use. Our scoring script reads transcripts for whether tests passed on their first run and how many findings a reviewer returned. These are observable events that the work already produces.

The script uses the actual test exit result where available, then the test summary. If neither gives reliable evidence, it does not guess. If the first run is unreliable, a later run cannot quietly become the first for scoring purposes.

That rule prevents a cleaner-looking report from replacing what really happened. Unknown evidence has to remain unknown. Otherwise a quality metric rewards gaps in the record.

The resulting model comparisons also need context. Our notes explicitly warn that the tasks were not comparable: Codex received narrower, planned steps. A better first-run test result in that mix would not prove that a model is universally better.

I want these signals to help us investigate how we assign work. They do not replace reviewing the result or checking whether it met the business need. A low-cost session that leaves work unfinished is not automatically efficient, and a test result is not a revenue result.

Keep the learning without keeping everything

Local storage made retention an operational question. In our measurements, ClickHouse's internal system logs had reached 6.0 GiB, compared with 1.9 GiB of trace data. ClickHouse is the database used here for analytical records. Its own operational logs were a separate source of growth.

We applied a 7-day retention period to those internal logs. That was distinct from deciding how long to keep the traces we use to understand assistant performance.

For traces, the decision was to preserve a monthly performance report before deleting older detail. The configured keep window is 60 days, and deletion is permitted only for months that have a report. The reports retain aggregates such as calls, tokens, cost, response times, tool errors and scores.

The point is to preserve evidence for comparison without treating every raw tool output as something to keep forever. The deletion script defaults to a dry run, which previews what would be removed. That makes the intended deletion reviewable before execution.

A monthly report cannot reconstruct every incident after the underlying trace has gone. That is part of the tradeoff. We retain the long-term measurements while accepting a limit on detailed investigation of older work.

What I would want an owner to take from this

Our local setup gives us a way to examine work that would otherwise disappear behind a finished answer. It has helped us find capture failures, correct duplicated usage, qualify model comparisons and put a boundary around storage growth.

It also keeps the unresolved questions visible. Sensitive-content exclusions still need evidence. Background coverage cannot be assumed. Missing traces and incomparable tasks must remain visible in how we interpret the reports.

If you cannot explain what your AI assistants did, why the work cost what it did, or where their records live, bring that workflow to a free audit. We can identify the evidence you need for operating decisions and the data you should avoid copying into the monitoring system.