Your Codex Token Total Can Be Wrong Even When Every Event Is Valid

A local usage ledger can overcount and undercount the same Codex history at once. The fix is to treat usage events as a stream with replay and archive semantics, not a pile of numbers to sum.

The first bad sign was a comparison that looked too large to dismiss as pricing drift. One long-running Codex rollout contributed hundreds of millions of tokens more than expected. The JSONL was valid. Every usage event parsed. Nothing was obviously corrupt.

The mistake was assuming that a valid last_token_usage object always represented a new billable turn.

A delta can be reported more than once

Codex writes token-count events with two useful views of usage. last_token_usage describes the latest delta. total_token_usage is the cumulative counter for the current run.

During a long session, the runtime can repeat the latest delta without advancing the cumulative counter. A naive scanner sees another valid event and adds it again. In one measured rollout, those repeats produced roughly 331 million phantom tokens.

The cumulative pair gives the scanner a replay key:

previous cumulative: input=150, output=15
new delta:           input=50,  output=5
new cumulative:      input=150, output=15
decision:            replay, skip the delta

The comparison must use both cumulative input and output. If the pair is exactly unchanged, the event is a re-report of usage already counted.

This is deliberately narrower than a monotonicity rule. Compaction can reset the cumulative counters to smaller values. A smaller pair may still accompany a new, legitimate delta, so the scanner counts it. Older Codex builds may omit total_token_usage; those events also pass through because there is no reliable replay signal.

The resulting invariant is simple:

same cumulative pair  -> skip
advanced pair         -> count
smaller pair          -> count as a reset
no cumulative pair    -> count as legacy data

That rule is covered by a focused fixture with six events: two original turns, two exact replays, one compaction reset, and one legacy event. Four events must survive, totaling 200 input and 20 output tokens.

The opposite error lives in another directory

Removing replayed deltas fixed the inflation, but it did not make the ledger complete.

Codex moves older rollout files from ~/.codex/sessions to ~/.codex/archived_sessions. A scanner that walks only the live tree silently loses history as soon as a session is archived. On one local data set, 2.5 billion July tokens existed only in the archive tree.

The reader now walks both roots:

~/.codex/sessions
~/.codex/archived_sessions

Both use the same cutoff, file filter, parser, and cache. This matters because archive support should not create a second accounting path with different behavior. A rollout is a rollout regardless of which lifecycle directory currently owns it.

The two bugs pushed totals in opposite directions. Replays inflated active files. Ignoring archives deleted older files. Either issue could partially mask the other and make a rough comparison look plausible.

Streaming matters more than clever JSON

Rollout files are not tidy accounting exports. They contain tool output, screenshots, and other response items that may be megabytes long. The accounting records are a small subset: turn-context lines establish the active model, while token-count lines carry usage.

The scanner therefore streams lines with a size backstop and performs a cheap byte-marker check before JSON parsing. Oversized records that cannot contain the markers are skipped before they are assembled into large in-memory strings. Parsed files are cached by path, modification time, and size, so steady-state refreshes avoid rereading unchanged sessions.

That design keeps the ledger local without turning every refresh into a full history import.

Attribution time is a product decision

After replay and archive handling, another difference can remain between two tools that read the same files. One may assign every token to the session's start date. Another may use each event's timestamp.

Agent Island uses event time in the user's local timezone. A session that runs across midnight contributes usage to both days. Assigning the whole file to its start day is easier, but it moves later work backward and can badly distort weekly or monthly views for long-running sessions.

Neither timestamp choice changes the lifetime total. It changes the story the daily chart tells.

What this ledger still cannot promise

Local token events support a usage estimate. They do not provide an invoice.

Model pricing tables can age. Provider billing may apply rules that are not represented in a local log. Cached input needs a different rate from uncached input. Early events may appear before a model context and require a documented fallback. A UI should call the result an estimated API value, not exact spend or a reconciliation with a provider dashboard.

The broader lesson is about event semantics. Parsing is only the first step. A trustworthy local ledger also needs to know when an event is new, where old files move, which timestamp owns the usage, and which claims the source data cannot support.

Agent Island 1.7.1 includes the replay guard and archive scan on macOS and Windows. The project is free, MIT licensed, and keeps session accounting on the user's machine.