​
Trace lifecycle

This is the current operational path for a trace from local collection to reviewable SellTraces data.

​
1. Local discovery

The CLI looks for supported transcript locations and source kinds. Discovery does not upload transcript content. Current source adapters include Claude Code projects, Cursor SQLite state, Codex sessions, Gemini CLI, OpenCode, and the local trace index.

​
2. Scan and normalize

The CLI splits each discovered location into scan tasks (one transcript file or one SQLite store) and parses them on a worker pool through the shared ingest package adapters. A session too large for one upload request is emitted as consecutive segments while it is read (see Trace format), so no session is skipped for size and a parse holds one segment of a session at a time. This is the first structural mutation: local source files become trace objects with source, source id, messages, timestamps, client/provider data, environment metadata, and optional fields such as title or summary.

​
3. Local upload plan

Every CLI upload path builds an upload plan before any HTTP request:
Accepted traces can first differ from local transcript content after local redaction. Long traces remain upload candidates. The CLI does not skip a trace because of its token count, message count, per-message content length, or serialized size: a long session is split into segments and a single message too large for one upload is truncated with a marker. Unchanged files whose traces are all synced are skipped without parsing; a session that grew is parsed again and only its changed last segment uploads.

​
4. Web ingest request

POST /api/ingest authenticates the user, refuses the upload with 403 terms_required until the account has accepted the current Sell Traces Contributor Terms, validates JSON or ZIP upload shape, recounts CLI JSON message tokens, writes the upload payload to blob storage, inserts an ingest_jobs row, and returns 202. Heavy parse, scrub, score, and write work happens outside the web request.

​
5. Worker claim and parse

The ingest worker claims queued jobs with row locking, updates status to parsing, loads the raw upload blob, parses ZIP exports or decodes JSON traces, then marks the job ingesting. Job statuses:

​
6. Server pipeline

For each trace, the server pipeline:
  1. Normalizes the trace again.
  2. Computes an original_hash from normalized role/content text unless supplied.
  3. Runs server-side scrub.
  4. Inserts a pii_rejections row and stops if scrub rejects the trace.
  5. Scores quality and uniqueness.
  6. Estimates value.
  7. Checks for existing duplicates by (user_id, source, source_id) or original_hash.
  8. Inserts a new traces row and message rows for new traces, or replaces the content of a stored trace the upload grew from.
  9. Writes per-trace raw and derived blobs.
The derived blob is the scrubbed trace JSON used for downloads.

​
7. Duplicate handling

Duplicate uploads do not create new trace rows. The pipeline may refresh nullable metadata on the canonical trace, such as model, summary, environment, timing, task type, or quality metadata. It does not rewrite existing message rows or per-trace blobs during that metadata refresh. The exception is a session that grew since it was stored. When the upload has the same (user_id, source, source_id) as an unsold trace, carries more messages, and its first message_count messages hash to the stored original_hash, the pipeline replaces that trace’s messages, blobs, and content columns in one transaction. The first segment (<source_id>#1) of a session stored unsplit before it outgrew one upload replaces the unsplit trace the same way. Sold traces never change.

​
8. Dashboard review

Dashboard include/exclude actions update traces.sell_excluded for unsold traces. Once a trace has a sale_traces row, include/exclude mutations are rejected because the trace has already sold. Trace downloads stream the derived scrubbed blob and do not mutate the trace.

​
9. Sale and payout state

Sales attach traces through sale_traces. Payouts attach user payment state through payouts. Once sold, a trace remains part of that sale record even if later dashboard filters or review state change elsewhere.