How do I use tokensift?

Paste raw terminal, VS Code, or CI failure output, click Sift & copy, then send the conservative debugging brief to an LLM or coding agent.

Is this an exact token counter?

No. tokensift uses a deterministic rough estimate so you can compare input and output size before sharing with an LLM.

Turn failure logs into
conservative debugging briefs.

Paste a CUDA OOM, Python traceback, Docker build failure, or CI log. TokenSift keeps root-cause evidence and folds repetitive noise before you copy it into Codex, Claude Code, or another coding agent.

Paste raw error output

Try sample log:

Failure log → conservative brief.
Review before sending.

Conservative debugging brief

Debug CUDA OOM.

Type:
- CUDA / PyTorch OOM

Keep:
- cmd: python train.py --model llama-7b --batch-size 64 --precision fp16
- frame: /workspace/train.py:214 -> loss = trainer.step(batch)
- error: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.25 GiB.
- detail: model=llama-7b; batch-size=64; precision=fp16
- memory: GPU0: 23.65 GiB total; 1.08 GiB free; 21.40 GiB allocated; 22.49 GiB reserved.

Folded:
- progress x1; wandb x2; NCCL x2; worker INFO x2; dependency internals x1

Ask: root cause, smallest fix, verify command.

Detected type: ex) CUDA / PyTorch OOM
Input characters: ex) 683
Output characters: ex) 561
Input tokens: ex) 172 approx.
Output tokens: ex) 133 approx.
Reduction: ex) 18%
Noise removed summary: ex) progress, wandb, NCCL, and worker INFO folded
ex) torch dependency internals folded

Method

A debugging brief, not a lossy summary

1. Detect the failure shape

Deterministic rules distinguish Python tracebacks, CUDA OOMs, Docker builds, pytest output, TypeScript builds, and generic terminal failures.

2. Keep evidence with diagnostic value

The brief prioritizes the primary error, command, environment hints, user-code frames, memory details, failing build steps, assertions, and exit signals.

3. Fold repetition and show what changed

Progress output, duplicate warnings, dependency internals, cached layers, and tool chatter are folded into an explicit removal summary for review.

Read the complete deterministic pipeline and its limitations.

FAQ

Common questions

How do I use it?

Paste raw terminal, VS Code, or CI failure output, click Sift & copy, review the conservative Markdown brief, then send it to your LLM or coding agent.

What happens to my logs?

On the backend-hosted deployment, tokensift stores the raw input, generated output, IP address, and timestamp in a private first-party SQLite database for diagnostics and retains hosted raw submissions for up to 30 days by default. Raw logs are not sent to analytics or ads.

What should I paste?

Paste the raw failure output you would normally copy from iTerm2, VS Code, CI, Docker, pytest, npm, or a training run.

What does it preserve?

The primary error, command, environment lines, user frames, memory/type/assertion/build details, and a final debugging question.

What does it fold?

Token-eating junk: progress bars, repeated warnings, timestamps, dependency frames, cached layers, wandb/NCCL spam, and noisy build or CI output.

Does it fix code automatically?

No. It prepares a cleaner prompt for ChatGPT, Claude Code, Codex, OpenCode, or another LLM.

Coverage

Raw outputs tokensift understands

Python & pytestTracebacks, assertions, fixtures, captured logs, and user-code frames. CUDA / PyTorch OOMGPU memory stats, batch size, precision, model hints, and training frames. TypeScript buildsTS error codes, file paths, line/column hints, npm, Vite, and Next.js output. Docker buildsFailing steps, Dockerfile context, package-manager errors, and exit codes. pytest and CI logsAssertions, fixtures, warnings, timestamps, progress output, and noisy job chatter. Package installspip, npm, pnpm, yarn, apt, and poetry failures with the failing package and exit signal. Permission errorsDenied file access, missing environment variables, path issues, and timeout context. Generic terminal outputFatal, error, failed, and timeout output with nearby root-cause context.

Practical guides

Prepare evidence before asking a coding agent

Evidence-preserving briefsDecide what must survive compression and how to review a brief before sharing it. Prepare failure logsCollect the command, environment, failure boundary, and safe verification context. How TokenSift worksFollow normalization, redaction, detection, parser selection, folding, and rendering. Support matrixSee which signals each parser keeps and when the generic fallback is more appropriate.

Privacy-aware, prompt-first

tokensift focuses on turning raw failure logs into conservative, evidence-preserving debugging briefs before you hand them to a coding agent. Analytics and ads are disabled by default; if enabled later, only aggregate events are allowed. Backend submissions are stored by tokensift for diagnostics. Feedback: [email protected].