DEV Community

Debashish Ghosal
Debashish Ghosal

Posted on

Codex Never Crashed. It Just Quietly Did Nothing.

Codex Never Crashed. It Just Quietly Did Nothing.

From the agentpolicy v0.1.0 release: a local-first, Cedar-based policy decision point for AI agents (decide allow, deny, ask, rate-limit, or redact for every agent action, with shipped rule packs). Now on PyPI. This story is journal Group 21 of the v0.1.0 field test report.

AI disclosure: this article was drafted and edited with AI assistance. The field test, the defects, the fixes, and every number are real and linked below.

Three bugs stood between me and a live Codex CLI integration. Not one of them printed an error. Every one of them let my harness sit there looking finished while producing nothing. This is the kind of debugging session that makes you stop trusting a quiet process.

Trap one: the API you assumed still exists

I wired Codex CLI against my local model server the way I had wired everything else: chat completions, the endpoint every OpenAI-compatible server speaks. Codex refused at startup: wire_api = "chat" is no longer supported. The CLI dropped chat completions entirely. The only transport it accepts now is the OpenAI Responses API.

The lesson is not "read the changelog," although yes. The lesson is that my local server, oMLX, happened to serve /v1/responses natively, and I only found that out with a curl probe instead of an assumption. One probe, a real resp_* body, HTTP 200. The entire integration unblocked with zero translation layers.

Vendor APIs drift. Probe the transport fresh for every CLI-to-endpoint pairing, every CLI version. Yesterday's working config is today's silent startup rejection.

Trap two: the stdin hang

codex exec, run headless from a test script, printed "Reading additional input from stdin..." and then sat there. No error. No timeout. No exit. My harness, which was watching the process and its output, looked "busy." The run produced zero evidence and appeared, from a process list, to be working.

The cause: codex exec reads stdin, and in a harness stdin is whatever the parent left connected, which is never a closing TTY. The fix is two characters of shell: </dev/null. Close stdin by default in every headless invocation.

This one deserves a rule of its own: any headless CLI that consumes stdin will hang your harness quietly. An unattended process with a blocked read is the most patient bug in the world. It will wait forever, and your CI will show a spinner.

Trap three: the 404s that blamed the server

With transport sorted and stdin closed, every model call returned 404. Uniform, instant 404s. I spent a while on the server side, checking routes, because 404 says "server problem," right?

Codex appends /responses to the provider base_url I configured. My proxy URL was http://127.0.0.1:PORT, so Codex called /responses. The server serves /v1/responses. The fix: base_url = "http://127.0.0.1:PORT/v1". Off by one suffix, every call dead, server completely healthy the whole time.

Paths are config. Config is a test surface. A uniform 404 across every request is almost never "the server is broken." It is "the client is asking somewhere else."

The honest scoping problem

There was a fourth decision, not a bug, a design call. Claude Code has a pausable per-call approval hook. I could wire the policy engine directly into the ask-a-human loop and test it live. Codex, headless, does not. Its governance surface is the sandbox and the config file: --sandbox read-only, provider config, executed commands.

I could have forced parity and written a fake approval flow. Instead the case tests what the surface actually guarantees. Allow means the command executed, a real cat that printed real output. Deny means the destructive command was blocked, the target directory stayed intact with its keep.txt, and the agent replanned. Plus a parity check: the same three commands decided by the engine before execution, with an audit file showing allow, escalate, deny matching the in-process engine exactly.

A harness test should assert what the surface guarantees and say so out loud. Stretching a different harness's shape over a vendor surface produces tests that pass and prove nothing.

The final wiring, for the record

Codex CLI 0.135.0 against the box-local model, no OpenAI key. A stdlib logging proxy (t1_proxy.py) between the CLI and the model server captured 4 request and 4 response pairs, sanitized, all 2xx. The parity audit trail has 3 decisions. The allow executed, the deny held and replanned, and the whole thing reproduces from the committed config (t1_codex.sh, codex-home/config.toml in the case's evidence directory).

What worked

  • Probing before wiring. The Responses API question was settled with one curl before any config existed.
  • Capturing LLM traffic as a hard requirement. Every pair on disk or the case fails. That rule is the only reason "looked busy, did nothing" runs got caught at all.
  • Testing the pieces standalone before composing them: policy through evaluate first, then the proxy with curl, then the CLI through the proxy. Each failure lived in exactly one layer.

What did not work

  • Trusting that chat completions would still be the transport. It was not, and finding out cost me a full debug cycle.
  • Leaving stdin connected. Obvious in hindsight, invisible in the process list.
  • Reading 404 as "server broken" before reading it as "client asking in the wrong place."

More detail and evidence

  • The full Codex story: field test report, journal Group 21 (FT-T1-CODEX), plus Field Notes 12 for the write-up version of the same traps.
  • The harness, in the repo: t1_codex.sh (config, parity triple, sandbox triple) · t1_proxy.py (the /responses-aware capture proxy) · t1_policy.yaml (the shared Tier-1 policy).
  • The captured evidence: field-test/v0.1.0/results/s18-tier1/cases/FT-T1-CODEX/ in the repo (live transcripts, audit trail, 4 LLM pairs, provider config).
  • The plan row that scoped the case honestly: FIELD_TEST_PLAN.md, the FT-T1-CODEX row and the Tier-1 R-map.
  • The product: agentpolicy, pip install agentsec-agentpolicy, PyPI 0.1.0, docs reference (adapters, authzen, surfaces), part of the agentic-suite agent-security stack.

What is the most patient bug you have shipped in a harness? The one that produced no output, no error, and no exit, just quiet nothing. I want to know I am not alone.

Top comments (2)

Collapse
 
piekwerk profile image
Piekwerk •

Mine was a loop over a host list that ran the first host and silently skipped the rest. ssh was eating the remaining lines off stdin, the loop hit EOF looking done, no error, no nonzero exit. ssh -n fixed it, same shape as your stdin trap, fd 0 was connected and something consumed it.

Two habits catch this class for me now. Every headless call gets a hard timeout with a kill signal, a blocked read never heals itself. And a step writing zero bytes for a few minutes fails the run, quiet nothing is a failure state, not busy.

Your base_url trap is the one I keep relearning, print the exact assembled URL before touching the server.

Collapse
 
thezerodollardev profile image
Aditya Vishwakarma •

Silent failure modes are by far the most insidious part of building agent harnesses.

We hit this exact phenomenon when profiling local model servers under agent loops:

  1. The transport handshake looks healthy, but an unhandled context-overflow doesn't throw an HTTP error — it silently falls back to CPU RAM. The agent looks 'hanging' for 90 seconds while CPU inference crawls at 1.5 tok/s with zero terminal warnings.
  2. The agent expects a structured JSON envelope, but gets truncated markdown when the context boundary is hit, producing a null action without an exception.

Building deterministic policy decision points (like Cedar) before the model dispatches tool calls is a massive step forward. Have you considered adding a pre-flight token budget / context threshold check to the policy rules?