neuron-tai/.scratch/distributed-inference-network/issues/05-openai-compatible-gateway.md

Status: ready-for-agent

05 — OpenAI-compatible gateway

What to build

Expose a production-shape HTTP API from the gateway so that any client using the OpenAI Python SDK (or any OpenAI-compatible tool) works by changing only the base_url. The gateway translates OpenAI-format requests into inference route execution and streams responses back in OpenAI-format.

Endpoints to implement:

• POST /v1/chat/completions — streaming (text/event-stream) and non-streaming
• GET /v1/models — returns the list of model presets currently routable on the network
• GET /v1/health — liveness check

The gateway selects an inference route from the tracker, executes the shard pipeline, and assembles the streamed response. If the tracker returns no route for the requested model, the gateway responds with a standard OpenAI-format error (model_not_available).

Authentication (API key → SOL/USDC balance) is a stub in this issue — return 200 for any non-empty Authorization header. Real payment gating comes in issue 06.

Acceptance criteria

• openai.OpenAI(base_url="http://localhost:8080/v1", api_key="test").chat.completions.create(model="stub-model", messages=[...]) returns a valid response
• Streaming works: stream=True returns text/event-stream chunks in OpenAI SSE format
• GET /v1/models returns a JSON array of available model preset names
• A request for an unavailable model returns an OpenAI-format error response with HTTP 503
• LangChain ChatOpenAI(base_url=..., api_key=...) works against the gateway
• An integration test covers streaming and non-streaming paths end-to-end through a real tracker and two stub nodes

Blocked by

• 03-tracker-registration-and-routing.md