issues, chat FPS; optimisations

.scratch/distributed-inference-performance/PRD.md

@@ -0,0 +1,40 @@
+# PRD: Distributed inference performance
+
+## Problem
+
+Distributed decode already avoids full-prompt recomputation when the local KV
+path is active, but each Activation Seam can still pay transport and data-plane
+overhead for every generated token. Relay logs show a new `request_id` per
+token; that is correct correlation, but the old relay implementation also
+opened a new WebSocket per token. Direct hops and relay bridge forwarding use
+fresh HTTP requests as well. Without timing and byte measurements, compression,
+copy, and buffering choices cannot be ranked safely.
+
+## Outcome
+
+For a cached Route Session, connection setup is amortized across the session,
+decode payloads remain one-step activations, progress reporting is bounded, and
+the benchmark can attribute latency to model execution, serialization, relay,
+HTTP, queueing, and backpressure. Optimizations must preserve output tokens,
+KV semantics, failure behavior, and compatibility with legacy one-shot peers.
+
+## Non-goals
+
+- No speculative decoding or multi-token model execution in this feature.
+- No QUIC/WebRTC/custom transport rewrite.
+- No centralized Hot KV State.
+- No silent reuse of a `request_id`; each activation remains independently
+  traceable.
+
+## Acceptance criteria
+
+- A reproducible local two-node and relay benchmark reports per-token and
+  per-seam timing plus bytes.
+- Cached decode does not perform a new TCP/WebSocket connection per token.
+- Direct and relay-to-local HTTP paths reuse connections safely or document why
+  a path cannot do so.
+- Compression and copy decisions are based on recorded traces, not guesses.
+- Slow prefill consumers apply bounded backpressure rather than unbounded body
+  buffering.
+- A benchmark regression threshold catches a meaningful transport slowdown.
+

.scratch/distributed-inference-performance/README.md

@@ -0,0 +1,37 @@
+# Distributed inference performance
+
+Status: draft scratch package.
+
+This feature measures and reduces avoidable overhead around the existing
+Route Session, prefill/decode, local Hot KV State, and binary activation path.
+It does not replace the distributed GGUF runtime plan. The goal is to make
+transport and data movement cheap enough that model execution, rather than
+connection setup, logging, or serialization, dominates token latency.
+
+## Scope
+
+- Baseline per-token compute, seam, connection, serialization, and queue time.
+- Keep one connection alive for a Route Session wherever protocol semantics allow.
+- Add bounded, actionable Generation Telemetry for each Activation Seam.
+- Tune compression and buffer conversion from measured activation traces.
+- Add bounded prefill backpressure and an end-to-end benchmark gate.
+
+## Existing decisions preserved
+
+- `X-Meshnet-Session` is stable for one Route Session.
+- `request_id` remains unique per activation request for correlation.
+- Hot KV State remains local to each Shard node.
+- v1 activation transfer remains binary HTTP-shaped traffic.
+- Streaming output remains preferred and telemetry remains mandatory.
+
+## Task order
+
+1. 01 — baseline and profiling harness
+2. 02 — persistent relay compatibility hardening
+3. 03 — direct and bridge HTTP keep-alive
+4. 04 — seam telemetry and bounded progress reporting
+5. 05 — adaptive activation compression
+6. 06 — activation framing and copy reduction
+7. 07 — prefill chunk backpressure
+8. 08 — end-to-end performance gate
+

.scratch/distributed-inference-performance/issues/01-baseline-profiling-harness.md

@@ -0,0 +1,27 @@
+Status: ready-for-agent
+
+# 01 — Baseline and profiling harness
+
+## What to build
+
+Create a deterministic stub-backed benchmark for a Route Session that measures
+prefill and cached decode across direct and relay paths. Attribute time to model
+execution, activation encoding/decoding, compression, connection setup, relay
+queueing, local HTTP forwarding, and end-to-end seam latency. Record payload
+sizes and connection counts without requiring a real model or external host.
+
+## Acceptance criteria
+
+- [ ] The harness runs a fixed prompt and fixed generated-token count through a
+  two-node route in direct and relay modes.
+- [ ] It reports p50/p95 per-token latency, per-hop latency, payload bytes,
+  compression ratio, connection attempts, and queue wait.
+- [ ] It distinguishes prefill from decode and cached from stateless mode.
+- [ ] It emits machine-readable JSON suitable for CI artifacts and a concise
+  human-readable summary.
+- [ ] A test fixture can assert connection attempts and output token identity.
+
+## Blocked by
+
+None - can start immediately.
+

.scratch/distributed-inference-performance/issues/02-relay-session-compatibility.md

@@ -0,0 +1,29 @@
+Status: ready-for-agent
+
+# 02 — Persistent relay compatibility hardening
+
+## What to build
+
+Harden the persistent `/rpc/<peer>` connection used by one Route Session.
+Preserve unique request correlation while allowing sequential binary and JSON
+requests on one socket. Handle peer disconnects, requester cancellation,
+legacy one-request relays, timeout cleanup, and generation-end close without
+leaking pending RPC entries or accidentally replaying a model mutation.
+
+## Acceptance criteria
+
+- [ ] A cached decode session uses one requester connection per relay Activation
+  Seam and sends one unique request id per activation.
+- [ ] Legacy relays that close after one response fail over clearly without
+  corrupting the Route Session or replaying an uncertain request.
+- [ ] Relay and bridge cleanup removes pending request state on normal close,
+  cancellation, timeout, and peer disconnect.
+- [ ] Concurrent Route Sessions do not share a non-thread-safe socket; responses
+  remain matched by request id.
+- [ ] Tests cover two sequential binary requests, JSON compatibility, close,
+  timeout, disconnect, cancellation, and no leaked pending entries.
+
+## Blocked by
+
+- 01 — Baseline and profiling harness.
+

.scratch/distributed-inference-performance/issues/03-http-keepalive.md

@@ -0,0 +1,28 @@
+Status: ready-for-agent
+
+# 03 — Direct and bridge HTTP keep-alive
+
+## What to build
+
+Amortize TCP connection setup for direct node hops and for the relay bridge's
+local request into the shard server. Use bounded per-session or per-worker
+connection ownership, explicit response lengths, and safe invalidation on
+errors. Do not share a connection across concurrent requests unless the client
+supports serialization.
+
+## Acceptance criteria
+
+- [ ] Direct cached decode reuses a connection to each downstream HTTP node.
+- [ ] Relay bridge forwarding reuses loopback HTTP connections without blocking
+  unrelated worker requests.
+- [ ] HTTP/1.1 framing is correct for success, error, empty, streamed, and
+  cancellation responses; no request hangs waiting for EOF.
+- [ ] Broken or stale connections are discarded and the current request follows
+  the existing safe failure/fallback policy.
+- [ ] Benchmark 01 shows connection attempts are independent of generated token
+  count for a healthy session.
+
+## Blocked by
+
+- 01 — Baseline and profiling harness.
+

.scratch/distributed-inference-performance/issues/04-seam-telemetry.md

@@ -0,0 +1,28 @@
+Status: ready-for-agent
+
+# 04 — Activation Seam telemetry and bounded progress reporting
+
+## What to build
+
+Expose structured timing and byte counters for each Activation Seam while
+keeping per-token progress overhead bounded. Report route/session, phase,
+hop/node, queue wait, model time, encode/decode time, compression time, wire
+bytes, response bytes, and connection reuse. Aggregate high-cardinality events
+instead of flushing a log line for every token.
+
+## Acceptance criteria
+
+- [ ] Generation Telemetry includes prefill/decode seam latency and rolling
+  tokens/sec without changing token output or cache behavior.
+- [ ] Every request can be correlated by stable Route Session plus unique
+  activation request id.
+- [ ] Counters are sampled or aggregated so telemetry work is bounded and does
+  not perform network I/O in the model hot loop.
+- [ ] Logs summarize decode progress by session and retain actionable failure
+  context.
+- [ ] Tests verify counters, aggregation cadence, and cleanup at session close.
+
+## Blocked by
+
+- 01 — Baseline and profiling harness.
+

.scratch/distributed-inference-performance/issues/05-adaptive-compression.md

@@ -0,0 +1,26 @@
+Status: ready-for-agent
+
+# 05 — Trace-driven activation compression
+
+## What to build
+
+Make zstd decisions from activation size, measured compression ratio, CPU cost,
+and route conditions. Keep small decode payloads on the fast path, avoid
+compressing data that does not shrink, and expose enough counters to compare
+wire savings against compression latency on LAN and relay routes.
+
+## Acceptance criteria
+
+- [ ] A compression policy is explicit and configurable for LAN, relay, and
+  benchmark environments.
+- [ ] Bodies that do not meet the configured savings threshold are sent raw.
+- [ ] Compression and decompression time plus input/output bytes are reported.
+- [ ] Prefill and decode policies can differ; decode latency is not regressed by
+  compressing small one-step activations.
+- [ ] Tests cover incompressible, compressible, threshold, malformed, and
+  legacy-uncompressed bodies.
+
+## Blocked by
+
+- 01 — Baseline and profiling harness.
+

.scratch/distributed-inference-performance/issues/06-activation-framing-copies.md

@@ -0,0 +1,27 @@
+Status: ready-for-agent
+
+# 06 — Activation framing and copy reduction
+
+## What to build
+
+Profile and reduce avoidable allocations while activation data crosses a seam:
+binary frame assembly, header JSON, base64 metadata, CPU/GPU conversion, and
+response decompression. Preserve the current binary wire contract and use
+zero-copy or pooled buffers only where ownership and lifetime are explicit.
+
+## Acceptance criteria
+
+- [ ] The benchmark identifies copy/allocation cost separately from model and
+  network time.
+- [ ] Decode hidden-state conversion has no unnecessary float32 round trip.
+- [ ] Binary framing avoids base64 for activation bodies and does not retain
+  buffers after a request completes.
+- [ ] Position/attention metadata is validated and encoded efficiently without
+  changing semantic headers or cache positions.
+- [ ] A focused test proves byte-for-byte wire compatibility and stable output
+  tokens before and after the optimization.
+
+## Blocked by
+
+- 01 — Baseline and profiling harness.
+

.scratch/distributed-inference-performance/issues/07-prefill-backpressure.md

@@ -0,0 +1,27 @@
+Status: ready-for-agent
+
+# 07 — Bounded prefill chunk backpressure
+
+## What to build
+
+Make large prefill transfer bounded across every Activation Seam. Chunk prompt
+activations, limit in-flight chunks, and propagate downstream congestion so a
+slow Shard node cannot cause the head or relay bridge to buffer the entire
+context in memory. Keep decode-step traffic sequential and low-latency.
+
+## Acceptance criteria
+
+- [ ] Configurable prefill chunk size and in-flight limit exist with safe
+  defaults.
+- [ ] Peak per-hop buffered bytes are bounded by the configured limits.
+- [ ] A slow downstream stub applies backpressure and does not lose or reorder
+  chunks within a Route Session.
+- [ ] Cancellation and route failure release queued chunks and local buffers.
+- [ ] Tests cover small prompts, multi-chunk prompts, slow consumers, retry/fail
+  closeout, and legacy single-chunk peers.
+
+## Blocked by
+
+- 01 — Baseline and profiling harness.
+- 04 — Activation Seam telemetry and bounded progress reporting.
+

.scratch/distributed-inference-performance/issues/08-end-to-end-performance-gate.md

@@ -0,0 +1,33 @@
+Status: ready-for-agent
+
+# 08 — End-to-end distributed performance gate
+
+## What to build
+
+Turn the benchmark and optimizations into a repeatable performance gate for a
+small two-node route and a relay route. Compare stateless legacy mode, cached
+decode, direct HTTP, and persistent relay. Fail only on stable regressions and
+publish the measurements needed to decide whether further work belongs in
+transport, serialization, queueing, or model execution.
+
+## Acceptance criteria
+
+- [ ] CI/local benchmark runs a deterministic fixed-token scenario without a
+  real model or external network.
+- [ ] The report compares tokens/sec, p50/p95 token latency, seam latency,
+  bytes/token, connection count, compression CPU, and peak buffered bytes.
+- [ ] Thresholds are documented and tolerant of normal host variance while
+  catching a meaningful regression.
+- [ ] A real-model opt-in command records the same metrics for LAN validation.
+- [ ] The gate verifies output token identity, Route Session stability, and
+  cleanup of sessions, sockets, queues, and telemetry state.
+
+## Blocked by
+
+- 02 — Persistent relay compatibility hardening.
+- 03 — Direct and bridge HTTP keep-alive.
+- 04 — Activation Seam telemetry and bounded progress reporting.
+- 05 — Trace-driven activation compression.
+- 06 — Activation framing and copy reduction.
+- 07 — Bounded prefill chunk backpressure.
+

.scratch/distributed-inference-performance/prd.json

@@ -0,0 +1,135 @@
+{
+  "name": "Distributed inference performance",
+  "description": "Measure and reduce avoidable transport, HTTP, telemetry, compression, buffering, and copy overhead around Route Session cached decode.",
+  "branchName": "ralph/distributed-inference-performance",
+  "userStories": [
+    {
+      "id": "DIP-001",
+      "title": "01 — Baseline and profiling harness",
+      "description": "Create a deterministic stub-backed direct and relay Route Session benchmark that reports per-token and per-seam timing, bytes, compression, queueing, and connection counts.",
+      "acceptanceCriteria": [
+        "Fixed prompt/token scenario runs in direct and relay modes",
+        "Reports p50/p95 latency, payload bytes, compression ratio, connections, and queue wait",
+        "Distinguishes prefill/decode and cached/stateless modes",
+        "Produces machine-readable JSON and human-readable summary",
+        "Can assert connection count and output token identity"
+      ],
+      "priority": 1,
+      "passes": false,
+      "notes": "Source issue: .scratch/distributed-inference-performance/issues/01-baseline-profiling-harness.md",
+      "dependsOn": []
+    },
+    {
+      "id": "DIP-002",
+      "title": "02 — Persistent relay compatibility hardening",
+      "description": "Harden one persistent relay RPC connection per Route Session seam while preserving unique request IDs, legacy compatibility, cleanup, cancellation, and safe failure behavior.",
+      "acceptanceCriteria": [
+        "One healthy relay connection serves sequential cached decode requests",
+        "Legacy one-request relays fail over without replaying uncertain mutations",
+        "Pending RPC state is cleaned on close, cancellation, timeout, and disconnect",
+        "Concurrent sessions do not share an unsafe socket",
+        "Tests cover binary, JSON, timeout, disconnect, cancellation, and cleanup"
+      ],
+      "priority": 2,
+      "passes": false,
+      "notes": "Source issue: .scratch/distributed-inference-performance/issues/02-relay-session-compatibility.md",
+      "dependsOn": ["DIP-001"]
+    },
+    {
+      "id": "DIP-003",
+      "title": "03 — Direct and bridge HTTP keep-alive",
+      "description": "Amortize TCP setup for direct node hops and relay bridge loopback forwarding with bounded connection ownership and correct HTTP framing.",
+      "acceptanceCriteria": [
+        "Direct cached decode reuses downstream HTTP connections",
+        "Bridge loopback forwarding reuses connections without blocking unrelated workers",
+        "HTTP/1.1 framing works for success, error, empty, stream, and cancellation",
+        "Stale connections are invalidated under the existing fallback policy",
+        "Benchmark shows healthy-session connection count independent of token count"
+      ],
+      "priority": 3,
+      "passes": false,
+      "notes": "Source issue: .scratch/distributed-inference-performance/issues/03-http-keepalive.md",
+      "dependsOn": ["DIP-001"]
+    },
+    {
+      "id": "DIP-004",
+      "title": "04 — Activation Seam telemetry and bounded progress reporting",
+      "description": "Expose per-seam timing and byte counters while aggregating progress work so telemetry does not become per-token hot-loop overhead.",
+      "acceptanceCriteria": [
+        "Telemetry includes seam latency and rolling tokens/sec",
+        "Stable session and unique activation IDs remain correlated",
+        "Counters are sampled or aggregated without network I/O in model execution",
+        "Decode logs summarize by session with actionable failures",
+        "Tests verify cadence and cleanup"
+      ],
+      "priority": 4,
+      "passes": false,
+      "notes": "Source issue: .scratch/distributed-inference-performance/issues/04-seam-telemetry.md",
+      "dependsOn": ["DIP-001"]
+    },
+    {
+      "id": "DIP-005",
+      "title": "05 — Trace-driven activation compression",
+      "description": "Choose zstd based on measured savings and CPU cost, preserving a raw fast path for small or incompressible decode activations.",
+      "acceptanceCriteria": [
+        "Compression policy is explicit and configurable per route condition",
+        "Bodies below savings threshold are sent raw",
+        "Compression timing and byte counters are reported",
+        "Prefill and decode can use different policies",
+        "Tests cover compressible, incompressible, threshold, malformed, and legacy bodies"
+      ],
+      "priority": 5,
+      "passes": false,
+      "notes": "Source issue: .scratch/distributed-inference-performance/issues/05-adaptive-compression.md",
+      "dependsOn": ["DIP-001"]
+    },
+    {
+      "id": "DIP-006",
+      "title": "06 — Activation framing and copy reduction",
+      "description": "Measure and reduce avoidable binary framing, metadata, CPU/GPU conversion, and decompression allocations without changing the wire contract.",
+      "acceptanceCriteria": [
+        "Benchmark attributes copy/allocation cost separately",
+        "Decode hidden state avoids unnecessary float32 conversion",
+        "Activation bodies remain binary and buffers have explicit ownership",
+        "Metadata encoding remains semantically compatible",
+        "Wire and token-output regression tests pass"
+      ],
+      "priority": 6,
+      "passes": false,
+      "notes": "Source issue: .scratch/distributed-inference-performance/issues/06-activation-framing-copies.md",
+      "dependsOn": ["DIP-001"]
+    },
+    {
+      "id": "DIP-007",
+      "title": "07 — Bounded prefill chunk backpressure",
+      "description": "Bound large prefill transfer with configurable chunking and in-flight limits, propagating downstream congestion without reordering or unbounded buffering.",
+      "acceptanceCriteria": [
+        "Chunk size and in-flight limit have safe defaults",
+        "Peak buffered bytes stay within configured bounds",
+        "Slow consumers apply backpressure and preserve order",
+        "Cancellation and route failure release queued buffers",
+        "Tests cover chunking, slow consumers, failure, and legacy peers"
+      ],
+      "priority": 7,
+      "passes": false,
+      "notes": "Source issue: .scratch/distributed-inference-performance/issues/07-prefill-backpressure.md",
+      "dependsOn": ["DIP-001", "DIP-004"]
+    },
+    {
+      "id": "DIP-008",
+      "title": "08 — End-to-end distributed performance gate",
+      "description": "Make direct, persistent-relay, cached, and stateless benchmark comparisons repeatable and fail on meaningful transport regressions while verifying output and cleanup.",
+      "acceptanceCriteria": [
+        "Deterministic stub benchmark runs locally and in CI",
+        "Report compares throughput, latency, bytes, connections, compression CPU, and buffers",
+        "Regression thresholds tolerate host variance and catch meaningful slowdowns",
+        "Opt-in real-model LAN command emits the same metrics",
+        "Gate verifies token identity, session stability, and resource cleanup"
+      ],
+      "priority": 8,
+      "passes": false,
+      "notes": "Source issue: .scratch/distributed-inference-performance/issues/08-end-to-end-performance-gate.md",
+      "dependsOn": ["DIP-002", "DIP-003", "DIP-004", "DIP-005", "DIP-006", "DIP-007"]
+    }
+  ]
+}