This commit is contained in:
Dobromir Popov
2026-07-08 20:01:31 +02:00
parent 1ecc599f7f
commit 0b39d80375
3 changed files with 20 additions and 8 deletions

View File

@@ -144,11 +144,14 @@ does not need any of this — it is a standard transformer with no FLA fast path
`python -c "import transformers; print(transformers.__version__)"`. `python -c "import transformers; print(transformers.__version__)"`.
- **GPU fast path (optional)** — without it inference still works; startup prints - **GPU fast path (optional)** — without it inference still works; startup prints
`The fast path is not available…` and linear-attention layers use a slower PyTorch `The fast path is not available…` and linear-attention layers use a slower PyTorch
fallback. Install **only for your platform**: fallback. **The fast path runs on NVIDIA CUDA GPUs on both Linux and native
Windows** — the FLA kernels are Triton-compiled, and `triton-windows` compiles them
for CUDA on Windows just like Linux Triton does. Only the pip command differs per
platform. Install **only for your platform**:
| Platform | Install | Notes | | Platform | Install | Notes |
|----------|---------|-------| |----------|---------|-------|
| **Native Windows + NVIDIA** | `pip install triton-windows` then `pip install flash-linear-attention` | **Fast path works.** FLA [officially supports `triton-windows`](https://github.com/fla-org/flash-linear-attention/pull/757) (tested Win11, PyTorch 2.10, triton-windows 3.6). Do **not** use the `[cuda]` extra on Windows — pip looks for Linux `triton` and fails. Do **not** install `causal-conv1d` — FLA ≥0.3.2 ships Triton conv1d; the separate package is Linux-only and breaks on Windows (`bare_metal_version` / nvcc errors). | | **Native Windows + NVIDIA CUDA** | `pip install triton-windows` then `pip install flash-linear-attention` | **Fast path works on the CUDA GPU** — no CUDA toolkit / `nvcc` needed; `triton-windows` bundles its own compiler. FLA [officially supports `triton-windows`](https://github.com/fla-org/flash-linear-attention/pull/757) (tested Win11, PyTorch 2.10, triton-windows 3.6). Do **not** use the `[cuda]` extra on Windows — that extra only pins Linux PyPI `triton` and fails; it is a packaging name, not a GPU requirement. Do **not** install `causal-conv1d` — FLA ≥0.3.2 ships Triton conv1d; the separate package is Linux-only and breaks on Windows (`bare_metal_version` / nvcc errors). |
| **Linux + NVIDIA CUDA** | `pip install flash-linear-attention[cuda]` | `causal-conv1d` optional (same FLA built-in conv1d note). Needs CUDA toolkit (`nvcc`) matching torch, or a prebuilt wheel. | | **Linux + NVIDIA CUDA** | `pip install flash-linear-attention[cuda]` | `causal-conv1d` optional (same FLA built-in conv1d note). Needs CUDA toolkit (`nvcc`) matching torch, or a prebuilt wheel. |
| **Linux + AMD ROCm** | `pip install flash-linear-attention[rocm]` | Same optional `causal-conv1d` note. | | **Linux + AMD ROCm** | `pip install flash-linear-attention[rocm]` | Same optional `causal-conv1d` note. |

View File

@@ -111,10 +111,10 @@ with `'Qwen3_5MoeConfig' object has no attribute 'vocab_size'`). If you install
into an existing conda/miniforge env instead of a fresh venv, run into an existing conda/miniforge env instead of a fresh venv, run
`pip install -U transformers` there. The startup warning about `pip install -U transformers` there. The startup warning about
`flash-linear-attention` / `causal-conv1d` ("fast path is not available") is `flash-linear-attention` / `causal-conv1d` ("fast path is not available") is
harmless on CPU — those are optional CUDA-only kernels. harmless on CPU — those are optional GPU kernels.
If you run the node from native Windows instead of WSL2, install the Triton shim If you run the node from native Windows instead of WSL2, install Triton for
in the same environment: Windows in the same environment:
```powershell ```powershell
python -m pip install triton-windows python -m pip install triton-windows
@@ -123,6 +123,12 @@ python -m pip install triton-windows
Without it, Qwen3.5/3.6-MoE startup can fail with the misleading message Without it, Qwen3.5/3.6-MoE startup can fail with the misleading message
`Could not import module 'Qwen3_5MoeForCausalLM'`. `Could not import module 'Qwen3_5MoeForCausalLM'`.
**NVIDIA GPU on native Windows:** the CUDA fast path works — after
`triton-windows`, install FLA with plain `pip install flash-linear-attention`
(no `[cuda]` extra, no `causal-conv1d`; both are Linux-only packaging and fail
on Windows). No CUDA toolkit / `nvcc` is needed. See the platform table in
[QUICKSTART.md](../QUICKSTART.md#qwen3536-moe-notes) for details.
--- ---
## Step 6 — Pre-download the model shard ## Step 6 — Pre-download the model shard

View File

@@ -829,7 +829,8 @@ def run_startup(
shard_label = _format_shard_label(shard_start, shard_end, total_layers) shard_label = _format_shard_label(shard_start, shard_end, total_layers)
public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host) public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
endpoint = f"http://{public_host}:{actual_port}" endpoint = f"http://{public_host}:{actual_port}"
node.set_advertised_endpoint(endpoint) if hasattr(node, "set_advertised_endpoint"):
node.set_advertised_endpoint(endpoint)
local_base_url = f"http://127.0.0.1:{actual_port}" local_base_url = f"http://127.0.0.1:{actual_port}"
relay_bridge, relay_fields = _start_relay_bridge_if_available( relay_bridge, relay_fields = _start_relay_bridge_if_available(
tracker_url, tracker_url,
@@ -978,7 +979,8 @@ def run_startup(
actual_port = node.start() actual_port = node.start()
public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host) public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
endpoint = f"http://{public_host}:{actual_port}" endpoint = f"http://{public_host}:{actual_port}"
node.set_advertised_endpoint(endpoint) if hasattr(node, "set_advertised_endpoint"):
node.set_advertised_endpoint(endpoint)
local_base_url = f"http://127.0.0.1:{actual_port}" local_base_url = f"http://127.0.0.1:{actual_port}"
relay_bridge, relay_fields = _start_relay_bridge_if_available( relay_bridge, relay_fields = _start_relay_bridge_if_available(
tracker_url, tracker_url,
@@ -1156,7 +1158,8 @@ def run_startup(
shard_label = f"{shard_label} (pinned)" shard_label = f"{shard_label} (pinned)"
public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host) public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
endpoint = f"http://{public_host}:{actual_port}" endpoint = f"http://{public_host}:{actual_port}"
node.set_advertised_endpoint(endpoint) if hasattr(node, "set_advertised_endpoint"):
node.set_advertised_endpoint(endpoint)
local_base_url = f"http://127.0.0.1:{actual_port}" local_base_url = f"http://127.0.0.1:{actual_port}"
relay_bridge, relay_fields = _start_relay_bridge_if_available( relay_bridge, relay_fields = _start_relay_bridge_if_available(
tracker_url, tracker_url,