md
This commit is contained in:
@@ -144,11 +144,14 @@ does not need any of this — it is a standard transformer with no FLA fast path
|
||||
`python -c "import transformers; print(transformers.__version__)"`.
|
||||
- **GPU fast path (optional)** — without it inference still works; startup prints
|
||||
`The fast path is not available…` and linear-attention layers use a slower PyTorch
|
||||
fallback. Install **only for your platform**:
|
||||
fallback. **The fast path runs on NVIDIA CUDA GPUs on both Linux and native
|
||||
Windows** — the FLA kernels are Triton-compiled, and `triton-windows` compiles them
|
||||
for CUDA on Windows just like Linux Triton does. Only the pip command differs per
|
||||
platform. Install **only for your platform**:
|
||||
|
||||
| Platform | Install | Notes |
|
||||
|----------|---------|-------|
|
||||
| **Native Windows + NVIDIA** | `pip install triton-windows` then `pip install flash-linear-attention` | **Fast path works.** FLA [officially supports `triton-windows`](https://github.com/fla-org/flash-linear-attention/pull/757) (tested Win11, PyTorch 2.10, triton-windows 3.6). Do **not** use the `[cuda]` extra on Windows — pip looks for Linux `triton` and fails. Do **not** install `causal-conv1d` — FLA ≥0.3.2 ships Triton conv1d; the separate package is Linux-only and breaks on Windows (`bare_metal_version` / nvcc errors). |
|
||||
| **Native Windows + NVIDIA CUDA** | `pip install triton-windows` then `pip install flash-linear-attention` | **Fast path works on the CUDA GPU** — no CUDA toolkit / `nvcc` needed; `triton-windows` bundles its own compiler. FLA [officially supports `triton-windows`](https://github.com/fla-org/flash-linear-attention/pull/757) (tested Win11, PyTorch 2.10, triton-windows 3.6). Do **not** use the `[cuda]` extra on Windows — that extra only pins Linux PyPI `triton` and fails; it is a packaging name, not a GPU requirement. Do **not** install `causal-conv1d` — FLA ≥0.3.2 ships Triton conv1d; the separate package is Linux-only and breaks on Windows (`bare_metal_version` / nvcc errors). |
|
||||
| **Linux + NVIDIA CUDA** | `pip install flash-linear-attention[cuda]` | `causal-conv1d` optional (same FLA built-in conv1d note). Needs CUDA toolkit (`nvcc`) matching torch, or a prebuilt wheel. |
|
||||
| **Linux + AMD ROCm** | `pip install flash-linear-attention[rocm]` | Same optional `causal-conv1d` note. |
|
||||
|
||||
|
||||
@@ -111,10 +111,10 @@ with `'Qwen3_5MoeConfig' object has no attribute 'vocab_size'`). If you install
|
||||
into an existing conda/miniforge env instead of a fresh venv, run
|
||||
`pip install -U transformers` there. The startup warning about
|
||||
`flash-linear-attention` / `causal-conv1d` ("fast path is not available") is
|
||||
harmless on CPU — those are optional CUDA-only kernels.
|
||||
harmless on CPU — those are optional GPU kernels.
|
||||
|
||||
If you run the node from native Windows instead of WSL2, install the Triton shim
|
||||
in the same environment:
|
||||
If you run the node from native Windows instead of WSL2, install Triton for
|
||||
Windows in the same environment:
|
||||
|
||||
```powershell
|
||||
python -m pip install triton-windows
|
||||
@@ -123,6 +123,12 @@ python -m pip install triton-windows
|
||||
Without it, Qwen3.5/3.6-MoE startup can fail with the misleading message
|
||||
`Could not import module 'Qwen3_5MoeForCausalLM'`.
|
||||
|
||||
**NVIDIA GPU on native Windows:** the CUDA fast path works — after
|
||||
`triton-windows`, install FLA with plain `pip install flash-linear-attention`
|
||||
(no `[cuda]` extra, no `causal-conv1d`; both are Linux-only packaging and fail
|
||||
on Windows). No CUDA toolkit / `nvcc` is needed. See the platform table in
|
||||
[QUICKSTART.md](../QUICKSTART.md#qwen3536-moe-notes) for details.
|
||||
|
||||
---
|
||||
|
||||
## Step 6 — Pre-download the model shard
|
||||
|
||||
@@ -829,7 +829,8 @@ def run_startup(
|
||||
shard_label = _format_shard_label(shard_start, shard_end, total_layers)
|
||||
public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
|
||||
endpoint = f"http://{public_host}:{actual_port}"
|
||||
node.set_advertised_endpoint(endpoint)
|
||||
if hasattr(node, "set_advertised_endpoint"):
|
||||
node.set_advertised_endpoint(endpoint)
|
||||
local_base_url = f"http://127.0.0.1:{actual_port}"
|
||||
relay_bridge, relay_fields = _start_relay_bridge_if_available(
|
||||
tracker_url,
|
||||
@@ -978,7 +979,8 @@ def run_startup(
|
||||
actual_port = node.start()
|
||||
public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
|
||||
endpoint = f"http://{public_host}:{actual_port}"
|
||||
node.set_advertised_endpoint(endpoint)
|
||||
if hasattr(node, "set_advertised_endpoint"):
|
||||
node.set_advertised_endpoint(endpoint)
|
||||
local_base_url = f"http://127.0.0.1:{actual_port}"
|
||||
relay_bridge, relay_fields = _start_relay_bridge_if_available(
|
||||
tracker_url,
|
||||
@@ -1156,7 +1158,8 @@ def run_startup(
|
||||
shard_label = f"{shard_label} (pinned)"
|
||||
public_host = advertise_host or (socket.getfqdn() if host == "0.0.0.0" else host)
|
||||
endpoint = f"http://{public_host}:{actual_port}"
|
||||
node.set_advertised_endpoint(endpoint)
|
||||
if hasattr(node, "set_advertised_endpoint"):
|
||||
node.set_advertised_endpoint(endpoint)
|
||||
local_base_url = f"http://127.0.0.1:{actual_port}"
|
||||
relay_bridge, relay_fields = _start_relay_bridge_if_available(
|
||||
tracker_url,
|
||||
|
||||
Reference in New Issue
Block a user