popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

dash

Browse Source
This commit is contained in:
Dobromir Popov
2026-07-08 18:24:45 +02:00
parent e06969fcb5
commit 29db25108f
3 changed files with 112 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files

77
QUICKSTART.md
View File

@@ -142,40 +142,52 @@ does not need any of this — it is a standard transformer with no FLA fast path
- **transformers ≥ 5.12 required** — older versions fail with
`'Qwen3_5MoeConfig' object has no attribute 'vocab_size'`. Check:
`python -c "import transformers; print(transformers.__version__)"`.
- **Optional GPU kernels** — without them inference still works; startup may print
`The fast path is not available…` (harmless on Windows; slower on linear-attention
layers). Install **only for your platform**:
- **GPU fast path (optional)** — without it inference still works; startup prints
`The fast path is not available…` and linear-attention layers use a slower PyTorch
fallback. Install **only for your platform**:
| Platform | Install | Fast path? |
|----------|---------|------------|
| **Native Windows** | `pip install triton-windows` (also pulled by `meshnet-node` on Windows) | **Load only** — model imports and runs on PyTorch fallback. Do **not** `pip install flash-linear-attention[cuda] causal-conv1d` (see below). |
| **Linux + NVIDIA CUDA** | `pip install flash-linear-attention[cuda]` | **Yes** — `causal-conv1d` optional since FLA ≥0.3.2 ships Triton conv1d; add it only if transformers still asks for it. Needs CUDA toolkit (`nvcc`) or a matching prebuilt wheel. |
| **Linux + AMD ROCm** | `pip install flash-linear-attention[rocm]` | **Yes** — same optional `causal-conv1d` note. |
| Platform | Install | Notes |
|----------|---------|-------|
| **Native Windows + NVIDIA** | `pip install triton-windows` then `pip install flash-linear-attention` | **Fast path works.** FLA [officially supports `triton-windows`](https://github.com/fla-org/flash-linear-attention/pull/757) (tested Win11, PyTorch 2.10, triton-windows 3.6). Do **not** use the `[cuda]` extra on Windows — pip looks for Linux `triton` and fails. Do **not** install `causal-conv1d` — FLA ≥0.3.2 ships Triton conv1d; the separate package is Linux-only and breaks on Windows (`bare_metal_version` / nvcc errors). |
| **Linux + NVIDIA CUDA** | `pip install flash-linear-attention[cuda]` | `causal-conv1d` optional (same FLA built-in conv1d note). Needs CUDA toolkit (`nvcc`) matching torch, or a prebuilt wheel. |
| **Linux + AMD ROCm** | `pip install flash-linear-attention[rocm]` | Same optional `causal-conv1d` note. |
On native Windows, `triton-windows` is enough for Qwen3.6-MoE to **load**. Without it,
**Windows verify** (after install):
```powershell
python -c "import triton; import fla; print('triton', triton.__version__, 'fla ok')"
```
`triton-windows` is also pulled by `meshnet-node` on Windows. Without it, Qwen3.6-MoE
startup fails with misleading `Could not import module 'Qwen3_5MoeForCausalLM'`.
<details>
<summary><strong>Why there is no supported Windows fast path (research notes)</strong></summary>
<summary><strong>Windows fast path — what failed and what actually works</strong></summary>
We checked upstream docs and real install attempts (2026-07):
The command that failed — `pip install flash-linear-attention[cuda] causal-conv1d` — mixes
two different things:
1. **`causal-conv1d`** — no official Windows wheels on PyPI. Source builds need `nvcc`
plus a torch/CUDA stack that matches; without `nvcc` pip fails with
`bare_metal_version is not defined`. Open upstream issues confirm Windows is
unsupported / broken for most setups.
2. **`flash-linear-attention[cuda]`** — FLA's install guide targets Linux (CUDA / ROCm /
XPU / NPU). The `[cuda]` extra pulls PyPI `triton`, which conflicts with
`triton-windows` on native Windows. FLA does not document or CI-test Windows.
3. **What works today on native Windows GPU** — CUDA torch + `triton-windows` → Qwen3.6
loads and infers via Transformers' pure-PyTorch fallback (~¾ of layers are
linear-attention; those hops are slower). This is what we use in alpha.
4. **If you need the fast path on a Windows PC** — run the GPU node in **WSL2 with Linux
CUDA** (or a Linux box): `pip install flash-linear-attention[cuda]` there. Keep
Windows native for reachability / relay if needed.
5. **Experimental / unsupported** — community `causal-conv1d` wheels and
`triton-windows` + pinned `flash-linear-attention --no-deps` hacks exist for narrow
Python/torch/CUDA combos; we do not support or test these for meshnet-node.
1. **`flash-linear-attention[cuda]` on Windows** — wrong extra. `[cuda]` pulls PyPI
`triton>=3.3`, which does not exist for Windows (`No matching distribution found`).
Use plain `pip install flash-linear-attention` **after** `triton-windows` is already
installed; FLA detects `triton-windows` and uses it.
2. **`causal-conv1d`** — separate Dao-AILab CUDA extension, **not required** for FLA or
Qwen3.6 when FLA is installed. No official Windows wheels. Source builds need `nvcc`
whose major version matches torch's CUDA (e.g. torch `+cu118` needs CUDA 11.8 toolkit,
not 12.5). Community wheels exist for narrow Python/torch combos
([PR #46](https://github.com/Dao-AILab/causal-conv1d/pull/46)) but we skip them.
**Working Windows stack** (confirmed on this repo's dev machine: Python 3.12, torch
2.7.1+cu118, triton-windows 3.7.1, flash-linear-attention 0.5.0):
```powershell
pip install triton-windows
pip install -U flash-linear-attention
python -c "import triton; import fla; print('ok')"
```
If the fast-path warning persists after that, upgrade FLA to ≥0.5.1 (includes the
`triton-windows` detection from PR #757) and restart the node.
</details>
@@ -289,9 +301,8 @@ curl -sI https://ai.neuron.d-popov.com/rpc/test-peer
no HuggingFace gating. Best for first-time setup.
**Alpha model:** `qwen3.6-35b-a3b` — 40 layers, ~72 GB BF16 download, MoE with hybrid
linear attention. Use on machines with enough RAM/VRAM; see Qwen3.5/3.6 notes above for
`triton-windows` (Windows) or `flash-linear-attention` (Linux GPU). Tracker accepts the
alias or full repo id (`unsloth/Qwen3.6-35B-A3B`).
linear attention. On Windows install `triton-windows` + `flash-linear-attention`; on Linux
GPU use `flash-linear-attention[cuda]`. Tracker accepts the alias or full repo id (`unsloth/Qwen3.6-35B-A3B`).
Downloads cache under `~/.meshnet/models/` (or `$HF_HOME` / `$env:HF_HOME`).
@@ -330,13 +341,17 @@ HF_HOME=/path/to/models .venv/bin/meshnet-node start --tracker http://localhost:
.venv/bin/meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
```
**Alpha model (Qwen3.6, Windows GPU — PyTorch fallback, no FLA fast path):**
**Alpha model (Qwen3.6, Windows GPU — enable fast path):**
```powershell
$env:HF_HOME = "D:\DEV\models"
pip install triton-windows
pip install -U flash-linear-attention
meshnet-node start --tracker http://192.168.0.179:8080 --model qwen3.6-35b-a3b --quantization bfloat16
```
Do not add `causal-conv1d` or `flash-linear-attention[cuda]` on Windows (see Qwen3.5/3.6 notes).
**Alpha model (Qwen3.6, Linux GPU — with fast path):**
```bash