6 lines
773 B
Markdown
6 lines
773 B
Markdown
# HuggingFace bootstrap + P2P propagation for shard distribution
|
|
|
|
When the tracker assigns a node a new shard, the node downloads it from HuggingFace (or our seed servers as fallback). Once downloaded, the node joins the P2P shard swarm for that shard and seeds to other nodes that need it. Popular shards propagate virally; new shards bootstrap from HF.
|
|
|
|
This mirrors how BitTorrent works in practice: a seed server provides the first copy, P2P takes over distribution as the swarm grows. Having nodes bring their own weights (manual download) makes tracker-driven shard rebalancing impossible — you can't instruct a node to switch shards if it doesn't have the weights. Pure P2P with no bootstrap source means the first node for any shard has nowhere to download from.
|