FSDP

Educational PyTorch repo for distributed training from scratch: DP, FSDP, TP, FSDP+TP, and PP [P]

This educational PyTorch repository implements various distributed training parallelism techniques, including DP, FSDP, TP, and PP, from scratch. It explicitly writes forward/backward logic and collectives, allowing users to directly understand the algorithms and communication patterns without high-level abstractions.