I've just released awesome-arXiv, a curated collection of tools, libraries, datasets, and resources for discovering, reading, and automating your work with arXiv papers.
Recently, we open-sourced YaFSDP, Yandex’s tool for efficient distributed training of LLMs.
Here are some of the key ideas used in YaFSDP to provide speedup and memory savings over FSDP: • Allocate and utilize just two buffers throughout the transformer for all collected weights to circumvent the torch memory allocator; • Gather small normalization layers at the beginning of the iteration and average the gradients only at the end; • Move gradient division to the very end of the backward pass.
