Skip to main content

Releasing MS-AMP v0.4

· One min read
Yuxiang Yang
MS-AMP Team

We are very happy to announce that MS-AMP 0.4.0 version is officially released today!

You can install and try MS-AMP by following Getting Started Tutorial.

MS-AMP 0.4.0 Release Notes#

MS-AMP Improvements#

  • Improve GPT-3 performance by optimizing the FP8-gradient accumulation with kernel fusion technology
  • Support FP8 in FSDP
  • Support DeepSpeed+TE+MSAMP and add cifar10 example
  • Support MSAMP+TE+DDP
  • Update DeepSpeed to latest version
  • Update TransformerEngin to V1.1 and flash-attn to latest version
  • Support CUDA 12.2
  • Fix several bugs in DeepSpeed integration

MS-AMP-Examples Improvements#

  • Improve document for data processing in GPT3
  • Add launch script for pretraining GPT-6b7
  • Use new API of TransformerEngine in Megatron-LM

Document Improvements#

  • Add docker usage in Installation page
  • Tell customer how to run FSDP and DeepSpeed+TE+MSAMP example in "Run Examples" page