opcode84

4 ms·

1.
Nvidia trains 10T model in 4 bit precision (NVFP4)(developer.nvidia.com)
9 points by opcode84 1y ago|3 comments
2.
by opcode84 1y ago
For narrow-precision formats to be practical in large-scale pretraining, they must ensure both model accuracy and stable convergence. To assess the viability of 4-bit precision in large-scale model training, experiments were conducted with
3.
Nvidia announces 4-bit training with NVFP4(developer.nvidia.com)
2 points by opcode84 1y ago|1 comments
4.
by opcode84 1y ago
A version of the 12B Hybrid Mamba-Transformer model was initially trained with 8-bit precision—FP8, which has been shown in previous studies to closely match 16-bit precision, and hence served as our baseline for comparison. We then success
5.
Blackwell Brings Native Support for NVFP4: 4-Bit Inference with High Accuracy(developer.nvidia.com)
1 points by opcode84 1y ago|1 comments
6.
by opcode84 1y ago
Introducing NVFP4 for Efficient and Accurate Low-Precision Inference
7.
New microscaling data formats supported in Blackwell and Triton(triton-lang.org)
3 points by opcode84 2y ago|1 comments
8.
by opcode84 2y ago
The formats supported in the tutorial are the OCP microscaling formats, including mxfp4 and mxfp8, as well as NVIDIA’s nvfp4 format. These matrix multiplications are accelerated by fifth generation tensor core instructions on CUDA devices w
9.
NVIDIA confirms Blackwell support for OCP microscaling formats MXFP8, MXFP4(developer.nvidia.com)
4 points by opcode84 2y ago|1 comments
10.
by opcode84 2y ago
From the article: "NVIDIA Blackwell introduces revolutionary block-scaled floating point formats, including the Open Computing Project’s microscaling formats, which Triton now unlocks for NVIDIA Blackwell-powered hardware acceleration.
11.
by opcode84 3y ago
https://arxiv.org/pdf/2310.10537.pdf Discussed in a previous post https://news.ycombinator.com/item?id=37930663
12.
by opcode84 3y ago
The Microsoft chip has MX data types: https://news.ycombinator.com/item?id=37930663 https://arxiv.org/abs/2310.10537
13.
by opcode84 3y ago
Earlier this year, AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Technologies, Inc. formed the Microscaling Formats (MX) Alliance with the goal of creating and standardizing next-generation 6- and 4-bit data types for AI training a
14.
Standardizing next-generation narrow precision data formats for AI(opencompute.org)
103 points by opcode84 3y ago|52 comments