Tried a replication here. The LayerNorm kernel is not numerically stable so cannot be counted as valid. They only test with zero mean and unit std, so the catas

Tried a replication here. The LayerNorm kernel is not numerically stable so cannot be counted as valid. They only test with zero mean and unit std, so the catastrophic cancellation doesn't show up until after. EDIT: looks like they've since generated another one that is numerically stable! great work