Drop in performance when changing dtype to float32

[blaisedelattre]

Hello everyone, has anyone noticed a drop in performance (test and validation loss) when training with dtype=float32 ? I'm doing training on the Shakespeare dataset with the train_shakespeare_char config file.

I have not changed anything but dtype from the original repo.

It seems related to the use of "torch.amp.autocast" but I don't understand why a higher precision from bfloat16 to float32 would cause a drop in perf.

Thank you for your help !

[kalgoritmi]

It is just a speculation, but maybe it introduces some form of regularization in the model, given the small size of the data and the model it may actually allow it to generalize better?

[vnsmv]

Hello!
Can you please provide some more context about you problem?
Its possible to share you code and your test/validation losses?