Can PLD technology be used on the Megatron-LM?

[chinoll]

Can PLD technology be used on the Megatron-LM?

[minjiaz]

Hi @chinoll, thank you for asking. From the idea perspective, I remember we tested PLD with GPT-2, so it should also work with GPT-like models in Megatron-LM in theory. However, it was done long ago and it was not based on the Megatron-LM code base, so we do not have any released code for this. On the other hand, the underlying parallelism strategies also matter when considering using PLD. Which parallelism strategies are you using in Megatron-LM?

[chinoll]

Hi @chinoll, thank you for asking. From the idea perspective, I remember we tested PLD with GPT-2, so it should also work with GPT-like models in Megatron-LM in theory. However, it was done long ago and it was not based on the Megatron-LM code base, so we do not have any released code for this. On the other hand, the underlying parallelism strategies also matter when considering using PLD. Which parallelism strategies are you using in Megatron-LM?

Hi @minjiaz, I use model parallelism，mp_size = 2.
How do I use PLD technology? When I enable PLD technology in the configuration file, deepspeed reports an error. Do I need to modify the source code of Megatron-LM? If I need to change the source code, what should I do?