How to keep a good performance balance between the sibling heads?

[makecent]

My detection model has two heads, one for the localization and the other for the classification. However, I found that it's hard to get both of them to be good.

For example, during the training, the classification metric (top-1 acc.) started to decline (overfitting) but the localization metric (IoU) was still rising (underfitting), resulting in a fluctuating detection performance (mAP).

Is there any good solution for this? I found adjusting the learning rates or the number of parameters of the two heads didn't work well.