MoELorA

[moghadas76]

Feature request

Feature request
The paper "MoELoRA: Contrastive Learning Guided Mixture of Experts on
Parameter-Efficient Fine-Tuning for Large Language Models" introduced MoLoRA, a Mixutre-of-Experts approach using LoRA adapters. I am using it to conduct some research for my MSc thesis, and have implemented it in peft. I was wondering if this method is interesting and would be worth it to clean up my code and submit a PR.

Motivation
The motivation is to include more PEFT methods that the community can benefit from.

Your contribution
I can contribute a PR with the implementation of MoLoRA.

Motivation

Feature request
The paper "MoELoRA: Contrastive Learning Guided Mixture of Experts on
Parameter-Efficient Fine-Tuning for Large Language Models" introduced MoLoRA, a Mixutre-of-Experts approach using LoRA adapters. I am using it to conduct some research for my MSc thesis, and have implemented it in peft. I was wondering if this method is interesting and would be worth it to clean up my code and submit a PR.

Motivation
The motivation is to include more PEFT methods that the community can benefit from.

Your contribution
I can contribute a PR with the implementation of MoLoRA.

Your contribution

Feature request
The paper "MoELoRA: Contrastive Learning Guided Mixture of Experts on
Parameter-Efficient Fine-Tuning for Large Language Models" introduced MoLoRA, a Mixutre-of-Experts approach using LoRA adapters. I am using it to conduct some research for my MSc thesis, and have implemented it in peft. I was wondering if this method is interesting and would be worth it to clean up my code and submit a PR.

Motivation
The motivation is to include more PEFT methods that the community can benefit from.

Your contribution
I can contribute a PR with the implementation of MoLoRA.

[BenjaminBossan]

Thank you for your proposal to add MoELoRA to PEFT. From skimming the paper, the method itself looks interesting. However, I can foresee some challenges with implementing it:

• AFAICT, there is no reference implementation of the method.
• Based on your name, you're also not one of the paper authors.
• The method includes the contrastive loss component, which will make this more difficult to implement, especially so since the training code is out of scope for PEFT.

All this does not mean it could not be done, but it will most likely be challenging to implement correctly.