Florence-2 QLoRA support

[0xD4rky]

Search before asking

• I have searched the Multimodal Maestro issues and found no similar feature requests.

Description

I was going through the maestro repo and found out that both paligemma and florence models didn't support the implementation of 4-bit quantization (i.e. using QLoRA config).

Use case

Using QLoRA, we could easily fine-tune vision language models on even low end devices without losing on precision a lot. As the models grow, we would eventually need to implement QLoRA to make finetuning fast and possible on memory constraints.

Additional

I would like to learn your take on implementing quantization.

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[SkalskiP]

Hi @0xD4rky 👋🏻 Thank you for your interest in maestro. Sorry for the late reply, I've been heavily involved in delivering maestro-1.0.0 over the past few weeks. I've managed to add QLoRA support for PaliGemma 2 and Qwen2.5-VL. Unfortunately, for Florence-2 we only have LoRA for now. It would be great if someone in the community would like to add QLoRA.

@0xD4rky I saw that you checked the Yes I'd like to help by submitting a PR! checkbox. Would you like to give it a try?

[0xD4rky]

Yes @SkalskiP, it would be absolutely great if I am able to contribute towards maestro. Will go through the code base, develop the QLoRA approach for Florence-2 and then raise the PR. This works?

[SkalskiP]

@0xD4rky that would be awesome! 🔥 I'm excited. We are building something cool here.

[SkalskiP]

@0xD4rky I'll assign this issue to you!

BTW I have a question: I'm thinking about launching a Discord server dedicated to VLM fine-tuning with Maestro) and while talking about current issues I'm trying to understand if people would like such a server to be created.

[0xD4rky]

@SkalskiP, people would love to hop on a Roboflow's discord server in general. It would be great to discuss the fine-tunings and approaches using Maestro, people would be definitely interested!

[SkalskiP]

@0xD4rky sounds great; I'll try to set something up this week! By the way, I updated the contributors guidelines today. Take a look, considering you want to add support for Florence-2 QLoRA.

[0xD4rky]

@SkalskiP sure, I will go through it before starting to contribute. Will keep you in touch with all the changes that I come up with