How to convert LlavaLlamaForCausalLM based models to GGUF

[dand-milestone]

Hey there,

my goal is to run LlavaLlamaForCausalLM based models on a jetson device through Ollama. As far as I could see there’s no “out-of-the-box” support to convert the model weights into the .gguf format without losing its vision component. I did some digging and found the /app/examples/convert_legacy_llama.py from llama.cpp but it still doesn’t do the trick for the vision tower.

Maybe I’m being ignorant about some obvious solution but so far I couldn’t get around to it. Any ideas how to convert the model properly?

Any pointers, guides, tutorials etc. to achieve the above goal will be much appreciated!

[LordLokator]

+1 on this it's really annoying

[illtellyoulater]

also looking for this

[LordLokator]

Some people say that either convert.py (deprecated / removed AFAIK) or the legacy llama converter (in './examples' if I remember correctly) should do the trick but for me neither works.

I want to test NVILA or some other Llama backboned VLM in Ollama and it's really frustrating to not be able to convert these models. Ollama has a Llava model available on it's website but there is no tool or tutorial on how to actually import a VLM. I think .safetensors should work as well but I haven't had success importing that yet.

[illtellyoulater]

Just posted the solution here: #2948 (comment)

The Ollama/quantize tool finally did it for me! https://hub.docker.com/r/ollama/quantize

[ngxson]

convert_hf_to_gguf will work out of the box with llava after #11292

[LordLokator]

Thanks for the heads up, reading 11292 (and 8010) was really informative.