completing the "bitsandbytes" option - based on https://docs.vllm.ai/en/stable/quantization/bnb.html

[mohamednaji7]

in https://docs.vllm.ai/en/stable/quantization/bnb.html there are 3 things to do

1. Installting bitsandbytes>=0.45.0
2. quantization="bitsandbytes"
3. load_format="bitsandbytes"

Current state for runpod-workers/worker-vllm:main

the load_format="bitsandbytes" is available

mohamednaji7/worker-vllm:main is ahead for

1. added bitsandbytes>=0.45.0 to the requirements.txt
2. [after checking args.load_format] forced args.quantization since it is the only quantization for load_format="bitsandbytes"
3. updated the QUANTIZATION row in RAEDME.md
4. updated QUANTIZATION option in worker-config.json
5. update to typing-extensions>=4.8.0 since there is a comapitablility issue for bitsandbytes>=0.45.0 and typing-extensions==4.7.1.

[pandyamarut]

How did you test this ? Can you please share @mohamednaji7

[mohamednaji7]

I deployed RunPod server-less endpoint using the repo and setting the environment variables and it works.
I tested and got the errors using the same deployment

device > GPU
I made a test now with the current worker-vllm

using None

logs - worker-vllm.txt

2025-02-01T09:19:14.726126817Z {"requestId": null, "message": "Uncaught exception | <class 'ValueError'>; BitsAndBytes load format and QLoRA adapter only support 'bitsandbytes' quantization, but got None; <traceback object at 0x700e46dea980>;", "level": "ERROR"}

---

deployment using this repo and setting the env vars
MODEL_NAME : unsloth/tinyllama-bnb-4bit
LOAD_FORMAT : bitsandbytes
QUANTIZATION : bitsandbytes

RESULT >>

Logs >>
logs - bitsandbytes.txt

THE BUILD was done before

build-logs-worker-vllm -fb.txt

[mohamednaji7]

@pandyamarut I wrote my tests above