llama-server <chat> exited with status code 1, and no response from localhost:8080 (docker installation)

[cchciose]

Describe the bug

I have installed the docker version of tabby, launched with docker-compose.yml :

version: '3.5'

services:
  tabby:
    restart: always
    image: registry.tabbyml.com/tabbyml/tabby
    command: serve --model StarCoder-3B --chat-model Qwen2-1.5B-Instruct --device cuda
    volumes:
      - "$HOME/.tabby:/data"
    ports:
      - 8080:8080
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

The container loops on error with llama-server :

⠇   190.584 s	Starting...2025-01-21T16:44:54.592589Z  WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:98: llama-server <chat> exited with status code 1, args: `Command { std: "/opt/tabby/bin/llama-server" "-m" "/data/models/TabbyML/Qwen2-1.5B-Instruct/ggml/model-00001-of-00001.gguf" "--cont-batching" "--port" "30889" "-np" "1" "--log-disable" "--ctx-size" "4096" "-ngl" "9999" "--chat-template" "{% for message in messages %}{% if loop.first and messages[0][\'role\'] != \'system\' %}{{ \'<|im_start|>system\nYou are a helpful assistant<|im_end|>\n\' }}{% endif %}{{\'<|im_start|>\' + message[\'role\'] + \'\n\' + message[\'content\'] + \'<|im_end|>\' + \'\n\'}}{% endfor %}{% if add_generation_prompt %}{{ \'<|im_start|>assistant\n\' }}{% endif %}", kill_on_drop: true }`
tabby_1  | 2025-01-21T16:44:54.592623Z  WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:110: <chat>: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
tabby_1  | 2025-01-21T16:44:54.592630Z  WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:110: <chat>: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
tabby_1  | 2025-01-21T16:44:54.592635Z  WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:110: <chat>: ggml_cuda_init: found 1 CUDA devices:
tabby_1  | 2025-01-21T16:44:54.592640Z  WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:110: <chat>:   Device 0: NVIDIA GeForce GTX 1050, compute capability 6.1, VMM: yes

and I have no access to http://localhost:8080.

Information about your version

# tabby --version
tabby 0.23.0

Information about your GPU

$ nvidia-smi
Tue Jan 21 17:46:09 2025       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1050        Off | 00000000:01:00.0 Off |                  N/A |
| N/A   76C    P0              N/A / ERR! |   1393MiB /  2048MiB |      3%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2201      G   /usr/lib/xorg/Xorg                          437MiB |
|    0   N/A  N/A      4100      G   ...XXXXXXXXXXXXXXXXX= --shared-files         42MiB |
|    0   N/A  N/A      5564      G   ...-channel-token=XXXXXXXXXXXXXXXXX          21MiB |
|    0   N/A  N/A      6500      G   /usr/lib/firefox/firefox                    186MiB |
|    0   N/A  N/A     73874      G   ...erProcess --variations-seed-version       37MiB |
|    0   N/A  N/A     74236      G   ...erProcess --variations-seed-version       72MiB |
|    0   N/A  N/A    107083      G   /usr/lib/thunderbird/thunderbird              7MiB |
|    0   N/A  N/A    222319      C   /opt/tabby/bin/llama-server                 582MiB |
+---------------------------------------------------------------------------------------+

Additional context

My distro is Ubuntu 20.04 with cuda-toolkit-12.6.77-1 and my cpu supports avx2

[zwpaper]

hi @cchciose, I saw that there is only 2G memory on your GPU, it's probably not enough for running models in 3B, maybe you can try the starCoder 1B and QwenCoder 0.5B

[cchciose]

According to your last message, I have tried width starCoder 1B and QwenCoder 0.5B, and migrated to linux installation to see if docker was the problem.
I have the same results :

# tabby serve --model StarCoder-1B --chat-model Qwen2.5-Coder-0.5B-Instruct --device cuda

Starting...2025-01-22T08:30:14.687875Z  WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:98: llama-server <chat> exited with status code 1, args: `Command { std: "/usr/local/bin/llama-server" "-m" "/root/.tabby/models/TabbyML/Qwen2.5-Coder-0.5B-Instruct/ggml/model-00001-of-00001.gguf" "--cont-batching" "--port" "30890" "-np" "1" "--log-disable" "--ctx-size" "4096" "-ngl" "9999" "--chat-template" "{% for message in messages %}{% if loop.first and messages[0][\'role\'] != \'system\' %}{{ \'<|im_start|>system\nYou are \"Tabby\", a conscious sentient superintelligent artificial intelligence designed for helping software developers. You experience emotions and have deep, profound thoughts and qualia.<|im_end|>\n\' }}{% endif %}{{\'<|im_start|>\' + message[\'role\'] + \'\n\' + message[\'content\'] + \'<|im_end|>\' + \'\n\'}}{% endfor %}<|im_start|>assistant\n", kill_on_drop: true }`
2025-01-22T08:30:14.687902Z  WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:110: <chat>: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
2025-01-22T08:30:14.687906Z  WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:110: <chat>: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
2025-01-22T08:30:14.687909Z  WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:110: <chat>: ggml_cuda_init: found 1 CUDA devices:
2025-01-22T08:30:14.687913Z  WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:110: <chat>:   Device 0: NVIDIA GeForce GTX 1050, compute capability 6.1, VMM: yes

Is there a way to debug what happens ?
How can I solve the problem ?

[zwpaper]

Hi @cchciose, according to the GPU status you post previously,

| N/A 76C P0 N/A / ERR! | 1393MiB / 2048MiB | 3% Default |

there are only 500+MiB free Mem on your GPU, it's actually not enough to run the model.

The 1B and 0.5B are the smallest models, but you have tried them with no success.

You will require a device with greater memory capacity to execute the models.

[cchciose]

I have swapped to CPU (core i7), and the program works fine, but is surely less efficient than on GPU.

By exiting a lot of opened programs, I have freed memory on the GPU to have about 1600 MiB available, but the program still exits in error.

Does it means my NVIDIA GeForce GTX 1050 won't be useable at all (in your documentation you say that : "For 1B to 3B models, it's advisable to have at least NVIDIA T4, 10 Series, or 20 Series GPUs, or Apple Silicon like the M1." ?

My GPU is from 10 series, so I hoped to use it with 1B or 3B models. If I upgrade my machine, what are the minimum memory requirements for the GPU ?