[Question]: Receiving a connection error while connecting to Ollama on my host machine

[joelclimbsthings]

What is your question?

I've been trying to get my local instance of Ollama to work with LibreChat for a while to no avail. I'll post my librechat.yml file in the "More Details" section below. I'm runng Librechat within Docker.

Ollama is running fine on my local machine--if I run the following, I get a valid response as expected:

curl http://localhost:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "phi3",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "Hello!"
            }
        ]
    }'

However I'm getting the following error within the ./logs directory.

{"cause":{"code":"ECONNREFUSED","errno":"ECONNREFUSED","message":"request to http://host.docker.internal:11434/v1/chat/completions failed, reason: connect ECONNREFUSED 172.17.0.1:11434","type":"system"},"level":"error","message":"[handleAbortError] AI response error; aborting request: Connection error.","stack":"Error: Connection error.\n    at OpenAI.makeRequest (/app/api/node_modules/openai/core.js:292:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async ChatCompletionStream._createChatCompletion (/app/api/node_modules/openai/lib/ChatCompletionStream.js:53:24)\n    at async ChatCompletionStream._runChatCompletion (/app/api/node_modules/openai/lib/AbstractChatCompletionRunner.js:314:16)"}

• [ ]

More Details

Here is my librechat.yml file:

For more information, see the Configuration Guide:

https://www.librechat.ai/docs/configuration/librechat_yaml

Configuration version (required)

version: 1.0.9

Cache settings: Set to true to enable caching

cache: true

endpoints:
custom:
- name: "Ollama"
apiKey: "ollama"
# use 'host.docker.internal' instead of localhost if running LibreChat in a docker container
baseURL: "http://host.docker.internal:11434/v1/chat/completions"
models:
default: [
"codestral",
"phi3",
"mixtral:8x7b-instruct-v0.1-q4_1",
]
# fetching list of models is supported but the name field must start
# with ollama (case-insensitive), as it does in this example.
fetch: true
titleConvo: true
titleModel: "current_model"
summarize: false
summaryModel: "current_model"
forcePrompt: false
modelDisplayLabel: "Ollama"

What is the main subject of your question?

Endpoints

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[fuegovic]

try to set the baseURL to : http://host.docker.internal:11434/v1/

[joelclimbsthings]

Thanks for the quick response, although (I should have mentioned) I had tried that as well. Same error:

{"cause":{"code":"ECONNREFUSED","errno":"ECONNREFUSED","message":"request to http://host.docker.internal:11434/v1/chat/completions failed, reason: connect ECONNREFUSED 172.17.0.1:11434","type":"system"},"level":"error","message":"[handleAbortError] AI response error; aborting request: Connection error.","stack":"Error: Connection error.\n    at OpenAI.makeRequest (/app/api/node_modules/openai/core.js:292:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async ChatCompletionStream._createChatCompletion (/app/api/node_modules/openai/lib/ChatCompletionStream.js:53:24)\n    at async ChatCompletionStream._runChatCompletion (/app/api/node_modules/openai/lib/AbstractChatCompletionRunner.js:314:16)"}

[danny-avila]

What's your operating system? is Ollama running without docker? you may need to use the IP of your machine

[ic-mw]

We have the same problem, running Ubuntu 23.10 on a server with the default docker version from apt/Ubuntu repositories.

Getting the following errors.
warn: [OpenAIClient.chatCompletion][stream] API error
error: [handleAbortError] AI response error; aborting request: Connection error.

Everything else is running fine, we only have a problem connecting to the Ollama server running on the host. And the Ollama instance is definitely working, a curl on the relevant endpoints goes through without a hitch.

[danny-avila]

How are you running Ollama? Networking for this is a bit tricky, the easiest solution is to share the same container network

[ic-mw]

Ollama is running as service on the host machine, and LibreChat is running via Docker.
Since your default setup looks like this:

So this should work, but it doesn't:
baseURL: "http://host.docker.internal:11434/v1"

I'm running out of ideas here...

[danny-avila]

So this should work, but it doesn't: baseURL: "http://host.docker.internal:11434/v1"

I'm running out of ideas here...

you're right it should, but this is mainly expected behavior for Docker Desktop:

This might be relevant: https://stackoverflow.com/questions/48546124/what-is-the-linux-equivalent-of-host-docker-internal

Try curling from the container shell.

AI convo on this topic:
New_Chat(3).md

[ic-mw]

Thanks for your reply and assistance so far. The "AI powered" suggestions are mostly valid (from a logical view), but the only configuration which actually works (at least when I try to curl it up from inside the librechat container) is the one where I allow Host networking for the container, which is not a robust option because of security concerns.
Unfortunately I don't have the time to do many more hours of trial & error, but if I happen to find a solution I'll share.

[joelclimbsthings]

I've still been unable to resolve this. I'm on linux, and oddly enough I've found that I can access a basic HTTP server running on my host with the host.docker.internal hostname, but still not ollama which is also work fine on the host. Perhaps the problem is with how the ollama service is made available on the host?

If I add the network_mode: host option to the docker-compose.yaml file, then I can access Ollama from the container, but the Librechat UI suddenly doesn't load. Would I need to tweak something else to make that work?

[joelclimbsthings]

It turns out the issue was that Ollama by default only responds to the 127.0.0.1 interface. Since my host service is run with systemd, I was able to add this to the ollama.service file to resolve the issue:

Environment="OLLAMA_HOST=0.0.0.0:11434"

[danny-avila]

Thank you for looking into this!

[ic-mw]

Yes, excellent find, thanks a lot.

[scoooducks]

Where is the ollama.service file?

[joelclimbsthings]

I'm on an Ubuntu system, and it's located at /etc/systemd/system

[ghost]

Great catch. This could be included in docs, in some troubleshooting section.

[PR4SAN]

2024-08-28 14:22:59,977 - root - ERROR - Error raised by inference endpoint: HTTPConnectionPool(host='host.docker.internal', port=11434): Max retries exceeded with url: /api/embeddings (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x78e4e250add0>: Failed to establish a new connection: [Errno 111] Connection refused'))

i am also facing this issue i am trying to use below for rag

RAG_API_URL=http://host.docker.internal:8000 EMBEDDINGS_PROVIDER=ollama OLLAMA_BASE_URL=http://host.docker.internal:11434 EMBEDDINGS_MODEL=nomic-embed-text

it works fine in my mac but my server is linux there i am getting the above error. can anyone help.

i did add Environment="OLLAMA_HOST=0.0.0.0:11434" to ollama.service file to resolve but its not working.

[vpnwiz]

Including the solution for Mac users, found this issue by searching ECONNREFUSED...

Open Terminal and type launchctl setenv OLLAMA_HOST "0.0.0.0"
You may need to preface it with sudo if there's an error.

Source: https://github.com/ollama/ollama/blob/main/docs/faq.md#setting-environment-variables-on-mac

[stawils]

Hey there,

For all people like my who strugles on this even after adding the environment
Environment="OLLAMA_HOST=0.0.0.0:11434",

It turns out that you need to add a forward slash at the end for it to work!!

http://host.docker.internal:11434/

[stawils]

starngly enough, for the embedding to work with ollama the url should be
http://host.docker.internal:11434