Host an AI chatbot with Ollama + Open WebUI in docker

Introduction

In this blog, we are going to install Ollama, which will allow us to run large language models (LLMs) locally - such as Llama3, Mistral etc.

We are are also going to install Open WebUI, which will provide us with a ChatGPT like WebUI so that we can chat with our LLMs via Ollama in a web browser.

We will be running both of these applications in docker containers utilising docker compose.

Important: Ollama can utilise the GPU in your server which in turn allow us to get faster response times and run bigger models, but in this blog I will only be utilising the CPU as my server don’t have a dedicated GPU.

Presequites

Please do note that following presequites are tailored to my personal usecase.

• Sever with Debian 12 or Ubuntu 22.04 insalled as OS
• Docker & docker compose
• Reverese proxy solution like Caddy to access Open WebUI via a domain name (optional)

Installing Ollama & Open WebUI

Ollama

To get started, first we need to create a directory where we will be storing all our data and langauge models.

$ mkdir -p ~/docker/ollama
$ cd ~/docker/ollama

Let’s create a docker-compose.yml file inside the directory.

$ nano docker-compose.yml

Inside the docker-compose.yml, add the following

version: "3.5"
services:
    ollama:
        image: ollama/ollama:latest
#        ports:
#            - 11434:11434
        volumes:
            - ./ollama/models:/root/.ollama
        container_name: ollama
        pull_policy: always
        tty: true
        restart: always

we are binding the directory /root/.ollama inside container to outside at ./ollama/models to store our models outisde of the container and make them persistent.

Open WebUI

Let’s add the code we need to install Open WebUi to the same docker-compose.yml file

    open-webui:
        volumes:
            - ./data:/app/backend/data
        environment:
            - 'OLLAMA_BASE_URL=http://ollama:11434'
        container_name: open-webui
        restart: always
        image: ghcr.io/open-webui/open-webui:main
        ports:
            - 127.0.0.1:8080:8080

This is what my full docker-compose.yaml looks like after adding both of thses:

version: "3.5"
services:
    ollama:
        image: ollama/ollama:latest
#        ports:
#            - 11434:11434
        volumes:
            - ./ollama/models:/root/.ollama
        container_name: ollama
        pull_policy: always
        tty: true
        restart: always
    open-webui:
        volumes:
            - ./data:/app/backend/data
        environment:
            - 'OLLAMA_BASE_URL=http://ollama:11434'
        container_name: open-webui
        restart: always
        image: ghcr.io/open-webui/open-webui:main
        ports:
            - 127.0.0.1:8080:8080
networks:
  default:
    external:
      name: caddy

Note: as you can see, I am running both of these containers in a custom docker network called caddy because I am going to use caddy (my reverse proxy solution) to access Open WebUI via chat.domain.com, but this is optional.

Finally, let’s run docker compose up -d to get it running.

Now you can access the Open WebUI by going to http://127.0.0.1:8080 OR via your domain name if you setup caddy or your preferred reverse proxy solution in advance to access Open WebUI.

Meke sure to create a account to get started. By default you won’t be able to chat since we did not install any language model in Ollama yet, so let’s do that.

Adding langauge models to Ollama

We can add models to Ollama directly from Open WebUI. Please follow the screenshots below.

Alternative Method:

We can also install models via command line by going into our ollama container (though this is not needed unless you are unable to install models via Open WebUI for some reason).

docker exec -it ollama bash

Now we are inside the ollama container. Let’s install Meta AI’s Llama3 as an example.

ollama run llama3

This will install download Llama3 8b model. After it is finished downloading, you will be able to chat with Llama3 via the command line.

You can exit from Llama3 and our Ollama container by pressing Ctrl + D and typing exit

Final thoughts

That’s it. Now you are have a perosnal AI chat assistant running locally in your server! 😄

You can install as many models as you like from the Ollama library. You can also chat with multiple models like below:

Please go to the Ollama website to see the list of langauge models. I personally use the aforementioned Llama3 and Deepseek coder for my day to day tasks.

Hope you find this blog post useful. Thanks for reading! 😊