Install LocalAI As A Stack On Portainer

Installing LocalAI as a Stack on Portainer (with NVIDIA GPU Support & Preloaded Models)

Running local AI models has become a practical alternative to cloud APIs, especially for privacy, cost control, and compliance. With tools like LocalAI, you can host your own AI inference server and expose it via an OpenAI-compatible API fully under your control.

In this guide, we’ll deploy LocalAI as a stack in Portainer using Docker and Docker Compose, including NVIDIA GPU support and the NVIDIA Container Toolkit.

We’ll also use a GPU-optimised prebuilt image that already includes some models out of the box

localai/localai:master-aio-gpu-nvidia-cuda-12

This significantly reduces setup time and gets you testing AI models faster.

What is LocalAI?

LocalAI is an open-source AI inference server designed as a drop-in replacement for OpenAI APIs. It allows you to run:

• Large language models (LLMs)
• Embeddings
• Speech-to-text models
• Image generation models

All locally, without sending data to external providers.

Why use Portainer?

Using Portainer provides:

• Web-based container management
• Easy Docker Compose stack deployment
• Centralised logging and monitoring
• Simplified lifecycle management

This makes it ideal for homelabs and production edge deployments.

Step 1: System requirements

Before starting, ensure your host meets:

• 8GB RAM minimum (16GB+ recommended)
• 4+ CPU cores
• SSD storage strongly recommended
• Linux OS (Ubuntu 20.04/22.04 recommended)

For GPU acceleration:

• NVIDIA GPU (CUDA-capable)
• Recent NVIDIA driver
• NVIDIA Container Toolkit installed

Step 1.5: Install NVIDIA drivers & toolkit (GPU setup)

This step enables GPU acceleration inside Docker containers.

1. Check GPU detection

lspci | grep -i nvidia

If output appears, your system detects the GPU.

2. Install NVIDIA drivers (Ubuntu)

Update system:

sudo apt update && sudo apt upgrade -y

sudo ubuntu-drivers devices

sudo ubuntu-drivers autoinstall

sudo reboot

nvidia-smi

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)

curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update

sudo apt install -y nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker

sudo systemctl restart docker

docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi

Step 2: Create the LocalAI stack in Portainer

1. Log into Portainer
2. Go to Stacks
3. Click Add stack
4. Name it: localai
5. Select Web editor

Step 3: Docker Compose stack configuration (GPU + preloaded models)

We will use the GPU-enabled AIO image:

👉 localai/localai:latest-aio-gpu-nvidia-cuda-12

This image includes:

• CUDA 12 GPU support
• Preloaded models (reduces setup time)
• Ready-to-use API endpoints
• Optimised runtime configuration

Stack YAML:

version: "3.9"

services:
  localai:
    image: localai/localai:master-aio-gpu-nvidia-cuda-12
    container_name: localai
    ports:
      - "8080:8080"
    environment:
      - THREADS=4
      - CONTEXT_SIZE=4096
      - NVIDIA_VISIBLE_DEVICES=all
    volumes:
      - localai-data:/models
      - localai-config:/config
    restart: unless-stopped

What this stack does

• Runs LocalAI with GPU acceleration enabled
• Uses CUDA 12 optimised runtime
• Exposes API on port 8080
• Stores models persistently
• Includes preloaded models (faster start-up)

Step 4: (Optional) Add explicit GPU reservation

If needed for stricter environments:

deploy:

  resources:
    reservations:
      devices:
       - driver: nvidia
         count: all
         capabilities: [gpu]

Step 5: Deploy the stack

Click Deploy Stack in Portainer.

Once running, LocalAI will be accessible at:

http://<server-ip>:8080

Step 6: Verify preloaded models

Because we are using the AIO GPU image, models are already available.

Check logs:

docker logs localai

Step 7: Test the API

curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
 "model": "gpt-4all-j",
 "messages": [
 {"role": "user", "content": "Hello, test LocalAI"}
 ]
 }'

If successful, you’ll receive an OpenAI-style JSON response.

Step 8: Performance tuning

CPU

• THREADS = number of physical cores
• Avoid overcommitting CPU

GPU

• Ensure nvidia-smi shows active usage
• Monitor with watch nvidia-smi

Context size

• 2048 = fast responses
• 4096–8192 = better reasoning (more RAM/GPU usage)

Step 9: Security considerations

Even internal AI services should be secured:

• Do not expose port 8080 publicly
• Restrict access via firewall rules
• Use reverse proxy with HTTPS if external access is needed
• Segment AI workloads on isolated VLANs

Example:

Step 10: Optional reverse proxy

For production deployments:

• Use Nginx Proxy Manager or Traefik
• Add SSL (Let’s Encrypt)
• Use a domain like ai.yourdomain.co.uk

Common issues

GPU not being used

• Check nvidia-smi
• Confirm NVIDIA Container Toolkit is installed
• Restart Docker

Container crash loop

• Check logs in Portainer
• Verify memory availability

Slow responses

• Ensure GPU is active
• Try smaller models
• Reduce context size

use cases

This setup is ideal for:

• GDPR-compliant AI systems
• Internal IT helpdesk automation
• Secure document processing
• SME AI assistants
• Offline/air-gapped environments

Final thoughts

Deploying LocalAI using the GPU-enabled AIO image localai/localai:master-aio-gpu-nvidia-cuda-12 inside Portainer is one of those setups that feels almost too easy for the amount of power you end up with. What you’re essentially doing is spinning up a fully self-contained AI inference stack that would normally take a fair bit of manual configuration, dependency wrangling, and “why is CUDA not seeing my GPU?” troubleshooting… and instead getting it running in a clean, repeatable, containerised way.

Once it’s up, you’re not just running “another Docker container” — you’re running your own self-hosted AI backend that behaves very similarly to OpenAI’s API, but without the external dependency, rate limits, or data leaving your infrastructure.

The real win here is how quickly it goes from zero to usable. Portainer gives you the visual control plane, Docker handles the orchestration, and the AIO GPU image brings everything else along for the ride — CUDA runtime, inference optimisations, and pre-packaged model support.

What you actually gain from this setup

Instead of thinking of it as just “installing LocalAI”, it’s better to think of it as deploying your own mini AI platform. Once it’s running, you immediately get:

⚡ GPU acceleration out of the box

No manual CUDA compilation nightmares, no driver-level tinkering inside containers. If your NVIDIA stack is set up correctly on the host, LocalAI automatically leverages it. That means faster inference, smoother responses, and the ability to actually run larger models without everything crawling.

📦 Preloaded models for faster deployment

The AIO image comes with a selection of ready-to-use models, so you’re not stuck waiting around downloading gigabytes of weights before you can test anything. You can literally deploy, hit the API, and start getting responses almost immediately. It’s very “plug in and play” for something this powerful.

🔌 OpenAI-compatible API

One of the most underrated features. Anything built for OpenAI’s API can usually be pointed at LocalAI with a simple endpoint change. That means tools, scripts, bots, automation pipelines, and apps you already use can often switch over with minimal or zero code changes. It’s basically a drop-in replacement layer for a huge ecosystem.

🧠 Full self-hosted control

Everything stays inside your environment. No external API calls, no data leaving your network, and no dependency on third-party uptime. You decide what models run, how they’re exposed, and how resources are allocated. For homelabs, SMEs, or anyone concerned about data privacy, this is a big deal.

Why this setup is genuinely powerful

What makes this combination of Portainer + Docker + LocalAI GPU AIO image so effective is the balance between simplicity and capability.

On one side, you’ve got a clean UI-driven deployment experience through Portainer. On the other, you’ve got serious backend performance thanks to CUDA-accelerated inference. And in the middle sits LocalAI quietly doing the heavy lifting, exposing everything through a familiar API layer.

It’s the kind of setup where you can start small — maybe a single model for experimentation — and then gradually scale it into a proper internal AI service for:

• Chat assistants
• Code generation tools
• Document summarisation
• Internal automation workflows
• API-backed applications
• Even agent-style systems running in your homelab or lab environment

The takeaway

At its core, this deployment gives you something very valuable: control without complexity.

You’re not locked into cloud pricing models. You’re not waiting on external APIs. You’re not fighting complicated install scripts every time you want to change something.

Instead, you get a fast, GPU-accelerated AI stack that you can spin up, tear down, modify, and scale entirely on your own terms and do it all from inside Portainer with a clean, repeatable setup.

Once it’s running