This guide will walk you through deploying the official Nextcloud Talk High-Performance Backend (HPBE). This backend, which includes a signaling server (Spreed), a STUN/TURN server (Coturn), and a WebRTC MCU (Janus), significantly improves the performance and reliability of video calls, especially for multiple participants.

This setup is designed to integrate seamlessly with an existing Traefik v3 reverse proxy, making it a powerful addition to your self-hosted infrastructure.

Changelog

DateChange
2025-09-21Initial Version: Guide created, focusing on Docker Compose deployment and Traefik v3 integration.

1. Prerequisites

This guide is part of a series and builds upon a secure Docker environment. Before you begin, you must have a fully functional Traefik v3 and CrowdSec stack.

⚠️ HARD REQUIREMENT

The following steps will not work correctly without the Traefik stack running as described in the prerequisite guide.

You will also need:

  • Docker and Docker Compose installed on your server.
  • A dedicated subdomain for the signaling server (e.g., signaling.your-domain.com) pointed to your server’s IP address.
  • sudo or root access.
  • The git and openssl utilities installed (sudo apt install git openssl).
  • Firewall ports 80, 443, 3478/tcp, 3478/udp, 5349/tcp, and 5349/udp open. The ports 3478 and 5349 are required by the Coturn (TURN) server.

1.1. Firewall Configuration (UFW Example)

If you are using ufw (Uncomplicated Firewall), you can open the required ports with the following commands:

sudo ufw allow 3478/tcp
sudo ufw allow 3478/udp
sudo ufw allow 5349/tcp
sudo ufw allow 5349/udp
ℹ️ SECURITY NOTE & CROWDSEC INTEGRATION

The TURN ports (3478, 5349) are exposed directly and are not protected by the Traefik CrowdSec Bouncer. Securing them should be done at the host level.

For advanced protection against abuse (e.g., brute-force attacks on the TURN server), you can feed Coturn’s logs into CrowdSec. This is achieved by configuring Docker’s logging driver for the coturn service (as done in the docker-compose.yml file) and telling CrowdSec where to find these logs.

  1. Configure CrowdSec to read Docker logs: Create a new file acquis.yaml in your CrowdSec configuration directory (e.g., /opt/containers/crowdsec/config/acquis.d/coturn.yaml) with the following content. This tells CrowdSec to read the logs for the container named coturn.

    source: docker
container_name_patterns:
  - coturn
labels:
  type: coturn

  2. Restart CrowdSec: For the new acquisition configuration to take effect, restart the CrowdSec container.

    # Navigate to your CrowdSec directory and restart
cd /opt/containers/crowdsec/
sudo docker compose restart

CrowdSec will now automatically parse the logs for the coturn container. Thanks to the logging driver configured in docker-compose.yml, the logs are automatically rotated (3 files, 10MB each) and the log collection survives server restarts.

1.2. Resource Planning

The High-Performance Backend, especially the Janus MCU, can be resource-intensive during video calls. For a small group of users (e.g., 3-5 concurrent participants in a call), plan for at least 1-2 dedicated CPU cores and 2-4 GB of RAM for the HPBE stack. For larger deployments, monitor your resource usage and scale accordingly. To keep an eye on performance, regularly check resource usage with tools like htop, docker stats, or a more comprehensive monitoring stack like Prometheus and Grafana.

2. Directory Structure and Download

First, we will clone the official repository, which contains all the necessary Docker configurations.

# Navigate to your main containers directory
cd /opt/containers/

# Clone the repository
sudo git clone https://github.com/strukturag/nextcloud-spreed-signaling.git nextcloud-hpbe

# Enter the new directory
cd nextcloud-hpbe

3. Generate Secrets and Keys

The backend requires several strong secret keys for secure operation. We will generate them using openssl.

ℹ️ SAVE THESE KEYS

Store these generated keys in a secure location, as you will need them for the configuration files in the next steps.

# 1. For the Spreed signaling server sessions (32 bytes required)
echo "HASH_KEY=$(openssl rand -base64 32)"
echo "BLOCK_KEY=$(openssl rand -base64 32)"

# 2. For the TURN server API communication
echo "TURN_API_KEY=$(openssl rand -base64 16)"

# 3. For authenticating Nextcloud instances with the signaling server
# You must generate a unique secret for EACH Nextcloud instance you want to connect.
# Instance 1:
echo "NEXTCLOUD_1_SHARED_SECRET=$(openssl rand -hex 16)"
# Instance 2 (example):
# echo "NEXTCLOUD_2_SHARED_SECRET=$(openssl rand -hex 16)"

# 4. For securing the TURN server itself
echo "TURN_STATIC_SECRET=$(openssl rand -hex 32)"

4. Environment and Configuration Files

In the .env file, we will manage variables needed by docker-compose.yml, specifically the domain (SIGNALING_DOMAIN) and the TURN secret (TURN_STATIC_SECRET). All other secrets will be hardcoded directly into server.conf.

4.1. Environment File (.env)

Create a file named .env in your nextcloud-hpbe directory.

sudo tee .env > /dev/null << 'EOF'
# --- Domain Configuration ---
# Used by Docker Compose for the Traefik labels and Coturn realm.
SIGNALING_DOMAIN=signaling.your-domain.com

# --- TURN Server Secret ---
# Used by the Coturn service in docker-compose.yml.
# This value MUST be identical to the one pasted into server.conf.
TURN_STATIC_SECRET=PASTE-YOUR-TURN_STATIC_SECRET-HERE

# --- Docker Compose Project Name (Optional) ---
# Sets a consistent project name for easier management.
# COMPOSE_PROJECT_NAME=nextcloud-hpbe
EOF
⚠️ IMPORTANT
  1. Replace the placeholder value for SIGNALING_DOMAIN with your actual domain.
  2. Add .env to your .gitignore file if you are using Git in this directory. Create a .gitignore file with the following content to prevent committing secrets and backup files:
    # Ignore sensitive files and backups

.env
*.bak

Now, we will create the configuration files. docker-compose.yml will read the SIGNALING_DOMAIN from the .env file, while for server.conf, you will need to paste the generated secrets manually.

4.2. Docker Compose File (docker-compose.yml)

Create the docker-compose.yml file. This version has been updated for Traefik v3 and includes best practices like CrowdSec protection.

sudo tee docker-compose.yml > /dev/null << 'EOF'

services:
  spreed-backend:
    build:
      context: .
      dockerfile: docker/server/Dockerfile
    container_name: spreed-backend
    restart: unless-stopped
    volumes:
      - ./server.conf:/config/server.conf:ro
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/api/v1/welcome"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s
    depends_on:
      - nats
      - janus
      - coturn
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.hpbe.rule=Host(`${SIGNALING_DOMAIN}`) && PathPrefix(`/standalone-signaling`)"
      - "traefik.http.routers.hpbe.entrypoints=websecure"
      - "traefik.http.routers.hpbe.tls=true"
      - "traefik.http.routers.hpbe.tls.certresolver=tls_resolver"
      - "traefik.http.middlewares.hpbe-strip.stripPrefix.prefixes=/standalone-signaling"
      - "traefik.http.routers.hpbe.middlewares=hpbe-strip,security-headers@file,crowdsec-bouncer@docker"
      - "traefik.http.services.hpbe.loadbalancer.server.port=8080"

  nats:
    image: nats:2.10
    container_name: nats
    restart: unless-stopped
    command: ["--config", "/config/gnatsd.conf"]
    healthcheck:
      test: ["CMD-SHELL", "timeout 2 bash -lc '</dev/tcp/127.0.0.1/4222' || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
    volumes:
      - ./gnatsd.conf:/config/gnatsd.conf:ro
    networks:
      - proxy

  janus:
    # The official repository provides a Dockerfile to build Janus from source.
    # This is the most stable and recommended method.
    build: docker/janus
    # As a faster alternative, you can use a pre-built community image.
    # To do so, comment out the 'build' line above and uncomment the 'image' line below.
    # image: canyan/janus-gateway:latest
    container_name: janus
    restart: unless-stopped
    command: ["janus", "--full-trickle"]
    healthcheck:
      test: ["CMD-SHELL", "timeout 2 bash -lc '</dev/tcp/127.0.0.1/8188' || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 15s
    networks:
      - proxy

  coturn:
    image: coturn/coturn:4.7
    container_name: coturn
    restart: unless-stopped
    volumes:
      # Mount your Let's Encrypt certificates for TLS support.
      # Traefik stores certificates in a JSON file. We mount it so Coturn can use it.
      # Ensure the path '/opt/containers/traefik/letsencrypt/certificates/' matches your Traefik setup.
      # For a simpler initial setup without TLS, you can remove this volume mount and the `--cert` / `--pkey` flags below.
      # If you do so, also remove the 'turns:' entries from server.conf and the 5349 ports.
      - "/opt/containers/traefik/letsencrypt/certificates/${SIGNALING_DOMAIN}.json:/certs/_cert.json:ro"
    healthcheck:
      test: ["CMD-SHELL", "timeout 2 bash -lc '</dev/tcp/127.0.0.1/3478' || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
    networks:
      - proxy
    ports:
      - "3478:3478/tcp"
      - "3478:3478/udp"
      - "5349:5349/tcp"   # Recommended for TLS
      - "5349:5349/udp"   # Recommended for DTLS
    command:
      - "--realm=${SIGNALING_DOMAIN}"
      - "--static-auth-secret=${TURN_STATIC_SECRET}"
      - "--tls-listening-port=5349"
      - "--cert=/certs/cert.pem"
      - "--pkey=/certs/key.pem"
      - "--no-stdout-log"
      - "--log-file=stdout"
      - "--stale-nonce=600"
      - "--use-auth-secret"
      - "--fingerprint"
      - "--no-software-attribute"
      - "--no-multicast-peers"

networks:
  proxy:
    external: true
EOF

4.3. Signaling Server Config (server.conf)

This file configures the core logic of the HPBE. Create server.conf and paste the template below.

sudo tee server.conf > /dev/null << 'EOF'
[http]
listen = 0.0.0.0:8080

[app]
debug = false

[sessions]
hashkey = PASTE-YOUR-HASH_KEY-HERE
blockkey = PASTE-YOUR-BLOCK_KEY-HERE

[backend]
# List all your Nextcloud backends here, separated by commas.
backends = nextcloud-1 #, nextcloud-2
allowall = false
timeout = 10
connectionsperhost = 8

# Optional bitrate limits to conserve bandwidth/resources
# 524288  = 512 kbit/s; 1048576 = 1 Mbit/s
maxstreambitrate = 524288
maxscreenbitrate = 1048576

[nextcloud-1]
url = PASTE-YOUR-NEXTCLOUD_1_URL-HERE
secret = PASTE-YOUR-NEXTCLOUD_1_SHARED_SECRET-HERE

# To add a second Nextcloud instance, uncomment the 'nextcloud-2' in the 'backends' list
# above and configure its section below. Make sure you have also defined the
# corresponding variables (e.g., NEXTCLOUD_2_URL, NEXTCLOUD_2_SHARED_SECRET) in your .env file.
#
# [nextcloud-2]
# url = https://another-cloud.com
# secret = PASTE-YOUR-NEXTCLOUD_2_SHARED_SECRET-HERE

[nats]
url = nats://nats:4222

[mcu]
type = janus
url = ws://janus:8188

# Optional bitrate limits (should mirror [backend] if used)
maxstreambitrate = 524288
maxscreenbitrate = 1048576

# [clients]
# Optional: uncomment this section if you run internal clients (e.g., recorder)
# internalsecret = PASTE-YOUR-INTERNAL-SECRET-HERE

[turn]
apikey = PASTE-YOUR-TURN_API_KEY-HERE
# This value MUST be identical to TURN_STATIC_SECRET in your .env file.
secret = PASTE-YOUR-TURN_STATIC_SECRET-HERE
servers = turn:coturn:3478?transport=udp,turn:coturn:3478?transport=tcp,turns:coturn:5349?transport=tcp,turns:coturn:5349?transport=udp
EOF

The server.conf file is now complete. Unlike docker-compose.yml, the signaling server does not substitute ${...} variables from the environment. It is critical that the secret in the [turn] section of this file exactly matches the TURN_STATIC_SECRET value in your .env file.

ℹ️ ADVANCED METHOD: USING ENVSUBST

If you prefer to manage all your variables in the .env file, you can use a template-based approach. To do this, create a server.conf.template file with ${...} variables, and then generate the final server.conf using the envsubst command:

envsubst < server.conf.template > server.conf

This method requires gettext to be installed (sudo apt install gettext).

Ensure the file has the correct permissions: sudo chmod 644 server.conf.

Note:

  • The maxstreambitrate and maxscreenbitrate values are set directly in server.conf. Adjust them to match your network and hardware capacity.
  • The [clients] section is only needed if you use internal services like a recording backend. If not required, you can remove or leave it with an unset INTERNAL_SECRET.

5. Launch the Stack

With the configuration complete, you can now build and start the services.

# From within the /opt/containers/nextcloud-hpbe directory
sudo docker compose up --build -d

The --build flag is important as it builds the spreed-backend and janus images from their Dockerfiles. The Janus build, in particular, can take several minutes.

You can monitor the logs to ensure everything starts correctly:

sudo docker compose logs -f

6. Configure Nextcloud

The final step is to tell your Nextcloud instance to use the new High-Performance Backend.

  1. Log in to Nextcloud as an administrator.
  2. Navigate to Administration Settings -> Talk.
  3. Scroll down to the “Signaling server” section.
  4. Check “Enable custom signaling server”.
  5. In the “Signaling server URL” field, enter the full path to your HPBE: https://signaling.your-domain.com/standalone-signaling.
  6. In the “Shared secret” field, paste the corresponding shared secret for this specific Nextcloud instance (e.g., the value of NEXTCLOUD_1_SHARED_SECRET for your first instance).
  7. Click “Save changes”.

Nextcloud will verify the connection. If everything is correct, you’re all set!

Clarifying Secret Mappings

To ensure all components communicate securely, it’s crucial to map the secrets correctly. Here is a quick reference:

server.conf Section & KeyMaps to…Purpose
[backend] -> secretNextcloud Admin -> Talk -> Shared secretAuthenticates Nextcloud with the signaling server.
[turn] -> apikeyjanus.jcfg -> turn_rest_api_key (handled internally by the setup)Allows signaling server to get TURN credentials.
[turn] -> secretcoturn service -> --static-auth-secret command argumentAuthenticates TURN users (generated by signaling).

7. Troubleshooting & Monitoring

If you encounter issues, here are a few steps to diagnose the problem:

  1. Verify the Signaling Server is Reachable

    You should get a 200 OK response from the /welcome endpoint. This confirms that Traefik is routing requests correctly.

    curl -i https://signaling.your-domain.com/standalone-signaling/api/v1/welcome

  2. Check the Container Logs

    The logs are the best source for identifying errors. Pay close attention to messages about secrets, WebSocket connections, or backend timeouts.

    # From within the /opt/containers/nextcloud-hpbe directory
sudo docker compose logs -f

  3. Common Errors & Fixes

    • ERROR: the sessions block key must be... or hash key should be...: This error occurs when the keys in server.conf have the wrong length or are missing. Ensure you have correctly generated the keys with openssl rand -base64 32 and pasted the actual values into server.conf, replacing the PASTE-YOUR-...-HERE placeholders.
    • Warning The "TURN_STATIC_SECRET" variable is not set: This means the TURN_STATIC_SECRET is missing from your .env file. Ensure it is defined in .env and that its value is identical to the secret in the [turn] section of server.conf.
    • “failed to establish signaling connection”: This is a classic error.
      • Check that the URL in Nextcloud is exactly https://.../standalone-signaling.
      • Ensure your Traefik labels are correct (especially the PathPrefix and stripPrefix rules).
      • Verify that the Shared secret in Nextcloud matches the one in server.conf.
    • Calls work for 2 people but fail with 3+: This often points to a problem with Janus (the MCU) or Coturn (the TURN server). Check their logs specifically.
    • No video/audio from external networks: This is a typical TURN server issue. Ensure ports 3478 and 5349 (TCP/UDP) are open on your firewall and correctly forwarded to the Coturn container.

8. Maintenance and Updates

To update your High-Performance Backend to the latest version:

# Navigate to the HPBE directory
cd /opt/containers/nextcloud-hpbe

# Stop the current services
sudo docker compose down

# (Optional but recommended) Back up your configuration files
sudo cp docker-compose.yml docker-compose.yml.bak
sudo cp server.conf server.conf.bak

# Pull the latest changes from the git repository
sudo git pull

# Rebuild the images and start the services
sudo docker compose up --build -d --remove-orphans

Conclusion

Congratulations! You have successfully deployed a Nextcloud Talk High-Performance Backend. Your users will now experience more stable and performant video calls, especially in group settings. This powerful, containerized setup integrates perfectly with a modern Traefik proxy, providing a scalable and secure solution for your communication needs.

📚OFFICIAL HPBE REPOSITORY