Deploying the Nextcloud Talk High-Performance Backend with Docker

This guide will walk you through deploying the official Nextcloud Talk High-Performance Backend (HPBE). This backend, which includes a signaling server (Spreed), a STUN/TURN server (Coturn), and a WebRTC MCU (Janus), significantly improves the performance and reliability of video calls, especially for multiple participants.

This setup is designed to integrate seamlessly with an existing Traefik v3 reverse proxy, making it a powerful addition to your self-hosted infrastructure.

Changelog

1. Prerequisites

This guide is part of a series and builds upon a secure Docker environment. Before you begin, you must have a fully functional Traefik v3 and CrowdSec stack.

You will also need:

• Docker and Docker Compose installed on your server.
• A dedicated subdomain for the signaling server (e.g., signaling.your-domain.com) pointed to your server’s IP address.
• sudo or root access.
• The git utility installed (sudo apt install git).
• Firewall ports 80, 443, 3478/tcp, 3478/udp open. The ports 3478 are required by the Coturn (TURN) server.

1.1. Firewall Configuration (UFW Example)

If you are using ufw (Uncomplicated Firewall), you can open the required ports with the following commands:

sudo ufw allow 3478/tcp
sudo ufw allow 3478/udp
# Example with media range 20000–20100 (choose any suitable range)
sudo ufw allow 20000:20100/udp   # match your Janus media range
# Only if you enable TURNS (section 3.4):
sudo ufw allow 5349/tcp         # required for TURNS
# sudo ufw allow 5349/udp       # optional; enable only if you need UDP on 5349

Note: Choose a media port range that is allowed by your provider/network. The exact same range must be configured in both docker-compose.yml (Janus service ports) and janus/janus.jcfg (media min/max).

1.2. Resource Planning

The High-Performance Backend, especially the Janus MCU, can be resource-intensive during video calls. For a small group of users (e.g., 3-5 concurrent participants in a call), plan for at least 1-2 dedicated CPU cores and 2-4 GB of RAM for the HPBE stack. For larger deployments, monitor your resource usage and scale accordingly. To keep an eye on performance, regularly check resource usage with tools like htop, docker stats, or a more comprehensive monitoring stack like Prometheus and Grafana.

2. Directory Structure and Download

First, we will clone the official repository, which contains all the necessary Docker configurations.

# Navigate to your main containers directory
cd /opt/containers/

# Clone the repository
sudo git clone https://github.com/strukturag/nextcloud-spreed-signaling.git nextcloud-hpbe

# Enter the new directory
cd nextcloud-hpbe

3. Configuration Files

This setup uses two main configuration files: docker-compose.yml for defining the services and server.conf for the signaling server itself. All configuration values will be hardcoded directly into these files for simplicity and clarity.

3.1. Docker Compose File (docker-compose.yml)

Create the docker-compose.yml file. This version uses a hardcoded configuration and is optimized for Traefik v3.

sudo tee docker-compose.yml > /dev/null << 'EOF'

services:
  spreedbackend:
    build:
      context: .
      dockerfile: docker/server/Dockerfile
      platforms:
        - "linux/amd64"
    container_name: spreedbackend
    depends_on:
      - nats
      - janus
      - coturn
    volumes:
      - ./server.conf:/config/server.conf:ro
    restart: unless-stopped
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      # Router for certificate acquisition (Host-only rule)
      - "traefik.http.routers.hpbe-cert.rule=Host(`signaling.your-domain.com`)"
      - "traefik.http.routers.hpbe-cert.entrypoints=web,websecure"
      - "traefik.http.routers.hpbe-cert.tls.certresolver=tls_resolver"
      - "traefik.http.routers.hpbe-cert.priority=1"
      # Router for the actual service (Host + Path rule)
      - "traefik.http.routers.hpbe.rule=Host(`signaling.your-domain.com`) && PathPrefix(`/standalone-signaling`)"
      - "traefik.http.routers.hpbe.entrypoints=websecure"
      - "traefik.http.routers.hpbe.tls=true"
      - "traefik.http.routers.hpbe.priority=100"
      # Middlewares: Set X-Forwarded-Proto and strip the path prefix
      - "traefik.http.middlewares.hpbe-headers.headers.customRequestHeaders.X-Forwarded-Proto=https"
      - "traefik.http.middlewares.hpbe-strip.stripprefix.prefixes=/standalone-signaling"
      - "traefik.http.routers.hpbe.middlewares=hpbe-headers@docker,hpbe-strip@docker,crowdsec-bouncer@docker"
      # Internal service port
      - "traefik.http.services.hpbe.loadbalancer.server.port=8080"

  nats:
    image: nats:2.10
    container_name: nats
    command: ["-c", "/config/gnatsd.conf"]
    volumes:
      - type: bind
        source: ./gnatsd.conf
        target: /config/gnatsd.conf
        read_only: true
    restart: unless-stopped
    networks:
      - proxy

  janus:
    # Build Janus from source using the provided Dockerfile
    build: docker/janus
    container_name: janus
    command: ["janus", "--full-trickle"]
    restart: unless-stopped
    networks:
      - proxy
    ports:
      - "20000-20100:20000-20100/udp"
      # Optional only if you enabled ice_tcp=true (see section 3.3)
      # - "20000-20100:20000-20100/tcp"
    volumes:
      - ./janus/janus.jcfg:/usr/local/etc/janus/janus.jcfg:ro

  coturn:
    image: coturn/coturn:4.6
    container_name: coturn
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
    command:
      - "--realm"
      - "signaling.your-domain.com"
      - "--static-auth-secret"
      - "PASTE-A-STRONG-RANDOM-32-CHAR-HEX-SECRET-HERE"
      - "--no-stdout-log"
      - "--log-file"
      - "stdout"
      - "--stale-nonce=600"
      - "--use-auth-secret"
      # Using shared-secret auth only; lt-cred-mech omitted to avoid mixed-auth warning
      - "--fingerprint"
      - "--no-software-attribute"
      - "--no-multicast-peers"
      - "--min-port"
      - "30000"
      - "--max-port"
      - "30100"
      - "--cert"
      - "/certs/fullchain.pem"
      - "--pkey"
      - "/certs/privkey.pem"
      - "--tls-listening-port"
      - "5349"
    ports:
      - "3478:3478/tcp"
      - "3478:3478/udp"
      - "5349:5349/tcp"
      - "5349:5349/udp"
      - "30000-30100:30000-30100/udp"
    volumes:
      - ./certs:/certs:ro
    restart: unless-stopped

networks:
  proxy:
    external: true
EOF

Note: Before launching, create janus/janus.jcfg as described in section 3.3 (set nat_1_1_mapping and the min_port/max_port media port range).

Note: On hosts with a direct public IP, you typically do not need to set Coturn’s --listening-ip, --relay-ip, or --external-ip. Relying on defaults avoids common binding errors (e.g., “Cannot assign requested address”, errno=99). Only set --external-ip PUBLIC_IP/PRIVATE_HOST_IP (and optionally --listening-ip/--relay-ip to the private host IP) if your host is behind NAT.

3.2. Signaling Server Config (server.conf)

This file configures the core logic of the HPBE. Create server.conf, paste the template below, and replace the placeholder values with your own secrets and URLs.

sudo tee server.conf > /dev/null << 'EOF'
[http]
listen = 0.0.0.0:8080

[app]
debug = false

[sessions]
# Use 'openssl rand -base64 16' to generate these
hashkey = PASTE-A-RANDOM-BASE64-KEY-HERE
blockkey = PASTE-A-RANDOM-BASE64-KEY-HERE

[backend]
backends = backend-1 #, backend-2
allowall = false
timeout = 10
connectionsperhost = 8

[backend-1]
url = https://cloud.your-domain.com
# Use 'openssl rand -hex 16' to generate this
secret = PASTE-A-RANDOM-HEX-SECRET-HERE

# To add a second backend, add it to the 'backends' list and create a new section
# [backend-2]
# url = https://another-cloud.com
# secret = PASTE-ANOTHER-RANDOM-HEX-SECRET-HERE

[nats]
url = nats://nats:4222

[mcu]
type = janus
url = ws://janus:8188

[turn]
# Use 'openssl rand -base64 16' to generate this
apikey = PASTE-A-RANDOM-BASE64-KEY-HERE
# This secret MUST be identical to the '--static-auth-secret' in docker-compose.yml
secret = PASTE-THE-SAME-STRONG-SECRET-AS-IN-DOCKER-COMPOSE
servers = turn:signaling.your-domain.com:3478?transport=udp,turn:signaling.your-domain.com:3478?transport=tcp
EOF

Ensure the file has the correct permissions: sudo chmod 644 server.conf.

3.2.1 Replace placeholders and generate secrets

Before launching, perform these steps so your setup works on your domain and with strong secrets.

1. Replace the domain placeholder

We use signaling.your-domain.com as a placeholder. Replace it with your real domain in both files:

cd /opt/containers/nextcloud-hpbe
sudo sed -i 's/signaling\.your-domain\.com/signaling.example.com/g' docker-compose.yml server.conf

Replace signaling.example.com with your actual TURN/signaling domain.

2. Generate the required secrets with openssl

• Sessions keys (base64) for server.conf [sessions]:

openssl rand -base64 16  # paste as sessions.hashkey
openssl rand -base64 16  # paste as sessions.blockkey

• Backend shared secret (hex) for each Nextcloud in server.conf [backend-*]:

openssl rand -hex 16     # paste as backend-1.secret (and for backend-2, backend-3, ...)

• TURN API key (base64) for server.conf [turn] apikey:

openssl rand -base64 16  # paste as turn.apikey

• TURN static secret (hex) used in BOTH places:
  • server.conf [turn] secret
  • docker-compose coturn command --static-auth-secret

openssl rand -hex 16     # paste into both places, values must be identical

3. Open firewall ports (host level)

sudo ufw allow 3478/tcp
sudo ufw allow 3478/udp
sudo ufw allow 20000:20100/udp   # Janus media range
sudo ufw allow 30000:30100/udp   # Coturn relay range
# The main docker-compose includes TURNS by default:
sudo ufw allow 5349/tcp         # required for TURNS
# sudo ufw allow 5349/udp       # optional; some clients might use it

3.2.2 Example: Multiple Nextcloud backends

If you operate more than one Nextcloud that should use the same signaling backend, model your [backend] section like this. Replace the example domains and secrets with your own.

[backend]
backends = backend-1, backend-2, backend-3
allowall = false
timeout = 10
connectionsperhost = 8

[backend-1]
url = https://cloud1.example.com
secret = PASTE-A-RANDOM-HEX-SECRET-HERE

[backend-2]
url = https://cloud2.example.com
secret = PASTE-A-RANDOM-HEX-SECRET-HERE

[backend-3]
url = https://cloud3.example.com
secret = PASTE-A-RANDOM-HEX-SECRET-HERE

Note: The secret for each backend must match the “Shared secret” you configure in that specific Nextcloud under Admin -> Talk.

3.2.3 Optional: Bind the HTTP listener to the service name

By default this guide uses listen = 0.0.0.0:8080 in [http], which is simple and works well behind Traefik. If you prefer to bind strictly to the Docker service name, you can set:

[http]
listen = spreedbackend:8080

Both are valid in a single Docker network; choose the variant that fits your operational preference.

3.3. Janus NAT and Media Ports (required for working audio/video)

WebRTC media flows (DTLS/SRTP) are carried on a dynamic UDP port range on the Janus SFU. If these ports are not reachable from the internet, calls will not establish and you will see errors like “publisher not sending yet” or repeated “requestoffer: context deadline exceeded” in the signaling logs.

Do the following:

1. Create a minimal Janus config that sets public IP mapping and a fixed media port range

Create ./janus/janus.jcfg with the following content. Replace PUBLIC.IP.OR.DNS with your server’s public address or a DNS name that resolves to it (no CDN/Orange-Cloud in front):

# Minimal but complete Janus config for operation behind Docker
# with a fixed UDP port range and correct 1:1 NAT mapping.

general: {
  configs_folder = "/usr/local/etc/janus"
  # Keep logging at "info" (default), set "debug = true" if needed
}

at: {
  # Important: public IP or DNS of the host that clients should see
  nat_1_1_mapping = "signaling.your-domain.com"

  # SRFLX candidates (optional, but doesn't hurt)
  stun_server = "signaling.your-domain.com"
  stun_port   = 3478

  # ICE Lite is fine for SFU operation and reduces complexity
  ice_lite = true
}

media: {
  # Fixed, small port range: must match docker-compose (ports:)
  min_port = 20000
  max_port = 20100

  # Prefer UDP (TCP disabled as it's often problematic/unnecessary)
  ice_tcp = false

  # (Optional) Enforce RTCP-MUX – common default, can help
  rtcp_mux = true
}

# Websockets (HPBE communicates internally via ws://janus:8188)
# Defaults are fine, no extra port publishing needed as it's in the same Docker network.
websockets: {
  ws = true
  ws_port = 8188
  ws_interface = "0.0.0.0"
}

# REST & Admin disabled by default – not needed
admin: {
  admin_http = false
  admin_secret = "changeit"
}

2. Ensure your main docker-compose includes the Janus media ports and config mount

In section 3.1, the janus service already contains the required ports and volumes lines. Verify that your configuration matches:

• ports: "20000-20100:20000-20100/udp" (optional TCP only if ice_tcp=true)
• volumes: ./janus/janus.jcfg:/usr/local/etc/janus/janus.jcfg:ro

3. Open the firewall for the media range (host level)

sudo ufw allow 20000:20100/udp
# Adjust to the exact range you configured in janus.jcfg and docker-compose
# Only if you enabled ice_tcp=true
# sudo ufw allow 20000:20100/tcp

4. Apply the changes

• If you have not launched the stack yet, skip this step. The changes will take effect when you start the stack in section 4.
• If Janus is already running, apply the updated config now:

sudo docker compose up -d --force-recreate janus

5. Verify during a call attempt

• In Firefox about:webrtc or Chrome chrome://webrtc-internals check that the remote ICE candidates from Janus show your public IP with ports in 20000–20100 and that the ICE state becomes connected/completed.
• On the server, you should see traffic on those ports while a call is setting up:

sudo tcpdump -ni any udp port 3478
sudo tcpdump -ni any udp portrange 20000-20100

If you exclusively rely on shared-secret auth for TURN, you can still keep section 3.4 (TURNS) as-is; TURN helps with client NAT traversal but cannot replace opening Janus’ own media ports.

3.4. Enable TURN over TLS (5349)

Enabling turns: adds TLS encryption to TURN traffic on port 5349. This often helps in restrictive networks and hides credentials from passive observers.

Steps:

1. Create the certs directory and export PEMs from Traefik’s ACME store

Since this guide builds on the prerequisite Traefik v3 and CrowdSec tutorial, your certificates are stored in Traefik’s ACME database at /opt/containers/traefik-stack/traefik/certs/acme.json. Export the certificate and key for your TURN domain into ./certs.

cd /opt/containers/nextcloud-hpbe

# Domain used for TURN/TURNS (anonymized placeholder)
TURN_DOMAIN="signaling.your-domain.com"
ACME="/opt/containers/traefik-stack/traefik/certs/acme.json"

# Ensure tools are available
sudo apt-get update && sudo apt-get install -y jq

# Create output directory
mkdir -p ./certs
# (Directory permissions are set in step 1.1 below)

# Extract certificate and key for the TURN domain from acme.json
sudo jq -r --arg d "$TURN_DOMAIN" '
  .. | objects
  | select(has("domain") and (.domain.main==$d or ((.domain.sans // []) | index($d))))
  | .certificate
' "$ACME" | base64 -d | sudo tee ./certs/fullchain.pem > /dev/null

sudo jq -r --arg d "$TURN_DOMAIN" '
  .. | objects
  | select(has("domain") and (.domain.main==$d or ((.domain.sans // []) | index($d))))
  | .key
' "$ACME" | base64 -d | sudo tee ./certs/privkey.pem > /dev/null

Note: The certificate must match the TURN realm/domain you advertise (e.g., signaling.your-domain.com).

1.1 Set Certificate and Directory Permissions

Coturn does not run as root inside the container, but as the nobody:nogroup user. Without the correct permissions on both the directory and the files, Coturn cannot read the certificates and the turns: connection will fail (often shown as red in Nextcloud Talk).

# In your project directory (e.g., /opt/containers/nextcloud-hpbe)

# Detect the group ID used by Coturn inside the container (commonly 65534 for 'nogroup')
GID=$(docker compose exec -T coturn sh -c 'id -g' | tr -d '\r')

# Ensure the certs directory is accessible to that group
sudo chgrp "$GID" ./certs
sudo chmod 750 ./certs

# Set group ownership and restrictive permissions on the files
sudo chgrp "$GID" ./certs/privkey.pem ./certs/fullchain.pem
sudo chmod 640 ./certs/privkey.pem
sudo chmod 644 ./certs/fullchain.pem

# Restart Coturn to load the updated certificates
sudo docker compose up -d --force-recreate coturn

Note: If your image uses the default nobody:nogroup, you may alternatively use sudo chgrp nogroup ... instead of the detected GID.

2. TLS is already configured in the main docker-compose.yml in this guide. After creating/exporting the certificates in step 1, simply recreate Coturn so it picks up the TLS files.

Note: If you exclusively use shared-secret auth (--use-auth-secret with --static-auth-secret), you can omit --lt-cred-mech to avoid a warning.

3. Update your server.conf servers list to include the turns: endpoint:

[turn]
servers = turn:signaling.your-domain.com:3478?transport=udp,turn:signaling.your-domain.com:3478?transport=tcp,turns:signaling.your-domain.com:5349?transport=tcp

4. Open the firewall for port 5349:

sudo ufw allow 5349/tcp         # required for TURNS
# sudo ufw allow 5349/udp       # optional; enable only if you need UDP on 5349

4. Launch the Stack

With the configuration complete, you can now build and start the services.

# From within the /opt/containers/nextcloud-hpbe directory
sudo docker compose up --build -d

The --build flag is important because it builds the Janus and the spreedbackend images from their Dockerfiles. The Janus build, in particular, can take several minutes.

You can monitor the logs to ensure everything starts correctly:

sudo docker compose logs -f

4.1 Verify TURN/TURNS endpoints

If nc is not installed, install it first:

sudo apt-get update && sudo apt-get install -y netcat-openbsd

Run these quick checks from a client or your server to verify connectivity:

# 1) Check TCP reachability (expect "succeeded")
nc -vz signaling.your-domain.com 3478
nc -vz signaling.your-domain.com 5349  # only if TURNS enabled

# 2) Verify TLS on 5349 (should show certificate details; only if TURNS is enabled)
openssl s_client -connect signaling.your-domain.com:5349 -servername signaling.your-domain.com -brief < /dev/null

# 3) Check Coturn runtime logs
docker compose logs -f coturn

If you have turnutils_uclient available (from the coturn package), you can perform an end-to-end TURN allocation test as an advanced check.

5. Configure Nextcloud

The final step is to tell your Nextcloud instance to use the new High-Performance Backend.

1. Log in to Nextcloud as an administrator.
2. Navigate to Administration Settings -> Talk.
3. Scroll down to the “Signaling server” section.
4. Check “Enable custom signaling server”.
5. In the “Signaling server URL” field, enter the full path to your HPBE: https://signaling.your-domain.com/standalone-signaling.
6. In the “Shared secret” field, paste the corresponding shared secret for this specific Nextcloud instance (e.g., the value of NEXTCLOUD_1_SHARED_SECRET for your first instance).
7. Click “Save changes”.

Nextcloud will verify the connection. If everything is correct, you’re all set!

Clarifying Secret Mappings

To ensure all components communicate securely, it’s crucial to map the secrets correctly. Here is a quick reference:

6. Troubleshooting & Monitoring

If you encounter issues, here are a few steps to diagnose the problem:

1. Verify the Signaling Server is Reachable

   You should get a 200 OK response from the /welcome endpoint. This confirms that Traefik is routing requests correctly.

   curl -i https://signaling.your-domain.com/standalone-signaling/api/v1/welcome
   

2. Check the Container Logs

   The logs are the best source for identifying errors. Pay close attention to messages about secrets, WebSocket connections, or backend timeouts.

   # From within the /opt/containers/nextcloud-hpbe directory
   sudo docker compose logs -f
   

3. Common Errors & Fixes

   • ERROR: the sessions block key must be... or hash key should be...: This error occurs when the keys in server.conf are missing or have the wrong format. Use openssl rand -base64 16 to generate new keys and paste them into the [sessions] section.
   • TURN Secret Mismatch: If calls fail to connect, verify that the secret in the [turn] section of server.conf is exactly the same as the --static-auth-secret in the coturn command in docker-compose.yml.
   • “failed to establish signaling connection”: This is a classic error.
     • Check that the URL in Nextcloud is exactly https://.../standalone-signaling.
     • Ensure your Traefik labels are correct (especially the PathPrefix and stripPrefix rules).
     • Verify that the Shared secret in Nextcloud matches the one in server.conf.
     • Media range not reachable: If ICE repeatedly fails or you see “publisher not sending yet”, ensure your chosen Janus media port range is open end-to-end (firewall/CrowdSec), and that the exact same range is configured in both docker-compose.yml (Janus ports) and janus/janus.jcfg.
   • Calls work for 2 people but fail with 3+: This often points to a problem with Janus (the MCU) or Coturn (the TURN server). Check their logs specifically.
   • No video/audio from external networks: This is a typical TURN server issue. Ensure ports 3478 (TCP/UDP) are open on your firewall and correctly forwarded to the Coturn container.
   • turn: works, but turns: fails (is red in Nextcloud): This is almost always a certificate permission issue. Coturn cannot read the TLS certificate or key. Verify that Coturn can access the files with docker compose exec coturn ls -l /certs. If you see permission errors, re-run the permission-setting steps in section 3.4. You can also test externally with: openssl s_client -connect signal.example.com:5349 -servername signal.example.com.

6.1 Diagnose ICE/RTP issues quickly

• Check remote candidates in the browser: Use about:webrtc (Firefox) or chrome://webrtc-internals (Chrome). You should see server-reflexive (srflx) and relayed (relay) candidates, and remote candidates from Janus on the public IP in the configured media range.
• Look for ICE state: iceConnectionState should reach connected/completed. If it stays in checking/failed, open ports or NAT/public IP mapping are missing.
• Server-side packet capture: While attempting a call, run sudo tcpdump -ni any udp portrange 20000-20100 to confirm media packets hit the host.
• TURN sanity: Use the Trickle ICE demo with your TURN URIs to confirm you get relay candidates; verify your Coturn realm and static secret match the values in server.conf and docker-compose.yml.

7. Maintenance and Updates

Updating the High-Performance Backend involves fetching the latest version while preserving your custom configurations. The recommended method is to back up your current installation, clone the new version, and restore your configuration files. This avoids potential conflicts from a direct git pull.

Step 1: Stop and Back Up the Current Installation

First, stop the running services and create a backup of your entire nextcloud-hpbe directory.

# Navigate to the parent directory of your HPBE installation
cd /opt/containers/

# Stop the services using the existing docker-compose file
sudo docker compose -f nextcloud-hpbe/docker-compose.yml down

# Create a backup by renaming the directory
sudo mv nextcloud-hpbe nextcloud-hpbe_BACKUP

Step 2: Clone the New Version

Clone the latest version of the repository into a clean directory with the original name.

# Stay in /opt/containers/
sudo git clone https://github.com/strukturag/nextcloud-spreed-signaling.git nextcloud-hpbe

Step 3: Restore Your Configuration

Copy your essential configuration files from the backup into the new directory. This ensures your secrets, domains, and other settings are preserved.

# Copy your docker-compose.yml and server.conf
sudo cp nextcloud-hpbe_BACKUP/docker-compose.yml nextcloud-hpbe/docker-compose.yml
sudo cp nextcloud-hpbe_BACKUP/server.conf nextcloud-hpbe/server.conf

Step 4: Launch the Updated Stack

Finally, navigate into the new directory and start the services. The --build flag will create new images if required, and --remove-orphans cleans up any old, unused containers.

# Navigate into the new HPBE directory
cd nextcloud-hpbe

# Build and start the updated services
sudo docker compose up --build -d --remove-orphans

After a few moments, your updated High-Performance Backend will be running. You can optionally remove the backup directory (sudo rm -rf /opt/containers/nextcloud-hpbe_BACKUP) once you have confirmed everything is working correctly.

Conclusion

Congratulations! You have successfully deployed a Nextcloud Talk High-Performance Backend. Your users will now experience more stable and performant video calls, especially in group settings. This powerful, containerized setup integrates perfectly with a modern Traefik proxy, providing a scalable and secure solution for your communication needs.