Log Locations
Systemd Journal
All Phonemos services log to the systemd journal:
1
2
3
4
5
6
7
8
9
10
11
# Follow all Phonemos service logs
sudo journalctl -u phonemos-* -f
# Logs for a specific service
sudo journalctl -u phonemos-keycloak -f
# Logs from the last hour
sudo journalctl -u phonemos --since "1 hour ago"
# Logs since last boot
sudo journalctl -u phonemos-* -bDocker Container Logs
1
2
3
4
5
# Follow a container's logs
docker logs -f hasura
# Last 200 lines with timestamps
docker logs --tail 200 -t backendKey containers and what they log:
Container | Log Content |
|---|---|
ingress | Nginx access and error logs (stdout/stderr) |
hasura | GraphQL engine startup, HTTP requests, webhook calls |
backend | Application logic, health checks, Keycloak integration |
postgres | Database startup, queries, connection errors |
keycloak | Authentication events, realm operations |
minio-filestore | Object storage requests |
Health Check Endpoints
Use these commands to verify individual services are responding:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# Hasura GraphQL engine
docker exec hasura curl -f http://localhost:8080/healthz
# Backend
docker exec backend wget --spider -q http://localhost:8080/health
# Backend PDF
docker exec backend-pdf curl -f http://localhost:8080/api/pdf/health
# MinIO
docker exec minio-filestore curl -I http://localhost:9000/minio/health/live
# Frontend
docker exec frontend curl -f http://localhost:8080/
# PostgreSQL
docker exec postgres pg_isready --user=postgres
# Keycloak PostgreSQL
docker exec keycloak-postgres pg_isready --user=postgresCommon Issues
Services Fail to Start
Symptoms: systemctl start phonemos-* fails or containers exit immediately.
1
2
3
4
5
6
7
8
9
# Service status and recent logs
sudo systemctl status phonemos-<name> -l
sudo journalctl -u phonemos-<name> --since "5 minutes ago"
# Docker container status
docker ps -a
# Docker daemon status
sudo systemctl status dockerCommon causes:
Docker daemon not running — sudo systemctl start docker
Insufficient disk space — df -h and docker system df
Port conflict — another process on port 80 or 443
Image not pulled — run sudo -u phonemos /opt/phonemos-linux/scripts/pull-images.sh
502 / 504 Errors from Nginx
Symptoms: Browser shows "502 Bad Gateway" or "504 Gateway Timeout".
Cause: The upstream service is not running or not ready.
1
2
3
4
5
6
7
8
# Check Nginx logs for the failing upstream
docker logs --tail 50 ingress
# Check if the target container is running
docker ps | grep <service-name>
# Check the service health
docker inspect --format='{{.State.Health.Status}}' <container-name>Fix: Start the missing service or wait for its health check to pass. Services should be started in the correct order (keycloak, minio, ingress, postgres, phonemos).
TLS Certificate Errors
Symptoms: Browser shows certificate warning, or services cannot connect via HTTPS.
1
2
3
4
5
6
7
8
# Verify certificate files exist
ls -la /etc/phonemos/ingress/certs/
# Check certificate details
openssl x509 -in /etc/phonemos/ingress/certs/cert.crt -noout -subject -dates
# Test TLS connection
openssl s_client -connect localhost:443 -servername <hostname>Common causes:
Certificate files missing — re-run scripts/setup-certificates.sh
Certificate expired — replace with a new certificate and restart ingress
Hostname mismatch — verify the certificate's CN/SAN matches the configured hostname
Self-signed cert not trusted — ensure it was added to the system CA store
Keycloak Unreachable
Symptoms: Login page does not load, or /auth returns an error.
1
2
3
4
5
6
7
8
9
10
11
# Is the Keycloak container running?
docker ps | grep keycloak
# Check Keycloak logs
docker logs --tail 100 keycloak
# Is keycloak-postgres healthy?
docker exec keycloak-postgres pg_isready --user=postgres
# Test from within the ingress network
docker exec ingress curl -f http://keycloak:8080/authCommon causes:
keycloak-postgres not healthy — check its logs with docker logs keycloak-postgres
Keycloak still starting — it can take 5–10 minutes on first boot
Network issue — verify phonemosKeycloak and phonemosIngress networks exist: docker network ls
Database Connection Errors
Symptoms: Hasura or backend fail to start with database connection errors.
1
2
3
4
5
6
7
8
9
# Is PostgreSQL running?
docker exec postgres pg_isready --user=postgres
# Check PostgreSQL logs
docker logs --tail 50 postgres
# Verify credentials match
grep DATABASE_PASSWORD /etc/phonemos/app/config
grep DB_PASSWORD /etc/phonemos/postgres/configCommon causes:
PostgreSQL not running — sudo systemctl start phonemos-postgres
Password mismatch — the password in /etc/phonemos/app/config must match /etc/phonemos/postgres/config
Database not initialized — check if init-databases.sh ran successfully in the postgres logs
Docker Image Pull Failures
Symptoms: pull-images.sh fails with authentication or network errors.
Fixes:
Re-run scripts/setup-docker.sh to reconfigure registry authentication.
Verify network connectivity to europe-west6-docker.pkg.dev.
Permission Errors
Symptoms: Services fail with "permission denied" errors.
1
2
3
4
5
6
7
8
9
10
11
# Verify phonemos user exists and is in docker group
id phonemos
# Fix data directory ownership
sudo chown -R phonemos:phonemos /data/volumes/
# Fix certificate ownership
sudo chown phonemos:phonemos /etc/phonemos/ingress/certs/*
# Ensure phonemos is in docker group
sudo usermod -aG docker phonemosApplication Not Accessible After Start
Symptoms: All services show as running, but the browser cannot reach the application.
Check:
DNS — verify the hostname resolves: nslookup <hostname>
Firewall — verify ports 80 and 443 are open: sudo iptables -L -n | grep -E "80|443"
Nginx — verify ingress is listening: sudo ss -tlnp | grep -E ":80|:443"
Certificates — verify TLS works: curl -v https://<hostname>/
Diagnostic Commands
Quick reference for common diagnostic commands:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# List all running containers
docker ps
# List all containers (including stopped)
docker ps -a
# List Docker networks
docker network ls
# Inspect a network (see connected containers)
docker network inspect phonemosIngress
# List Phonemos systemd units
systemctl list-units phonemos-*
# Check disk usage
df -h
docker system df
# Check memory usage
free -h
# Check running processes
docker stats --no-stream
# Inspect a container's configuration
docker inspect <container-name>
# Execute a command inside a container
docker exec -it <container-name> /bin/sh
# Restart Docker daemon (last resort)
sudo systemctl restart docker