Backups

New in 0.2.1. Upgrade teriac server to 0.2.1 or later to use backups.

teriac server keeps runs, locks, the audit log, drift history and metrics in a SQLite database on its volume. Lose the volume and you lose all of that. Backups copy the database to a bucket on a schedule, and put it back by themselves when the server starts on an empty volume.

How it works

At each scheduled time the server:

1. copies its database (SQLite's VACUUM INTO, a complete copy even while runs are writing);
2. gzips the copy;
3. uploads it to S3 (or an S3-compatible service), Google Cloud Storage or Azure Blob Storage;
4. deletes the local copy, whether the upload worked or not.

Nothing builds up on the volume. A backup only needs free space about the size of the database while it runs. Each file is a whole database on its own, named by its time in UTC:

s3://acme-backups/teriac/teriac/teriac/teriac-20261111T030000000Z.db.gz

Turn it on with Helm

Fill in one of backup.backend.s3, .gcs or .azure. The one you fill in is used.

backup:
  enabled: true
  schedule: "0 3 * * *"          # every day at 03:00
  timezone: Europe/Berlin
  backend:
    s3:
      bucket: acme-backups       # name only; folders go in path
      region: eu-west-1
  notify: [infra-alerts]         # a channel from repos.yaml, told when a backup fails and when it works again

The chart checks the settings at install, so a mistake fails helm install, not a backup at 3am: exactly one backend, a five-field cron schedule, a bucket name without /, an Azure accountName, and an S3 endpoint starting with https:// or http://.

backup:
  enabled: true
  backend:
    gcs:
      bucket: acme-teriac-backups

backup:
  enabled: true
  backend:
    azure:
      accountName: acmebackups   # storage account
      container: teriac

backup:
  enabled: true
  backend:
    s3:
      bucket: teriac
      endpoint: https://minio.acme.dev
      region: us-east-1
    existingSecret: teriac-backup   # keys access-key-id and secret-access-key

All backup values

Turn it on with Docker

docker run -d -p 8080:8080 -v teriac-data:/data \
  -e TERIAC_BACKUP_ENABLED=true -e TERIAC_BACKUP_SCHEDULE='0 3 * * *' \
  -e TERIAC_BACKUP_BACKEND=s3 -e TERIAC_BACKUP_BUCKET=acme-backups -e TERIAC_BACKUP_S3_REGION=eu-west-1 \
  -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY \
  … ghcr.io/marcdomain/teriac-server:0.2.1

With backups on, every mistake (a bad cron, an unknown time zone, a bucket with /, a channel that isn't in repos.yaml) stops the server at start-up with one clear error.

Credentials

By default the server signs in as itself, so no keys are stored:

Otherwise give it keys:

Bucket permissions

teriac only lists, reads and writes. It never needs to delete objects or create buckets.

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::acme-backups" },
    { "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject"], "Resource": "arn:aws:s3:::acme-backups/teriac/*" }
  ]
}

Old backups

teriac never deletes backups. Remove old ones with a lifecycle (expiration) rule on the bucket, for the backup path. Since each backup is a whole database, expiring old ones never breaks newer ones.

Keep the expiry well above the time between backups (for daily backups, 7 days or more), so there's always a recent one to restore.

resource "aws_s3_bucket_lifecycle_configuration" "teriac_backups" {
  bucket = aws_s3_bucket.backups.id
  rule {
    id     = "expire-teriac-backups"
    status = "Enabled"
    filter { prefix = "teriac/" }
    expiration { days = 14 }
  }
}

Restoring

There's no restore command; it happens on start. When the server starts and its volume has no database (a new PVC, a lost disk, or persistence.enabled: false):

1. it lists the backups and takes the newest;
2. downloads it and runs SQLite's integrity check;
3. if that one is broken, isn't a teriac database, or the lifecycle rule removed it after it was listed, it tries the one before, up to the five newest;
4. then opens the restored database and starts as usual.

Changes made since the last backup are lost with the volume, so back up as often as that matters to you.

To move teriac to a new cluster, install it there with the same backup settings and an empty volume. It restores the newest backup on its first start.

Good to know

• One backup at a time. A Back up now while a backup is running joins that one instead of starting another.
• Shutdown waits for a backup that's uploading, so a rolling update doesn't cut it off.
• Crashes leave nothing behind. Half-written copies from an interrupted backup or restore are removed on the next start.
• No extra bucket settings. S3 buckets with ACLs disabled ("bucket owner enforced", the AWS default) and GCS buckets with uniform bucket-level access work as they are.
• No config file. teriac drives rclone through environment variables, so no credentials are written to the volume.

Watching it

• Dashboard: the Guardrails page shows where backups go, when the next one runs, the latest results, and a Back up now button for admins. Every manual backup is audited.
• Chat: with notify set, the channel hears when a backup fails, and again when backups work.
• Prometheus: /metrics has teriac_backup_last_ok and teriac_backup_last_success_timestamp_seconds.
• API: GET /api/backups, and POST /api/backups/run (admin) to start one.

# No good backup in 26 hours (daily schedule)
time() - teriac_backup_last_success_timestamp_seconds > 26 * 3600