Self-hosting (Enterprise)

Upgrades, backups and restore

Upgrade a self-hosted installation with helm upgrade (the migrate Job runs first), roll back, take daily backups with the worker or a CronJob, rehearse restores and run a real restore with the backup-restore command.

The commands below assume the release name dataxland in the namespace dataxland. Operational commands run in the worker pod:

kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli <command>

Upgrading

  1. Read the release notes for the new version: does it change the schema, can it be rolled back, is there any manual step?

  2. Back up right away and confirm the new backup has status ok:

    kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli backup
    kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli backup-list
  3. Closed network: copy the new version's images into your internal registry first.

  4. Upgrade:

    helm upgrade dataxland <new chart path> -n dataxland -f values-prod.yaml

    Helm runs the migrate Job first (python -m pipelines.cli db-migrate): it creates any missing tables, columns and indexes in Postgres. The Job is safe to run repeatedly; its time limit is set by migrations.activeDeadlineSeconds (default 600 seconds). Once the Job finishes, api, web and mcp are replaced gradually, while worker stops and starts on the new version: background work (builds, refresh schedules, backups) pauses for a few minutes, dashboards and asking Si keep working.

  5. Check: kubectl -n dataxland get pods, call the api's /readyz as during installation, open a dashboard and look at Admin → Operations.

If the migrate Job fails, Helm stops and the running components stay on the old version. Read the logs with kubectl -n dataxland logs job/dataxland-migrate.

Rolling back

helm rollback dataxland <revision> is only safe when the new version has not changed the schema in a way the old version cannot read; the release notes say so explicitly. If in doubt, the default path is to reinstall the old version and restore the backup taken in step 2.

Backups

A backup contains the Postgres metadata, each workspace's files (uploads, configuration, approvals, encrypted secrets) and the data warehouse of every workspace. Every file is encrypted with DX_BACKUP_ENCRYPTION_KEY before it is written to the backup bucket. Warehouse data is uploaded once and later backups reference it, so subsequent runs are smaller than the first.

Schedule

By default the worker backs up once a day:

deployment:
  timezone: Asia/Ho_Chi_Minh
objectStorage:
  backup:
    enabled: true
    bucket: dx-backups
    prefix: ""
    endpoint: ""        # empty = same endpoint as the warehouse
    at: "02:00"         # time in deployment.timezone
    keep: "7,4,6"       # keep 7 daily, 4 weekly, 6 monthly backups

To let Kubernetes own the schedule, enable the CronJob. The chart then turns off the worker's own schedule:

backupCronJob:
  enabled: true
  schedule: "0 19 * * *"          # 02:00 Vietnam time if the cluster runs on UTC
  restoreDrill:
    enabled: true
    schedule: "0 21 * * 0"        # weekly drill

Recommendations:

  • Put the backup bucket in a different region or system from the warehouse bucket. You can use separate keys (DX_BACKUP_S3_KEY_ID, DX_BACKUP_S3_SECRET) and a separate endpoint (objectStorage.backup.endpoint).
  • Enable versioning and write-once locking (Object Lock, retention, immutability) on the backup bucket.
  • Never delete the _objects/ folder in the backup bucket by hand: every backup needs it to restore the warehouse.
  • Keep the backups of your managed Postgres service as well; the two layers complement each other.

Keep both keys in two places

These two keys are not part of any backup and cannot be recovered:

  • DX_SECRET_KEY: decrypts stored source passwords and tokens. Lose it and every data source has to be reconnected.
  • DX_BACKUP_ENCRYPTION_KEY: decrypts backups. Lose it and every backup encrypted with it becomes unusable.

Keep each key in two places: your company secret store and an offline copy (a printout or an encrypted USB drive in a safe). Keep the two keys apart, managed by different people if possible.

Restore drills

kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli restore-drill
kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli restore-drill --backup <backup ID>

The command restores the newest backup (or the one you specify) into a temporary database and a temporary storage area, compares row counts and the totals of numeric columns with the time of the backup, verifies file checksums, then deletes the temporary copies. It never touches live data. The Postgres user needs permission to create databases.

Run it at least weekly (or enable backupCronJob.restoreDrill). A successful drill also proves the DX_BACKUP_ENCRYPTION_KEY you hold is the right one. The metric dx_backup_last_success_timestamp_seconds (label kind) shows the most recent successful backup and drill.

Real restore

backup-restore overwrites data. Pick the backup first:

kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli backup-list

Parameters of backup-restore:

ParameterEffect
--backup <ID>The backup; required
--metaLoad the Postgres metadata; every table in the target database must be empty
--tenant <workspace ID>Restore this workspace's files and warehouse; can be repeated
--all-tenantsEvery workspace in the backup
--forceOverwrite files and replace the existing warehouse

One damaged workspace

The metadata is intact, only one workspace's warehouse or files are damaged:

kubectl -n dataxland exec deploy/dataxland-worker -- \
  python -m pipelines.cli backup-restore --backup <ID> --tenant <workspace ID> --force

Then run a refresh for that workspace to load source data newer than the backup.

Full restore

When you have lost the Postgres metadata, lost the warehouse bucket, or are rebuilding on a new cluster:

  1. Create a new Postgres with empty dx and dx_lake databases.

  2. Install the chart at the same version as the installation at backup time, with the same DX_SECRET_KEY, DX_BACKUP_ENCRYPTION_KEY and backup location. Keep users out for now: kubectl -n dataxland scale deploy/dataxland-web --replicas=0 (turn off web.autoscaling if it is on).

  3. Load the metadata:

    kubectl -n dataxland exec deploy/dataxland-worker -- \
      python -m pipelines.cli backup-restore --backup <ID> --meta
  4. Load the files and warehouse of every workspace:

    kubectl -n dataxland exec deploy/dataxland-worker -- \
      python -m pipelines.cli backup-restore --backup <ID> --all-tenants

    Warehouse data comes from the backup bucket, so the new warehouse bucket can start empty.

  5. Restart the components (kubectl -n dataxland rollout restart deploy), call /readyz, open a workspace's dashboard and compare it with the numbers from before the incident. You can also run restore-drill --backup <ID>.

  6. Reopen web (kubectl -n dataxland scale deploy/dataxland-web --replicas=2). The next scheduled refresh reloads source data from after the backup; Excel files uploaded after that point must be uploaded again.

If --meta reports that the target database already has data, or you are unsure which backup to pick, contact DataxLand before going further.

Markdown