Upgrades, backups and restore
Upgrade a self-hosted installation with helm upgrade (the migrate Job runs first), roll back, take daily backups with the worker or a CronJob, rehearse restores and run a real restore with the backup-restore command.
The commands below assume the release name dataxland in the namespace dataxland. Operational commands run in the worker
pod:
kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli <command>Upgrading
-
Read the release notes for the new version: does it change the schema, can it be rolled back, is there any manual step?
-
Back up right away and confirm the new backup has status
ok:kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli backup kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli backup-list -
Closed network: copy the new version's images into your internal registry first.
-
Upgrade:
helm upgrade dataxland <new chart path> -n dataxland -f values-prod.yamlHelm runs the
migrateJob first (python -m pipelines.cli db-migrate): it creates any missing tables, columns and indexes in Postgres. The Job is safe to run repeatedly; its time limit is set bymigrations.activeDeadlineSeconds(default 600 seconds). Once the Job finishes,api,webandmcpare replaced gradually, whileworkerstops and starts on the new version: background work (builds, refresh schedules, backups) pauses for a few minutes, dashboards and asking Si keep working. -
Check:
kubectl -n dataxland get pods, call theapi's/readyzas during installation, open a dashboard and look at Admin → Operations.
If the migrate Job fails, Helm stops and the running components stay on the old version. Read the logs with
kubectl -n dataxland logs job/dataxland-migrate.
Rolling back
helm rollback dataxland <revision> is only safe when the new version has not changed the schema in a way the old version
cannot read; the release notes say so explicitly. If in doubt, the default path is to reinstall the old version and restore
the backup taken in step 2.
Backups
A backup contains the Postgres metadata, each workspace's files (uploads, configuration, approvals, encrypted secrets) and the
data warehouse of every workspace. Every file is encrypted with DX_BACKUP_ENCRYPTION_KEY before it is written to the backup
bucket. Warehouse data is uploaded once and later backups reference it, so subsequent runs are smaller than the first.
Schedule
By default the worker backs up once a day:
deployment:
timezone: Asia/Ho_Chi_Minh
objectStorage:
backup:
enabled: true
bucket: dx-backups
prefix: ""
endpoint: "" # empty = same endpoint as the warehouse
at: "02:00" # time in deployment.timezone
keep: "7,4,6" # keep 7 daily, 4 weekly, 6 monthly backupsTo let Kubernetes own the schedule, enable the CronJob. The chart then turns off the worker's own schedule:
backupCronJob:
enabled: true
schedule: "0 19 * * *" # 02:00 Vietnam time if the cluster runs on UTC
restoreDrill:
enabled: true
schedule: "0 21 * * 0" # weekly drillRecommendations:
- Put the backup bucket in a different region or system from the warehouse bucket. You can use separate keys
(
DX_BACKUP_S3_KEY_ID,DX_BACKUP_S3_SECRET) and a separate endpoint (objectStorage.backup.endpoint). - Enable versioning and write-once locking (Object Lock, retention, immutability) on the backup bucket.
- Never delete the
_objects/folder in the backup bucket by hand: every backup needs it to restore the warehouse. - Keep the backups of your managed Postgres service as well; the two layers complement each other.
Keep both keys in two places
These two keys are not part of any backup and cannot be recovered:
DX_SECRET_KEY: decrypts stored source passwords and tokens. Lose it and every data source has to be reconnected.DX_BACKUP_ENCRYPTION_KEY: decrypts backups. Lose it and every backup encrypted with it becomes unusable.
Keep each key in two places: your company secret store and an offline copy (a printout or an encrypted USB drive in a safe). Keep the two keys apart, managed by different people if possible.
Restore drills
kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli restore-drill
kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli restore-drill --backup <backup ID>The command restores the newest backup (or the one you specify) into a temporary database and a temporary storage area, compares row counts and the totals of numeric columns with the time of the backup, verifies file checksums, then deletes the temporary copies. It never touches live data. The Postgres user needs permission to create databases.
Run it at least weekly (or enable backupCronJob.restoreDrill). A successful drill also proves the
DX_BACKUP_ENCRYPTION_KEY you hold is the right one. The metric dx_backup_last_success_timestamp_seconds (label kind) shows
the most recent successful backup and drill.
Real restore
backup-restore overwrites data. Pick the backup first:
kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli backup-listParameters of backup-restore:
| Parameter | Effect |
|---|---|
--backup <ID> | The backup; required |
--meta | Load the Postgres metadata; every table in the target database must be empty |
--tenant <workspace ID> | Restore this workspace's files and warehouse; can be repeated |
--all-tenants | Every workspace in the backup |
--force | Overwrite files and replace the existing warehouse |
One damaged workspace
The metadata is intact, only one workspace's warehouse or files are damaged:
kubectl -n dataxland exec deploy/dataxland-worker -- \
python -m pipelines.cli backup-restore --backup <ID> --tenant <workspace ID> --forceThen run a refresh for that workspace to load source data newer than the backup.
Full restore
When you have lost the Postgres metadata, lost the warehouse bucket, or are rebuilding on a new cluster:
-
Create a new Postgres with empty
dxanddx_lakedatabases. -
Install the chart at the same version as the installation at backup time, with the same
DX_SECRET_KEY,DX_BACKUP_ENCRYPTION_KEYand backup location. Keep users out for now:kubectl -n dataxland scale deploy/dataxland-web --replicas=0(turn offweb.autoscalingif it is on). -
Load the metadata:
kubectl -n dataxland exec deploy/dataxland-worker -- \ python -m pipelines.cli backup-restore --backup <ID> --meta -
Load the files and warehouse of every workspace:
kubectl -n dataxland exec deploy/dataxland-worker -- \ python -m pipelines.cli backup-restore --backup <ID> --all-tenantsWarehouse data comes from the backup bucket, so the new warehouse bucket can start empty.
-
Restart the components (
kubectl -n dataxland rollout restart deploy), call/readyz, open a workspace's dashboard and compare it with the numbers from before the incident. You can also runrestore-drill --backup <ID>. -
Reopen
web(kubectl -n dataxland scale deploy/dataxland-web --replicas=2). The next scheduled refresh reloads source data from after the backup; Excel files uploaded after that point must be uploaded again.
If --meta reports that the target database already has data, or you are unsure which backup to pick, contact DataxLand before
going further.
Related
Air-gapped mode
Run DataxLand on a network with no Internet access - the allow-list of hosts, the required NetworkPolicy, an in-house AI model, internal SMTP, features that turn off, data sources that still work and how to copy images into an internal registry.
Offline license
The license file of a self-hosted installation - what it contains, how to mount it as a Secret, the states and what they mean, the grace period, a read-only mode that never locks your data, renewal, the license-status command and the dx_license_days_left metric.