# Upgrades, backups and restore

> Upgrade a self-hosted installation with helm upgrade (the migrate Job runs first), roll back, take daily backups with the worker or a CronJob, rehearse restores and run a real restore with the backup-restore command.

The commands below assume the release name `dataxland` in the namespace `dataxland`. Operational commands run in the `worker`
pod:

```bash
kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli <command>
```

## Upgrading

1. **Read the release notes** for the new version: does it change the schema, can it be rolled back, is there any manual step?
2. **Back up right away** and confirm the new backup has status `ok`:

   ```bash
   kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli backup
   kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli backup-list
   ```

3. Closed network: [copy the new version's images](/en/docs/self-host/airgapped#7-copy-images-into-an-internal-registry) into
   your internal registry first.
4. **Upgrade**:

   ```bash
   helm upgrade dataxland <new chart path> -n dataxland -f values-prod.yaml
   ```

   Helm runs the `migrate` Job first (`python -m pipelines.cli db-migrate`): it creates any missing tables, columns and indexes
   in Postgres. The Job is safe to run repeatedly; its time limit is set by `migrations.activeDeadlineSeconds` (default 600
   seconds). Once the Job finishes, `api`, `web` and `mcp` are replaced gradually, while `worker` stops and starts on the new
   version: background work (builds, refresh schedules, backups) pauses for a few minutes, dashboards and asking Si keep working.
5. **Check**: `kubectl -n dataxland get pods`, call the `api`'s `/readyz` as during
   [installation](/en/docs/self-host/helm#6-check-and-sign-in), open a dashboard and look at **Admin → Operations**.

If the `migrate` Job fails, Helm stops and the running components stay on the old version. Read the logs with
`kubectl -n dataxland logs job/dataxland-migrate`.

### Rolling back

`helm rollback dataxland <revision>` is only safe when the new version **has not changed the schema in a way the old version
cannot read**; the release notes say so explicitly. If in doubt, the default path is to reinstall the old version and restore
the backup taken in step 2.

## Backups

A backup contains the Postgres metadata, each workspace's files (uploads, configuration, approvals, encrypted secrets) and the
data warehouse of every workspace. Every file is **encrypted with `DX_BACKUP_ENCRYPTION_KEY`** before it is written to the backup
bucket. Warehouse data is uploaded once and later backups reference it, so subsequent runs are smaller than the first.

### Schedule

By default the `worker` backs up once a day:

```yaml
deployment:
  timezone: Asia/Ho_Chi_Minh
objectStorage:
  backup:
    enabled: true
    bucket: dx-backups
    prefix: ""
    endpoint: ""        # empty = same endpoint as the warehouse
    at: "02:00"         # time in deployment.timezone
    keep: "7,4,6"       # keep 7 daily, 4 weekly, 6 monthly backups
```

To let Kubernetes own the schedule, enable the CronJob. The chart then turns off the `worker`'s own schedule:

```yaml
backupCronJob:
  enabled: true
  schedule: "0 19 * * *"          # 02:00 Vietnam time if the cluster runs on UTC
  restoreDrill:
    enabled: true
    schedule: "0 21 * * 0"        # weekly drill
```

Recommendations:

- Put the backup bucket in a different region or system from the warehouse bucket. You can use separate keys
  (`DX_BACKUP_S3_KEY_ID`, `DX_BACKUP_S3_SECRET`) and a separate endpoint (`objectStorage.backup.endpoint`).
- Enable versioning and write-once locking (Object Lock, retention, immutability) on the backup bucket.
- Never delete the `_objects/` folder in the backup bucket by hand: every backup needs it to restore the warehouse.
- Keep the backups of your managed Postgres service as well; the two layers complement each other.

### Keep both keys in two places

These two keys are **not part of any backup** and cannot be recovered:

- `DX_SECRET_KEY`: decrypts stored source passwords and tokens. Lose it and every data source has to be reconnected.
- `DX_BACKUP_ENCRYPTION_KEY`: decrypts backups. Lose it and **every backup encrypted with it becomes unusable**.

Keep each key in two places: your company secret store and an offline copy (a printout or an encrypted USB drive in a safe).
Keep the two keys apart, managed by different people if possible.

## Restore drills

```bash
kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli restore-drill
kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli restore-drill --backup <backup ID>
```

The command restores the newest backup (or the one you specify) into a temporary database and a temporary storage area,
compares row counts and the totals of numeric columns with the time of the backup, verifies file checksums, then deletes the
temporary copies. **It never touches live data.** The Postgres user needs permission to create databases.

Run it at least weekly (or enable `backupCronJob.restoreDrill`). A successful drill also proves the
`DX_BACKUP_ENCRYPTION_KEY` you hold is the right one. The metric `dx_backup_last_success_timestamp_seconds` (label `kind`) shows
the most recent successful backup and drill.

## Real restore

`backup-restore` overwrites data. Pick the backup first:

```bash
kubectl -n dataxland exec deploy/dataxland-worker -- python -m pipelines.cli backup-list
```

Parameters of `backup-restore`:

| Parameter | Effect |
| --- | --- |
| `--backup ` | The backup; required |
| `--meta` | Load the Postgres metadata; every table in the target database must be empty |
| `--tenant <workspace ID>` | Restore this workspace's files and warehouse; can be repeated |
| `--all-tenants` | Every workspace in the backup |
| `--force` | Overwrite files and replace the existing warehouse |

### One damaged workspace

The metadata is intact, only one workspace's warehouse or files are damaged:

```bash
kubectl -n dataxland exec deploy/dataxland-worker -- \
  python -m pipelines.cli backup-restore --backup  --tenant <workspace ID> --force
```

Then run a refresh for that workspace to load source data newer than the backup.

### Full restore

When you have lost the Postgres metadata, lost the warehouse bucket, or are rebuilding on a new cluster:

1. Create a new Postgres with empty `dx` and `dx_lake` databases.
2. Install the chart at **the same version** as the installation at backup time, with **the same** `DX_SECRET_KEY`,
   `DX_BACKUP_ENCRYPTION_KEY` and backup location. Keep users out for now:
   `kubectl -n dataxland scale deploy/dataxland-web --replicas=0` (turn off `web.autoscaling` if it is on).
3. Load the metadata:

   ```bash
   kubectl -n dataxland exec deploy/dataxland-worker -- \
     python -m pipelines.cli backup-restore --backup  --meta
   ```

4. Load the files and warehouse of every workspace:

   ```bash
   kubectl -n dataxland exec deploy/dataxland-worker -- \
     python -m pipelines.cli backup-restore --backup  --all-tenants
   ```

   Warehouse data comes from the backup bucket, so the new warehouse bucket can start empty.
5. Restart the components (`kubectl -n dataxland rollout restart deploy`), call `/readyz`, open a workspace's dashboard and
   compare it with the numbers from before the incident. You can also run `restore-drill --backup `.
6. Reopen `web` (`kubectl -n dataxland scale deploy/dataxland-web --replicas=2`). The next scheduled refresh reloads source data
   from after the backup; Excel files uploaded after that point must be uploaded again.

If `--meta` reports that the target database already has data, or you are unsure which backup to pick, contact DataxLand before
going further.

## Related

- [Install with Helm](/en/docs/self-host/helm)
- [Offline license](/en/docs/self-host/license)
- [Security and privacy](/en/docs/security-privacy)
