Requirements and sizing
The infrastructure you need before self-hosting DataxLand (Kubernetes, Postgres 16, S3-compatible object storage, a shared volume, ingress and TLS), the default resources of each component and how to pick object storage on each cloud.
Infrastructure you need
| Item | Requirement |
|---|---|
| Kubernetes | 1.27 or later, amd64 or arm64 nodes (images are multi-arch) |
| Postgres | Version 16, a managed service (RDS, Cloud SQL, Azure Database for PostgreSQL) or your own Postgres cluster, with backups |
| Databases | Two databases with the same user: dx (metadata) and dx_lake (warehouse catalog). The application creates dx_lake itself if the user has CREATEDB |
| Object storage | S3-compatible: two buckets, one for the warehouse (default dx-lake) and one for backups (default dx-backups) |
| Shared volume | A StorageClass that can provision ReadWriteMany PVCs (EFS, Filestore, Azure Files, NFS, CephFS). Single node: ReadWriteOnce is enough |
| Ingress | An ingress controller in the cluster and a domain for web, for example dataxland.corp.example |
| TLS | A certificate in a Secret, or cert-manager issuing one automatically |
| An SMTP server for invitations, password resets and reports | |
| AI model | An API key from a model provider, or an in-house OpenAI-compatible model (vLLM, Ollama, LM Studio, TGI) |
Optional: External Secrets Operator (reads secrets from your secret store), Prometheus Operator (collects metrics), a CNI that supports NetworkPolicy (Calico, Cilium, Azure NPM, GKE Dataplane V2, AWS VPC CNI with policy enabled).
Pods run as a non-root user (uid 10001) with a read-only root filesystem, all capabilities dropped and
seccompProfile: RuntimeDefault. These settings match the Pod Security "restricted" standard; we have not yet tested on a
cluster that enforces it.
Default resources
From the chart's values.yaml. "Request" is what Kubernetes reserves; "limit" is the memory ceiling of each pod.
| Component | Replicas | CPU request | Memory request | Memory limit |
|---|---|---|---|---|
api | 2 | 0.5 | 1 Gi | 3 Gi |
web | 2 | 0.1 | 256 Mi | 768 Mi |
mcp | 1 | 0.25 | 512 Mi | 2 Gi |
worker | 1 | 1 | 2 Gi | 7 Gi |
| Total | 2.45 | 5 Gi | 16.5 Gi |
Tune them with api.resources, web.resources, mcp.resources and worker.resources. Most of the memory of api and mcp
goes to read-only warehouses: api.readers (default max: 8, memory: 256MB) and mcp.readers (max: 4, memory: 256MB);
the product of the two numbers is the read-warehouse memory ceiling per pod. worker.threads (default 4) is the number of
threads used for builds.
Sizing guide
The numbers below are starting estimates, not measurements of your workload. Watch memory and build times for the first few weeks, then adjust.
| Small | Medium | Large | |
|---|---|---|---|
| Workspaces | up to 10 | 10 to 50 | over 50 |
| Kubernetes nodes | 3 × 4 vCPU, 16 GB | 3 × 8 vCPU, 32 GB | 4 or more × 8 vCPU, 32 GB |
worker memory limit | 7 Gi (default) | 12 Gi | 16 Gi or more |
api | 2 replicas (default) | enable api.autoscaling | enable api.autoscaling, raise maxReplicas |
| Postgres | 2 vCPU, 4 GB | 4 vCPU, 16 GB | 8 vCPU, 32 GB |
| Shared volume | 100 Gi (default) | 250 Gi | 500 Gi or more |
Object storage capacity depends on how much source data you have; compressed Parquet is usually much smaller than the original. The backup bucket needs room for the retained copies (by default 7 daily, 4 weekly, 6 monthly).
worker always runs as one replica, so as data grows you raise its memory and worker.threads, not its replica count.
Object storage on each cloud
DataxLand speaks the S3 protocol. How to connect each option, set under objectStorage in your values:
| Storage | objectStorage.endpoint | urlStyle | Credentials | Notes |
|---|---|---|---|---|
| AWS S3 | s3.<region>.amazonaws.com | vhost | Static access key of an IAM user | IRSA and Pod Identity are not supported |
| Google Cloud Storage | storage.googleapis.com | path | HMAC key of a service account | Through the S3-compatible API; paths stay s3://, not gs:// |
| Azure Blob Storage | leave empty, enable s3proxy.enabled | path | An S3 key pair you set for s3proxy | s3proxy runs in the cluster in front of the Storage Account. Not tested on a real account |
| On-prem | MinIO, Ceph RGW, or a local provider's S3-compatible object storage | usually path | Per system | MinIO allows per-workspace keys |
- On S3, GCS and s3proxy, set
objectStorage.tenantKeys: "off": one key for the whole bucket, each workspace's data under its own prefix. - With MinIO,
objectStorage.tenantKeys: miniogives each workspace a key that can only read and write its own area. The MinIO admin key then lives only inworker; the chart removes it fromapiandmcpautomatically. - An alternative on Azure if you would rather not use s3proxy: run your own MinIO on Azure Disk.
- Enable versioning on both buckets, and write-once locking (Object Lock, retention, immutability) on the backup bucket.
Network
Pods must be able to reach Postgres, object storage, the AI model server, SMTP, the source databases you plan to connect, and the cluster DNS. In air-gapped mode that is also the complete allow-list; see Air-gapped mode.
Related
Self-hosting (Enterprise plan)
Run DataxLand in your own Kubernetes infrastructure with a Helm chart, an offline license and an air-gapped mode; when to choose it, what it includes, how it differs from the cloud and its known limits.
Install with Helm
Install DataxLand on Kubernetes with the Helm chart, including the secrets Secret, the license Secret, minimal production values, post-install checks, a trial setup with in-cluster Postgres and MinIO, External Secrets, NetworkPolicy and ServiceMonitor.