Self-hosting (Enterprise)

Requirements and sizing

The infrastructure you need before self-hosting DataxLand (Kubernetes, Postgres 16, S3-compatible object storage, a shared volume, ingress and TLS), the default resources of each component and how to pick object storage on each cloud.

Infrastructure you need

ItemRequirement
Kubernetes1.27 or later, amd64 or arm64 nodes (images are multi-arch)
PostgresVersion 16, a managed service (RDS, Cloud SQL, Azure Database for PostgreSQL) or your own Postgres cluster, with backups
DatabasesTwo databases with the same user: dx (metadata) and dx_lake (warehouse catalog). The application creates dx_lake itself if the user has CREATEDB
Object storageS3-compatible: two buckets, one for the warehouse (default dx-lake) and one for backups (default dx-backups)
Shared volumeA StorageClass that can provision ReadWriteMany PVCs (EFS, Filestore, Azure Files, NFS, CephFS). Single node: ReadWriteOnce is enough
IngressAn ingress controller in the cluster and a domain for web, for example dataxland.corp.example
TLSA certificate in a Secret, or cert-manager issuing one automatically
EmailAn SMTP server for invitations, password resets and reports
AI modelAn API key from a model provider, or an in-house OpenAI-compatible model (vLLM, Ollama, LM Studio, TGI)

Optional: External Secrets Operator (reads secrets from your secret store), Prometheus Operator (collects metrics), a CNI that supports NetworkPolicy (Calico, Cilium, Azure NPM, GKE Dataplane V2, AWS VPC CNI with policy enabled).

Pods run as a non-root user (uid 10001) with a read-only root filesystem, all capabilities dropped and seccompProfile: RuntimeDefault. These settings match the Pod Security "restricted" standard; we have not yet tested on a cluster that enforces it.

Default resources

From the chart's values.yaml. "Request" is what Kubernetes reserves; "limit" is the memory ceiling of each pod.

ComponentReplicasCPU requestMemory requestMemory limit
api20.51 Gi3 Gi
web20.1256 Mi768 Mi
mcp10.25512 Mi2 Gi
worker112 Gi7 Gi
Total2.455 Gi16.5 Gi

Tune them with api.resources, web.resources, mcp.resources and worker.resources. Most of the memory of api and mcp goes to read-only warehouses: api.readers (default max: 8, memory: 256MB) and mcp.readers (max: 4, memory: 256MB); the product of the two numbers is the read-warehouse memory ceiling per pod. worker.threads (default 4) is the number of threads used for builds.

Sizing guide

The numbers below are starting estimates, not measurements of your workload. Watch memory and build times for the first few weeks, then adjust.

SmallMediumLarge
Workspacesup to 1010 to 50over 50
Kubernetes nodes3 × 4 vCPU, 16 GB3 × 8 vCPU, 32 GB4 or more × 8 vCPU, 32 GB
worker memory limit7 Gi (default)12 Gi16 Gi or more
api2 replicas (default)enable api.autoscalingenable api.autoscaling, raise maxReplicas
Postgres2 vCPU, 4 GB4 vCPU, 16 GB8 vCPU, 32 GB
Shared volume100 Gi (default)250 Gi500 Gi or more

Object storage capacity depends on how much source data you have; compressed Parquet is usually much smaller than the original. The backup bucket needs room for the retained copies (by default 7 daily, 4 weekly, 6 monthly).

worker always runs as one replica, so as data grows you raise its memory and worker.threads, not its replica count.

Object storage on each cloud

DataxLand speaks the S3 protocol. How to connect each option, set under objectStorage in your values:

StorageobjectStorage.endpointurlStyleCredentialsNotes
AWS S3s3.<region>.amazonaws.comvhostStatic access key of an IAM userIRSA and Pod Identity are not supported
Google Cloud Storagestorage.googleapis.compathHMAC key of a service accountThrough the S3-compatible API; paths stay s3://, not gs://
Azure Blob Storageleave empty, enable s3proxy.enabledpathAn S3 key pair you set for s3proxys3proxy runs in the cluster in front of the Storage Account. Not tested on a real account
On-premMinIO, Ceph RGW, or a local provider's S3-compatible object storageusually pathPer systemMinIO allows per-workspace keys
  • On S3, GCS and s3proxy, set objectStorage.tenantKeys: "off": one key for the whole bucket, each workspace's data under its own prefix.
  • With MinIO, objectStorage.tenantKeys: minio gives each workspace a key that can only read and write its own area. The MinIO admin key then lives only in worker; the chart removes it from api and mcp automatically.
  • An alternative on Azure if you would rather not use s3proxy: run your own MinIO on Azure Disk.
  • Enable versioning on both buckets, and write-once locking (Object Lock, retention, immutability) on the backup bucket.

Network

Pods must be able to reach Postgres, object storage, the AI model server, SMTP, the source databases you plan to connect, and the cluster DNS. In air-gapped mode that is also the complete allow-list; see Air-gapped mode.

Markdown