Self-hosting (Enterprise)

Reference Terraform (AWS, GCP, Azure)

Sample Terraform modules that build the network, a managed Kubernetes cluster, Postgres, object storage, a shared volume and a secret store, then install the DataxLand Helm chart on AWS, GCP or Azure; a reference template, not yet applied to a real account.

These modules are reference templates. They pass terraform fmt -check and terraform validate, but have not been run with plan or apply against a real cloud account. SKU names, regional quotas, least-privilege IAM permissions and timeouts may need adjusting. Your infrastructure team or implementation partner copies the folder for your cloud into your own infrastructure repo and adapts it.

DataxLand provides three folders: aws/, gcp/ and azure/. Each one reads on its own, with no shared modules. The last step of each folder installs the Helm chart with helm_release.

What each module builds

AWSGCPAzure
NetworkVPC, public and private subnets per AZ, NAT, S3 gateway endpointVPC, subnet with secondary ranges for pods and services, Cloud NAT, Private Service AccessVNet, subnets for AKS, Postgres and private endpoints, private DNS
KubernetesEKS, 3 m6i.xlarge nodes (4 vCPU, 16 GB)Regional GKE, private nodes, Dataplane V2, 3 e2-standard-4 nodesAKS, Azure CNI Overlay and Cilium, 3 Standard_D4s_v5 nodes
Postgres 16RDS Multi-AZ, VPC only, TLS requiredCloud SQL REGIONAL, private IP only, encrypted connections onlyFlexible Server ZoneRedundant, VNet integration
dx_lake databaseA psql Job in the clusterCreated by TerraformCreated by Terraform
Shared volumeEFS, class efs-rwxFilestore, class filestore-rwx (1 TiB minimum)Azure Files, built-in class azurefile-csi
Object storageS3, vhostGCS through the S3-compatible API, pathBlob through in-cluster s3proxy, path
Object storage keysIAM user and access key, scoped to the two bucketsService account and HMAC key, scoped to the two bucketsRandom S3 key pair for s3proxy
Write-once lock for backups (optional)S3 Object LockRetention policyImmutability policy
Secret storeSecrets ManagerSecret ManagerKey Vault
IngressNot installedNot installedApp Routing add-on (can be disabled)

Each module generates the application's secrets, stores them in the cloud secret store and copies them into the Kubernetes Secret the chart reads. Object storage keys are static, one key for the whole bucket: the application does not support IRSA, Workload Identity or Managed Identity.

Running it

The general flow; details are in each folder's README.md:

cd aws            # or gcp, azure
cp terraform.tfvars.example terraform.tfvars   # edit for your account
terraform init
terraform apply -target=<cluster resource>      # first run: build the cluster first
kubectl create namespace dataxland
kubectl -n dataxland create secret generic dataxland-license --from-file=license.json
terraform import kubernetes_namespace_v1.this dataxland
terraform apply

The cluster has to exist first because the kubernetes and helm providers need it. The operator creates the license Secret outside Terraform, before the final apply.

AWS

Requires the AWS CLI and permission to create the resources above.

terraform apply -target=aws_eks_node_group.default -target=aws_eks_addon.efs_csi
  • No ingress controller is installed: use the AWS Load Balancer Controller or one you already run.
  • The node count is fixed; for autoscaling, install Cluster Autoscaler or Karpenter.
  • Deletion protection is on for RDS and the cluster; RDS takes a final snapshot on deletion.

GCP

Requires gcloud auth application-default login.

terraform apply -target=google_container_node_pool.default
gcloud container clusters get-credentials <prefix>-gke --region <region>
  • The Filestore standard tier needs at least 1 TiB per PVC. For something smaller: self-managed NFS, or a different tier.
  • Filestore root directory permissions for uid 10001 have not been verified.

Azure

Requires az login and the Owner role (or Contributor plus User Access Administrator) on the subscription.

terraform apply -target=azurerm_kubernetes_cluster.this
az aks get-credentials --resource-group <prefix>-rg --name <prefix>-aks
  • Object storage goes through s3proxy, which has not been tested with the DataxLand warehouse on a real account.
  • azurefile-csi is SMB-based Azure Files; build performance on SMB has not been measured. For better POSIX behaviour, create an Azure Files NFS class.
  • The first apply may fail with a 403 when writing to Key Vault because the role assignment has not taken effect yet: wait a few minutes and run it again.

After the run

  • Terraform state contains every secret: use an encrypted remote backend with restricted access.
  • Copy DX_BACKUP_ENCRYPTION_KEY (output backup_encryption_key) to offline storage. Lose the key and you lose every backup.
  • In production, switch to External Secrets Operator reading the secret store directly instead of the Secret Terraform copies; see Install with Helm. Extra chart values go in the helm_extra_values variable.
  • The modules don't install an ingress controller, cert-manager, External Secrets Operator or a monitoring stack.

Markdown