Reference Terraform (AWS, GCP, Azure)
Sample Terraform modules that build the network, a managed Kubernetes cluster, Postgres, object storage, a shared volume and a secret store, then install the DataxLand Helm chart on AWS, GCP or Azure; a reference template, not yet applied to a real account.
These modules are reference templates. They pass terraform fmt -check and terraform validate, but have not been run
with plan or apply against a real cloud account. SKU names, regional quotas, least-privilege IAM permissions and timeouts
may need adjusting. Your infrastructure team or implementation partner copies the folder for your cloud into your own
infrastructure repo and adapts it.
DataxLand provides three folders: aws/, gcp/ and azure/. Each one reads on its own, with no shared modules. The last step
of each folder installs the Helm chart with helm_release.
What each module builds
| AWS | GCP | Azure | |
|---|---|---|---|
| Network | VPC, public and private subnets per AZ, NAT, S3 gateway endpoint | VPC, subnet with secondary ranges for pods and services, Cloud NAT, Private Service Access | VNet, subnets for AKS, Postgres and private endpoints, private DNS |
| Kubernetes | EKS, 3 m6i.xlarge nodes (4 vCPU, 16 GB) | Regional GKE, private nodes, Dataplane V2, 3 e2-standard-4 nodes | AKS, Azure CNI Overlay and Cilium, 3 Standard_D4s_v5 nodes |
| Postgres 16 | RDS Multi-AZ, VPC only, TLS required | Cloud SQL REGIONAL, private IP only, encrypted connections only | Flexible Server ZoneRedundant, VNet integration |
dx_lake database | A psql Job in the cluster | Created by Terraform | Created by Terraform |
| Shared volume | EFS, class efs-rwx | Filestore, class filestore-rwx (1 TiB minimum) | Azure Files, built-in class azurefile-csi |
| Object storage | S3, vhost | GCS through the S3-compatible API, path | Blob through in-cluster s3proxy, path |
| Object storage keys | IAM user and access key, scoped to the two buckets | Service account and HMAC key, scoped to the two buckets | Random S3 key pair for s3proxy |
| Write-once lock for backups (optional) | S3 Object Lock | Retention policy | Immutability policy |
| Secret store | Secrets Manager | Secret Manager | Key Vault |
| Ingress | Not installed | Not installed | App Routing add-on (can be disabled) |
Each module generates the application's secrets, stores them in the cloud secret store and copies them into the Kubernetes Secret the chart reads. Object storage keys are static, one key for the whole bucket: the application does not support IRSA, Workload Identity or Managed Identity.
Running it
The general flow; details are in each folder's README.md:
cd aws # or gcp, azure
cp terraform.tfvars.example terraform.tfvars # edit for your account
terraform init
terraform apply -target=<cluster resource> # first run: build the cluster first
kubectl create namespace dataxland
kubectl -n dataxland create secret generic dataxland-license --from-file=license.json
terraform import kubernetes_namespace_v1.this dataxland
terraform applyThe cluster has to exist first because the kubernetes and helm providers need it. The operator creates the license Secret
outside Terraform, before the final apply.
AWS
Requires the AWS CLI and permission to create the resources above.
terraform apply -target=aws_eks_node_group.default -target=aws_eks_addon.efs_csi- No ingress controller is installed: use the AWS Load Balancer Controller or one you already run.
- The node count is fixed; for autoscaling, install Cluster Autoscaler or Karpenter.
- Deletion protection is on for RDS and the cluster; RDS takes a final snapshot on deletion.
GCP
Requires gcloud auth application-default login.
terraform apply -target=google_container_node_pool.default
gcloud container clusters get-credentials <prefix>-gke --region <region>- The Filestore
standardtier needs at least 1 TiB per PVC. For something smaller: self-managed NFS, or a different tier. - Filestore root directory permissions for uid 10001 have not been verified.
Azure
Requires az login and the Owner role (or Contributor plus User Access Administrator) on the subscription.
terraform apply -target=azurerm_kubernetes_cluster.this
az aks get-credentials --resource-group <prefix>-rg --name <prefix>-aks- Object storage goes through s3proxy, which has not been tested with the DataxLand warehouse on a real account.
azurefile-csiis SMB-based Azure Files; build performance on SMB has not been measured. For better POSIX behaviour, create an Azure Files NFS class.- The first
applymay fail with a 403 when writing to Key Vault because the role assignment has not taken effect yet: wait a few minutes and run it again.
After the run
- Terraform state contains every secret: use an encrypted remote backend with restricted access.
- Copy
DX_BACKUP_ENCRYPTION_KEY(outputbackup_encryption_key) to offline storage. Lose the key and you lose every backup. - In production, switch to External Secrets Operator reading the secret store directly instead of the Secret Terraform copies;
see Install with Helm. Extra chart values go in the
helm_extra_valuesvariable. - The modules don't install an ingress controller, cert-manager, External Secrets Operator or a monitoring stack.
Related
Install with Helm
Install DataxLand on Kubernetes with the Helm chart, including the secrets Secret, the license Secret, minimal production values, post-install checks, a trial setup with in-cluster Postgres and MinIO, External Secrets, NetworkPolicy and ServiceMonitor.
Air-gapped mode
Run DataxLand on a network with no Internet access - the allow-list of hosts, the required NetworkPolicy, an in-house AI model, internal SMTP, features that turn off, data sources that still work and how to copy images into an internal registry.