Skip to content

Files and storage

Connect S3 to DataxLand

The S3 connector reads CSV, Parquet, Avro or JSONL files in an Amazon S3 bucket (or an S3-compatible store with a public address). You declare each stream with a path pattern, for example every orders/*.csv file, and each stream becomes a table in the workspace warehouse. It suits sales software or ERPs that export files to S3 on a schedule - Si reads a masked data sample to propose a mapping to the standard model, and you get net revenue, cost of goods sold and contribution margin just as with an Excel upload, but refreshed automatically on a schedule.

Free plan with no time limit · No card required · Sample workspace ready in minutes

What you can do with S3

  • Refresh your profit and loss dashboard automatically from the daily exports of your sales software.
  • Combine order files in S3 with marketplace and ad data.
  • Read Parquet files exported by internal systems without opening up a database.

What data you get

Each stream lands as its own table in the workspace warehouse, as is; personal-data columns are hashed on load.

Stream (table)Contents
(stream you name)Each stream has a name, a path pattern (glob) and a format; all files matching the pattern are combined into one table.
CSVOptions for delimiter, character encoding and header row.
Parquet, AvroRead using the schema stored in the file.
JSONLOne JSON record per line; nested fields become JSON strings.

Connection details

Sign-in methodConnection details (API key, account), encrypted at rest
Sync modesFull (reloaded each run) · Incremental (new or changed rows only)
Fields to fill inBucket, Authentication, Streams
DeveloperAirbyte (certified), run inside DataxLand
LicenseELv2

Passwords and keys are encrypted at rest. Personal-data columns are hashed on load.

Metrics computed from this source

Real definitions from the analytics packs; dashboards and Si share them.

MetricDefinitionUnit
Net revenueLine value after line discounts and allocated order vouchers, minus refunds to buyers. Refunds are deducted only here.Currency
Cost of goods soldQuantity times unit cost on the order date (half-open validity interval). SKUs without a cost count as 0; see COGS coverage. Returned lines do not reverse their COGS.Currency
Contribution marginNet revenue minus COGS, marketplace fees, payment fees, shipping and return costs (before advertising).= Net revenue - Cost of goods sold - Marketplace fees - Payment fees - Seller-paid shipping cost - Return costCurrency
Valid ordersOrders that were not cancelled (standard status other than cancelled).Count

Connect S3 in 4 steps

  1. 1

    Create read access

    Create an IAM user (or role) that can only list and read the bucket you need, and get the access key ID and secret access key.

  2. 2

    Add the S3 source

    In the workspace choose Add source, find S3, and enter the Bucket, the IAM User (or IAM Role) authentication, and the AWS region if needed.

  3. 3

    Define streams

    Add each stream with a name, a path pattern such as orders/*.csv and the file format, then click Test connection.

  4. 4

    Review and build

    Si proposes a mapping for each stream to the standard model; you approve, then build.

Limits and caveats

  • The metrics on this page are examples for files that contain orders and cost prices; the actual metrics depend on the files you provide.
  • Loads are incremental by file modification time, so each refresh reads only new or recently modified files. If new files repeat old rows, you need a primary key to merge them without duplicates.
  • S3-compatible stores (MinIO, R2...) need a public address; DataxLand blocks private network addresses.
  • Requires the Starter plan or above. An Airbyte connector (ELv2) run inside DataxLand.

Frequently asked questions

Can it read Excel files in S3?

This page covers CSV, Parquet, Avro and JSONL only; for .xlsx files, use DataxLand's Excel source or export to CSV.

Are access keys stored safely?

Access keys are encrypted when saved and never shown again; use a user that can only read that one bucket.

What about many files with the same structure?

Declare one stream with a path pattern that matches all of them; they are combined into one table.

Dashboard templates that use this source

All data sources →

Connect S3 and get verified dashboards

Free plan with no time limit · No card required · Sample workspace ready in minutes