# File

Connect File with Amfelt: setup, required privileges, configuration reference, and type mapping.

Available v1.10.5 Released October 31, 2026

## Overview

The File connector extracts data from **CSV**, **Parquet**, and flat **JSONL** files as a
**source**. It reads a single file per connection, derives the logical object name from the
file name, and hands the records to the agent as Arrow record batches — Parquet schemas are
read from the file's own metadata, while CSV and JSONL schemas are inferred from the data.

Typical uses are one-off historical loads, reference data maintained as files, and
exports from systems that have no direct connector yet. The file must be reachable from
the Amfelt agent's file system (a local path or a mounted share).

## Capabilities

| Feature | Source | Target |
|-|-|-|
| Full load | Yes | — |
| Incremental load | — | — |
| Change Data Capture (CDC) | — | — |
| Schema drift | Yes | — |

## Prerequisites

Before connecting a file source:

- **File access** from the Amfelt agent: the file must exist on the agent's file system
  (a local path or a mounted network share) and be readable by the agent process.
- **A supported format:** CSV (with a header row), Parquet, or flat JSONL: one JSON object
  per line, with scalar values only. Nested objects and arrays are not read yet. The format is
  resolved from the file extension, or set explicitly with a `?format=` query parameter
  on the connection path.
- **Stable encoding:** CSV and JSON files are read as UTF-8. A UTF-8 byte-order mark at the
  start of the file, as Excel's "CSV UTF-8" export writes, is skipped.

## Connect

1. In Amfelt, add a **file source** and select the File connector.
2. Provide the connection properties (see *Configuration reference*): the file's
   absolute `path` on the agent's machine and its `format` (`csv`, `parquet`, or `json` for JSONL).
3. Validate the connection and choose **Discover schema** — the logical object takes its
   name from the file name (for example `/data/customers.csv` becomes `customers`).

A connection resolves to an ODCS server descriptor like:

```yaml
servers:
  - server: files-drop
    type: local
    path: /data/customers.csv
    format: csv
```

### Schema handling

- **Parquet** files carry their own typed schema; it is used as-is.
- **CSV** and **JSON** schemas are inferred from every row of the file when metadata is
  ingested. On a typed target such as PostgreSQL, deploying creates the columns with those
  types; DuckDB stores every column as text (`VARCHAR`). A column whose values do not all
  share one type is inferred as text and loads every value exactly as written; cast it
  downstream or switch the export to Parquet for fully typed extraction.
- Ingestion never infers a column as a number when any of its values would change as
  one: a leading zero (`007`, `-01`), an integer beyond the 64-bit range, or a decimal
  with more digits than a double holds. Such a column, typically zip codes, account
  numbers or long identifiers, is inferred as text and keeps its exact text. A difference
  in form only, such as `1.10` for `1.1` or `1e3` for `1000`, still loads as a number.
- On a typed target such as PostgreSQL, a deployed column keeps its type: deploying again
  under the same schema prefix does not change the type of a column that already exists.
  If a later version of the file holds values that a deployed column's type cannot hold
  exactly, such as a zip code with a leading zero in a column deployed as a number,
  re-ingest the file's metadata and deploy it under a new schema prefix before you load
  that file.
- Because ingestion infers a CSV or JSON schema from every row, it reads the whole file on
  the agent's machine, so its time grows with the file's size, and a file of a few
  gigabytes can take minutes. The agent stops metadata ingestion that runs longer than its
  import timeout, 2 minutes by default, and the ingestion fails. To ingest a larger file,
  set the agent's `AMFELT_IMPORT_TIMEOUT` environment variable to a longer duration, such
  as `10m` or `1h`, and restart the agent. A Parquet file carries its own typed schema, so
  ingestion does not infer one from its rows.
- Validating the connection reads only the header and the first 4,096 rows, so it stays
  fast on a large file; it reports the number of columns, not their types.
- Each run reads the file's schema again when it starts, then reads the rows. If the file
  changes in between, so that a value no longer fits the type the run read, the run stops
  at that row and names the column and the row. That value is not loaded, and staging
  keeps the rows loaded before it. From agent v4.18.3 on, the run then skips the history,
  bridge and business layers, so they keep what the last successful run loaded. An
  earlier agent still runs the history layer over the partial staging.

## Sync modes

| Mode | What it does |
|-|-|
| Full load | Reloads all selected data on every run. |

## Configuration reference

| Property | Type | Required | Description |
|-|-|-|-|
| `format` | string | Yes | The format of the file(s) |
| `path` | string | Yes | The relative or absolute path to the data file(s). |

## Troubleshooting

### File not found

The path is resolved on the Amfelt agent's file system, not the control plane's. Verify the file exists on the machine running the agent and that any network share is mounted before the agent starts.

### The path must be absolute

The connection test refuses a relative path, because it would resolve against the agent's working directory rather than a fixed location. Give the file's full path on the agent's machine, such as `/data/customers.csv`.

### Permission denied

The agent process needs read permission on the file and traverse permission on every directory in the path.

### Unknown or wrong format

The format is resolved from the file extension. For files with a non-standard extension, set it explicitly with a `?format=csv`, `?format=parquet`, or `?format=json` (for JSONL) query parameter on the connection path.

### Wrong inferred column types

CSV and JSON schemas are inferred from every row of the file when metadata is ingested. A column whose values do not share one type, or that holds a value a number would change (a leading zero, an integer beyond 64 bits, a decimal with more digits than a double holds), is inferred as text and loads exactly as written. Cast it downstream, or switch the export to Parquet, which carries its own typed schema.

### Metadata ingestion of a large file times out

CSV and JSON schemas are inferred from every row, so ingesting a CSV or JSON file reads the whole file on the agent's machine, and a large file takes longer. The agent stops metadata ingestion that runs longer than its import timeout, 2 minutes by default, and the ingestion fails. Set the agent's `AMFELT_IMPORT_TIMEOUT` environment variable to a longer duration, such as `10m` or `1h`, restart the agent, and run the ingestion again. A Parquet export avoids the inference, because it carries its own typed schema.

### A later file holds values its deployed column types cannot hold

On a typed target such as PostgreSQL, a deployed column keeps the type it was inferred with, and deploying again under the same schema prefix does not change an existing column. If the file now holds values that type cannot hold exactly, for example zip codes with a leading zero in a column deployed as a number, re-ingest the file's metadata and deploy it under a new schema prefix before you load that file. DuckDB stores every column as text, so this does not arise there.

### The file has changed since its schema was read

A run reads the file's schema when it starts, then reads the rows. If the file is rewritten in between so that a value no longer fits the type the run read, the run stops at that row and names the column and the row. That value is not loaded, and staging keeps the rows loaded before it. From agent v4.18.3 on, history and the later layers keep what the last successful run loaded. Let whatever writes the file finish before the run starts. If the new values no longer fit the deployed column types, follow the previous entry before running again; otherwise run again.

## Changelog

Release history for the File connector, newest first.

### v1.10.5 (October 31, 2026)

**Features**

- Keep the state schema and the file ledger

### v1.10.0 (October 16, 2026)

**Features**

- Publish plugin releases on the connectors repo

### v1.9.2 (October 13, 2026)

**Bug Fixes**

- Move connector module paths to github.com/amfeltlabs

### v1.9.1 (October 11, 2026)

**Features**

- Open PostgreSQL with pgx; core v3.41.2, plugin-sdk v5.17.4

### v1.9.0 (October 10, 2026)

**Features**

- Write the amf_ metadata columns and name Amfelt in what connectors ship

### v1.8.1 (October 9, 2026)

**Bug Fixes**

- Bump golang.org/x/net to v0.60.0 (GO-2026-6617)

### v1.8.0 (October 7, 2026)

**Features**

- Ship third-party notices in postgres and file archives

### v1.7.5 (October 5, 2026)

**Bug Fixes**

- Say that ingesting a CSV or JSON file reads the whole file

### v1.7.4 (October 5, 2026)

**Bug Fixes**

- Skip a leading UTF-8 byte-order mark in CSV and JSONL files

### v1.7.3 (October 5, 2026)

**Bug Fixes**

- Never load a CSV or JSON value altered by type inference

### v1.7.0 (October 4, 2026)

**Features**

- Record third-party marks per connector

### v1.6.2 (October 4, 2026)

**Features**

- Compare scope in the postgres and duckdb templates

### v1.6.0 (October 3, 2026)

**Features**

- answer the source connection test on plugin-sdk v5.12.2

### v1.5.10 (October 3, 2026)

**Features**

- Postgres and duckdb sources answer the source connection test

**Bug Fixes**

- Read a header-only csv as an empty table instead of panicking

### v1.5.9 (October 1, 2026)

**Bug Fixes**

- Require plugin-sdk v5.9.3 so a read-only-default role is not reported ok

### v1.5.3 (August 27, 2026)

**Features**

- Pass the declared override surface to the renderer (ADR-117)
- Declare the override surface and govern the declared tier (ADR-117)

### v1.5.1 (August 9, 2026)

**Bug Fixes**

- Select the modules that consume a changed module

### v1.5.0 (August 5, 2026)

**Features**

- Bind the readers to their catalog pages

### v1.4.1 (August 3, 2026)

**Features**

- Add the source connector

### v1.4.0 (August 2, 2026)

**Breaking Changes**

- Newconnector generator, docbundle gate, file docbundle

### v1.3.3 (August 2, 2026)

**Breaking Changes**

- Adopt plugin-sdk /v5 — re-pin, ctx seams, quoting, BaseRenderer, DTOs

### v1.3.0 (July 16, 2026)

**Features**

- Add JSONL file-format reader (inference + dispatch)

### v1.2.0 (July 16, 2026)

**Features**

- Parquet reader and multi-format file dispatch

### v1.1.0 (July 16, 2026)

**Features**

- Per-property target DDL typer (durable B4b, CSV renders distinct columns)

### v1.0.0 (July 16, 2026)

**Features**

- Source-file CSV connector (proven CSV to postgres)

## Licensing

Third-party open-source packages bundled by this connector, with their licenses. Attribution only: informational, not an evaluation of the connector.

### Open-source dependency licenses (47 packages)

| Package | License |
|-|-|
| [github.com/Masterminds/semver](https://pkg.go.dev/github.com/Masterminds/semver@v1.5.0?tab=licenses) | MIT |
| [github.com/andybalholm/brotli](https://pkg.go.dev/github.com/andybalholm/brotli@v1.2.3?tab=licenses) | BSD-3-Clause AND MIT |
| [github.com/apache/arrow-go/v18](https://pkg.go.dev/github.com/apache/arrow-go/v18@v18.8.0?tab=licenses) | Apache-2.0 AND BSD-3-Clause |
| [github.com/apache/thrift](https://pkg.go.dev/github.com/apache/thrift@v0.24.0?tab=licenses) | Apache-2.0 AND BSD-3-Clause AND GNU-All-permissive-Copying-License |
| [github.com/bmatcuk/doublestar/v4](https://pkg.go.dev/github.com/bmatcuk/doublestar/v4@v4.10.3?tab=licenses) | MIT |
| [github.com/cespare/xxhash/v2](https://pkg.go.dev/github.com/cespare/xxhash/v2@v2.3.0?tab=licenses) | MIT |
| [github.com/fsnotify/fsnotify](https://pkg.go.dev/github.com/fsnotify/fsnotify@v1.10.1?tab=licenses) | BSD-3-Clause |
| [github.com/gabriel-vasile/mimetype](https://pkg.go.dev/github.com/gabriel-vasile/mimetype@v1.4.15?tab=licenses) | MIT |
| [github.com/go-playground/locales](https://pkg.go.dev/github.com/go-playground/locales@v0.14.1?tab=licenses) | MIT |
| [github.com/go-playground/universal-translator](https://pkg.go.dev/github.com/go-playground/universal-translator@v0.18.1?tab=licenses) | MIT |
| [github.com/go-playground/validator/v10](https://pkg.go.dev/github.com/go-playground/validator/v10@v10.30.5?tab=licenses) | MIT |
| [github.com/go-viper/mapstructure/v2](https://pkg.go.dev/github.com/go-viper/mapstructure/v2@v2.5.0?tab=licenses) | MIT |
| [github.com/goccy/go-json](https://pkg.go.dev/github.com/goccy/go-json@v0.10.6?tab=licenses) | MIT |
| [github.com/goccy/go-yaml](https://pkg.go.dev/github.com/goccy/go-yaml@v1.19.2?tab=licenses) | MIT |
| [github.com/google/flatbuffers](https://pkg.go.dev/github.com/google/flatbuffers@v25.12.19+incompatible?tab=licenses) | Apache-2.0 |
| [github.com/google/uuid](https://pkg.go.dev/github.com/google/uuid@v1.6.0?tab=licenses) | BSD-3-Clause |
| [github.com/hashicorp/errwrap](https://pkg.go.dev/github.com/hashicorp/errwrap@v1.1.0?tab=licenses) | MPL-2.0 |
| [github.com/hashicorp/go-multierror](https://pkg.go.dev/github.com/hashicorp/go-multierror@v1.1.1?tab=licenses) | MPL-2.0 |
| [github.com/iancoleman/strcase](https://pkg.go.dev/github.com/iancoleman/strcase@v0.3.0?tab=licenses) | MIT |
| [github.com/jinzhu/copier](https://pkg.go.dev/github.com/jinzhu/copier@v0.4.0?tab=licenses) | MIT |
| [github.com/klauspost/compress](https://pkg.go.dev/github.com/klauspost/compress@v1.19.2?tab=licenses) | Apache-2.0 AND BSD-3-Clause AND MIT |
| [github.com/klauspost/cpuid/v2](https://pkg.go.dev/github.com/klauspost/cpuid/v2@v2.4.0?tab=licenses) | MIT |
| [github.com/leodido/go-urn](https://pkg.go.dev/github.com/leodido/go-urn@v1.5.0?tab=licenses) | MIT |
| [github.com/lmittmann/tint](https://pkg.go.dev/github.com/lmittmann/tint@v1.2.1?tab=licenses) | MIT |
| [github.com/mattn/go-isatty](https://pkg.go.dev/github.com/mattn/go-isatty@v0.0.24?tab=licenses) | MIT |
| [github.com/mitchellh/go-wordwrap](https://pkg.go.dev/github.com/mitchellh/go-wordwrap@v1.0.1?tab=licenses) | MIT |
| [github.com/pelletier/go-toml/v2](https://pkg.go.dev/github.com/pelletier/go-toml/v2@v2.3.1?tab=licenses) | MIT |
| [github.com/pierrec/lz4/v4](https://pkg.go.dev/github.com/pierrec/lz4/v4@v4.1.29?tab=licenses) | BSD-3-Clause |
| [github.com/sagikazarmark/locafero](https://pkg.go.dev/github.com/sagikazarmark/locafero@v0.12.0?tab=licenses) | MIT |
| [github.com/santhosh-tekuri/jsonschema/v6](https://pkg.go.dev/github.com/santhosh-tekuri/jsonschema/v6@v6.0.3?tab=licenses) | Apache-2.0 |
| [github.com/spf13/afero](https://pkg.go.dev/github.com/spf13/afero@v1.15.0?tab=licenses) | Apache-2.0 |
| [github.com/spf13/cast](https://pkg.go.dev/github.com/spf13/cast@v1.10.0?tab=licenses) | MIT |
| [github.com/spf13/cobra](https://pkg.go.dev/github.com/spf13/cobra@v1.10.2?tab=licenses) | Apache-2.0 |
| [github.com/spf13/pflag](https://pkg.go.dev/github.com/spf13/pflag@v1.0.10?tab=licenses) | BSD-3-Clause |
| [github.com/spf13/viper](https://pkg.go.dev/github.com/spf13/viper@v1.21.0?tab=licenses) | MIT |
| [github.com/subosito/gotenv](https://pkg.go.dev/github.com/subosito/gotenv@v1.6.0?tab=licenses) | MIT |
| [github.com/zeebo/xxh3](https://pkg.go.dev/github.com/zeebo/xxh3@v1.1.0?tab=licenses) | BSD-2-Clause |
| [go.yaml.in/yaml/v3](https://pkg.go.dev/go.yaml.in/yaml/v3@v3.0.5?tab=licenses) | MIT |
| [golang.org/x/crypto](https://pkg.go.dev/golang.org/x/crypto@v0.57.0?tab=licenses) | BSD-3-Clause |
| [golang.org/x/exp](https://pkg.go.dev/golang.org/x/exp@v0.0.0-20260410095643-746e56fc9e2f?tab=licenses) | BSD-3-Clause |
| [golang.org/x/net](https://pkg.go.dev/golang.org/x/net@v0.60.0?tab=licenses) | BSD-3-Clause |
| [golang.org/x/sync](https://pkg.go.dev/golang.org/x/sync@v0.24.0?tab=licenses) | BSD-3-Clause |
| [golang.org/x/sys](https://pkg.go.dev/golang.org/x/sys@v0.48.0?tab=licenses) | BSD-3-Clause |
| [golang.org/x/text](https://pkg.go.dev/golang.org/x/text@v0.43.0?tab=licenses) | BSD-3-Clause |
| [google.golang.org/genproto/googleapis/rpc](https://pkg.go.dev/google.golang.org/genproto/googleapis/rpc@v0.0.0-20260928230214-8a89bd6388cc?tab=licenses) | Apache-2.0 |
| [google.golang.org/grpc](https://pkg.go.dev/google.golang.org/grpc@v1.85.0?tab=licenses) | Apache-2.0 |
| [google.golang.org/protobuf](https://pkg.go.dev/google.golang.org/protobuf@v1.36.12?tab=licenses) | BSD-3-Clause |