File

Available v1.10.5 Released October 31, 2026

Overview  

The File connector extracts data from CSV, Parquet, and flat JSONL files as a source. It reads a single file per connection, derives the logical object name from the file name, and hands the records to the agent as Arrow record batches — Parquet schemas are read from the file’s own metadata, while CSV and JSONL schemas are inferred from the data.

Typical uses are one-off historical loads, reference data maintained as files, and exports from systems that have no direct connector yet. The file must be reachable from the Amfelt agent’s file system (a local path or a mounted share).

Capabilities  

Prerequisites  

Before connecting a file source:

• File access from the Amfelt agent: the file must exist on the agent’s file system (a local path or a mounted network share) and be readable by the agent process.
• A supported format: CSV (with a header row), Parquet, or flat JSONL: one JSON object per line, with scalar values only. Nested objects and arrays are not read yet. The format is resolved from the file extension, or set explicitly with a ?format= query parameter on the connection path.
• Stable encoding: CSV and JSON files are read as UTF-8. A UTF-8 byte-order mark at the start of the file, as Excel’s “CSV UTF-8” export writes, is skipped.

Connect  

1. In Amfelt, add a file source and select the File connector.
2. Provide the connection properties (see Configuration reference): the file’s absolute path on the agent’s machine and its format (csv, parquet, or json for JSONL).
3. Validate the connection and choose Discover schema — the logical object takes its name from the file name (for example /data/customers.csv becomes customers).

A connection resolves to an ODCS server descriptor like:

servers:
  - server: files-drop
    type: local
    path: /data/customers.csv
    format: csv

Schema Handling  

• Parquet files carry their own typed schema; it is used as-is.
• CSV and JSON schemas are inferred from every row of the file when metadata is ingested. On a typed target such as PostgreSQL, deploying creates the columns with those types; DuckDB stores every column as text (VARCHAR). A column whose values do not all share one type is inferred as text and loads every value exactly as written; cast it downstream or switch the export to Parquet for fully typed extraction.
• Ingestion never infers a column as a number when any of its values would change as one: a leading zero (007, -01), an integer beyond the 64-bit range, or a decimal with more digits than a double holds. Such a column, typically zip codes, account numbers or long identifiers, is inferred as text and keeps its exact text. A difference in form only, such as 1.10 for 1.1 or 1e3 for 1000, still loads as a number.
• On a typed target such as PostgreSQL, a deployed column keeps its type: deploying again under the same schema prefix does not change the type of a column that already exists. If a later version of the file holds values that a deployed column’s type cannot hold exactly, such as a zip code with a leading zero in a column deployed as a number, re-ingest the file’s metadata and deploy it under a new schema prefix before you load that file.
• Because ingestion infers a CSV or JSON schema from every row, it reads the whole file on the agent’s machine, so its time grows with the file’s size, and a file of a few gigabytes can take minutes. The agent stops metadata ingestion that runs longer than its import timeout, 2 minutes by default, and the ingestion fails. To ingest a larger file, set the agent’s AMFELT_IMPORT_TIMEOUT environment variable to a longer duration, such as 10m or 1h, and restart the agent. A Parquet file carries its own typed schema, so ingestion does not infer one from its rows.
• Validating the connection reads only the header and the first 4,096 rows, so it stays fast on a large file; it reports the number of columns, not their types.
• Each run reads the file’s schema again when it starts, then reads the rows. If the file changes in between, so that a value no longer fits the type the run read, the run stops at that row and names the column and the row. That value is not loaded, and staging keeps the rows loaded before it. From agent v4.18.3 on, the run then skips the history, bridge and business layers, so they keep what the last successful run loaded. An earlier agent still runs the history layer over the partial staging.

Sync Modes  

Configuration Reference  

Troubleshooting  

Changelog  

Release history for the File connector, newest first.

Licensing  

Third-party open-source packages bundled by this connector, with their licenses. Attribution only: informational, not an evaluation of the connector.