# Stop asking AI to guess. Make your pipelines deterministic. An LLM changed my build check until a failing Hugo build looked successful. That experience shaped how I think about AI-generated pipeline code and the checks it must pass. I was days into a Hugo site refactor I had handed to an LLM. More than five hundred pages, custom shortcodes, complex components, and a live-editing CMS layer all had to keep working. My local build kept failing, while the model reported success. Eventually I read the build script line by line and found that it had changed the success condition. The original check used Hugo's exit code. The replacement accepted either a successful exit code or the presence of any file in the output folder. Hugo could report hundreds of errors and still leave a file behind. The check passed, and the model treated that as a successful build. In mid-2025 I had started using LLMs for workshop agendas and interview questions. I was skeptical about using them for software, but the early results were useful enough that I tried coding tasks too. Over the following months I gave them larger assignments. The Hugo refactor was the first substantial one I trusted the model to carry through, and I spent much of those two weeks untangling the result. I came away with a more cautious workflow: smaller changes, a plan I could review, and checks whose meaning I understood. The build-script change mattered more than any individual template error. I had allowed the tool doing the work to weaken the test I relied on to assess it. --- That experience also affects how I think about data pipelines. An LLM can help write the extraction code or suggest a key. Once a pipeline is running, I need to know exactly which rules it applies and what would cause a run to fail. For a full load, that includes which records to read and whether to replace the destination. An incremental load needs a cursor and a rule for records arriving late. Change data capture reads changes from a transaction log and needs to preserve its position through a restart. Each pattern has failure cases. Choosing a familiar pattern does not make those cases disappear, but it lets us specify and test the behavior. By deterministic, I mean that the same captured input, starting state, and configuration produce the same logical result. Querying a live source again tomorrow does not meet that condition: the source may have changed. A pipeline also needs an explicit retry policy. An append operation can be deterministic and still duplicate every record when repeated. Avoiding that duplication is an idempotency requirement, which has to be designed separately. These distinctions matter when a number changes. I want to be able to inspect the source version and the rules used for each run, rather than guess whether a model made a different choice during ingestion. --- In the Hugo refactor, the observable failure was straightforward: a check that should have rejected the output had been changed to accept it. I cannot infer the model's reasoning from that edit. I can say that its report was no longer supported by the check I thought I was running. The equivalent in a pipeline might be an invalid date quietly becoming null because someone relaxed a conversion rule to get a load through. That can be an acceptable policy if the affected records are recorded and the team approved it. It is a data-quality failure if the expected behavior was to reject the batch and nobody knows the rule changed. This is why I want generated code and changes to its checks reviewed together. A successful run tells me little if the definition of success changed in the same patch. For a conversion, I would want examples of valid and invalid input, with expected outcomes fixed before accepting the implementation. The code can change; those expectations need a reason to change. --- There is still plenty of useful work for an LLM. Given a source schema and a statistical profile, it can propose that a table contains transactions or that a column is a likely customer identifier. A data engineer can inspect that recommendation, correct it, and save the decision in configuration. The pipeline then uses the approved value on every run, without asking the model to classify the table again. The same applies to code generation. A model can draft the SQL for a history table. Once reviewed, that SQL should execute according to the selected key and history rules. Using an LLM to write it does not require using an LLM to decide how each incoming record behaves. We can keep the generated artifact and the configuration that produced it, so a later change is something we can compare and review. Some proposed changes deserve more scrutiny than others. A draft column description is easy to correct before publishing it. A proposed customer-key mapping can affect joins and every metric built on them. Calling both recommendations does not make their consequences equal. The review should account for where the decision will be used. --- I still use AI for coding. What changed after that refactor is how much trust I place in its account of the result. For a pipeline, I want that account backed by rules and checks I can inspect, including evidence that a failed load would actually be reported as failed. We're building Amfelt so a model's recommendation becomes a stored setting: when the advisor is switched on it can apply the answers it is most sure of, and every later run uses the stored value without asking it again: [join the waitlist](/waitlist/).