I Had to Read the Code to Understand the Data Platform

A few years ago I joined a large international wholesaler as a data architect. The company had grown through acquisitions, and each region had its own data warehouse. Several legacy warehouses were still running too. Some fed into regional ones; others had been moved to newer infrastructure with their original pipelines and transformations left in place.

My job was to turn an existing regional Snowflake environment into the global warehouse, simplify the systems landscape, and establish a shared development and deployment process. We would be ingesting and processing many of the existing sources on the target platform. Before I could plan that work, I needed to understand what we had.

I started with the source descriptions in the wiki. I expected an overview of the systems feeding the platform, with enough detail about each integration to begin assessing it.

The list was incomplete. Some sources had a page with nothing on it. Others had headings and empty tables waiting to be filled in. The more substantial entries often listed the tables and jobs on the warehouse side, which was useful, but left me looking elsewhere for the integration details.

Finding a table in Snowflake didn’t tell me whether its data came directly from an operational system or through another warehouse. If it came through another warehouse, I needed to understand what had happened to it before it arrived. That mattered when deciding whether the global platform could replace an existing flow or still depended on it. An inventory of the target tables alone couldn’t answer those questions.

I found more integration details under technical architecture, external systems, and individual jobs. Piecing those pages together still left gaps, so I went into the integration code itself.

Reading the code didn’t give me one pattern I could learn and then follow through the rest of the platform. There were push-based feeds alongside pull-based extractions, custom Python scripts, and different tools for moving data. Understanding one integration gave me limited help with the next.

My impression was that the codebase had accumulated over years, with different developers bringing their own styles and preferences. Architectural guidelines and their enforcement seemed limited. I could see how each implementation might have made sense to the person building it, but I kept having to learn another way of doing things.

I worked through ingestion scripts, the logic that kept historical versions of records, intermediate models, and the layers feeding reports, recording what I found in an inventory.

For this assignment, understanding the existing behavior was necessary even when we intended to replace the implementation. Moving a source to the global warehouse meant assessing which processing we needed to preserve. The different code patterns made that assessment harder: I couldn’t assume that a familiar-looking table had been loaded or transformed in the same way as one I had already reviewed.

We used that inventory to assess what to keep, what to refactor, and what to archive. I consolidated the findings into an entirely new wiki describing the target platform.

So my response to an incomplete wiki was to write another wiki. We needed somewhere to describe the target design and how the existing sources would fit into it.

The inventory and the new wiki served different purposes. The inventory helped us assess the existing platform and make migration decisions. The wiki described the platform we intended to build from those decisions. Both were needed; copying the old source pages into a new structure would have carried the same gaps into the target documentation.

The next pipeline change would still need someone to update the corresponding page. A new wiki gave us a starting point, but keeping it useful would take ongoing work.

Every integration should declare its source, destination, and load behavior in a common format. The extraction methods wouldn’t need to match. Given those declarations, I could build the inventory without first learning my way around every implementation.

That would require architectural conventions and checks as well as documentation tooling. A custom Python extraction could still have a good reason to exist. It would need to expose the same basic information as an integration built with another tool, so the inventory could cover both. Otherwise, the exceptions would gradually become another set of flows someone had to discover by reading code.

Source schemas and pipeline definitions could then supply the technical reference pages. Those pages would need to show when they were refreshed and which deployed version they described. Generating them from an old configuration would leave me with the same problem.

I would still write the parts of the wiki explaining our choices: why an integration was being replaced, or why we had decided to keep it. That was work I expected to do as the architect. Having to reconstruct the source inventory from the code was the part I wanted to avoid repeating.

That experience is part of why we’re building Amfelt around shared metadata and pipeline definitions. If you’re dealing with a similar platform, Join the Waitlist .