With the advancements of LLM-based tooling, the responsibilities (and maybe even the title!) of data engineers are changing.
Origin
Before we talk about the changes, let's reflect on the inception of data engineering.
The practice of data engineering, and the need for dedicated roles, was born out of complexity and the need for someone to manage it. With an explosion in data volumes and an increase in tools to manage it, a domain-specific engineer was born with a specialized set of skills. Some of these engineers came from the world of database administration, others from software engineering, and some from analytics. As the domain that they worked in evolved, so did their responsibilities and skill sets.
Questions
While the tools and challenges that these engineers solved changed, the underlying need for their work remained the same: to support the answering of questions.
- "Why do I have no customers?"
- "What is the significance of these protein structures?"
- "What is the airspeed velocity of an unladen swallow?"
The wrangling of data, storage, processing, and everything in between was done out of necessity so that you, or some stakeholder, could make sense of it.
Context
We're reaching another inflection point where the underlying tools are evolving. This time, we're seeing an emergence of systems that can make sense of large volumes of data. However the new "complexity" is that these systems rely on carefully managed metadata, or context. This context can store how data is loaded, transformed, and queried. You no longer have to write the tool-specific code to interface with various APIs, nor the queries to make sense of it. You just have to ensure that the information is present and curated.
The new responsibility of this kind of engineer is to manage that context.
Some folks are speculating that we will see the rise of the "context engineer", but while it's uncertain what the title will be, it's clear that we are seeing a change in how data engineers operate, and an evolution in the responsibilities that they hold to support the answering of questions.