“Hey, you should start a Data Engineering Bootcamp!” 

That is an actual quote from a VP at one of our hiring partners - or at least, that’s as close as I can recall what he said. Anyway, that’s my story and I’m sticking to it. He then proceeded to say that as much as he liked hiring our graduates in data analytics and data science for his data team, their biggest challenge was finding people with training and/or experience as data engineers and he thought that was an opportunity for NSS. 

That caught our attention because we’re always on the lookout for trends and needs in the tech talent market in Nashville and Tennessee. An almost identical statement several years before ultimately led to NSS launching our second career pathway - Data Science (followed by Data Analytics). In a similar fashion to data science, the data engineering comment led us to a long period of monitoring the local job market, talking to employers and working tech professionals, and researching broader trends in tech employment. 

While we’ve been exploring data engineering we navigated the COVID trainwreck of 2020 to 2022, the slowdown in tech hiring circa 2023 to 2024, and the generative AI disruption that’s still ongoing. Separating the signal from the noise during all that was difficult but we think we’ve finally managed to do so. Curiously, one strong signal emerged from watching over a few years and seeing a steady stream of NSS graduates - from all programs, data analytics and data science for sure but also full-stack web development and software engineering - either get hired into data engineering roles or switch into data engineering roles during the first year or two after starting their careers. That might have been the final factor that led to the news we’re announcing in this blog post. 

Announcing our Data Engineering Bootcamp

I think I buried the lede sufficiently, so let me cut to the chase, as they say. We are launching a new career pathway at NSS: Data Engineering. To kick things off we are starting to enroll a first cohort into our Data Engineering Bootcamp - enrollment will commence on April 9. Next week we will publish a follow-up blog post to this one which will provide a lot more detail on the NSS bootcamp, the local job market for data engineering, and other information specific to the launch of our new program. And we will talk about how eligible applicants will be able to attend the Data Engineering bootcamp for FREE. 

So watch this space for next week’s post if you’d like more information on our new program. For the rest of this week’s post, I’m going to lay out a general introduction to what data engineering is, how it relates to other tech career fields, what’s driving the job growth in data engineering, and some of the core skills and technologies used by today’s data engineers. 

What is Data Engineering?

Behind every slick dashboard, AI model, or data-driven business decision, there’s an invisible layer of engineering making it all possible. That’s where data engineers come in.

Data engineering is the discipline of designing, building, and maintaining the infrastructure and architecture that allows organizations to collect, store, transform, and analyze large volumes of data effectively. Data engineers create robust data pipelines that extract information from multiple sources, clean and structure the data, and make it accessible for data analysts, data scientists, and business intelligence teams to derive insights and support decision-making.

The Pragmatic Engineer blog borrows a definition from Fundamentals of Data Engineering by Reis and Housley. 

“Data engineering is the development, implementation, and maintenance of systems and processes that take in raw data and produce high-quality, consistent information that supports downstream use cases, such as analysis and machine learning.

Data engineering is the intersection of security, data management, DataOps, data architecture, orchestration, and software engineering. A data engineer manages the data engineering lifecycle, beginning with getting data from source systems and ending with serving data for use cases, such as analysis or machine learning.”

Data engineers develop expertise around getting data, loading data to different structures, cleaning and transforming data, and creating pipelines and other automations to support data-centric tasks like Analysis, Machine Learning, and Generative AI.

In essence, data engineers are the architects and builders of the data world. They ensure that raw data is transformed into a format that can be easily used, providing the critical bridge between data generation and data utilization. Their work involves a combination of software engineering, database design, cloud computing, and data management skills to create scalable and efficient systems that can handle massive amounts of information from diverse sources.

Factors Driving Data Engineering Growth

Data engineering has emerged as one of the most rapidly expanding career fields in the technology sector over the past decade. The explosive growth of digital data generation—driven by cloud computing, IoT devices, social media, and complex business applications—is driving the need for professionals who can transform raw data into meaningful, actionable insights.

Among the factors driving growth in demand for data engineers are:

1. Data-driven Decision-Making: Governments and companies across industries are increasingly relying on data-driven decision-making. From healthcare and finance to retail and manufacturing, organizations are seeking to leverage data as a strategic asset.
   
   
2. Explosion of Data: Over the last two decades, companies of all sizes have been generating and collecting massive amounts of data—from app usage, IoT sensors, transactions, customer interactions, logs, and more. Handling this volume, velocity, and variety of data requires specialized engineering talent.
   
   
3. Cloud Computing Expansion: The migration of business infrastructure to cloud platforms like AWS, Google Cloud, and Microsoft Azure has created complex data ecosystems that require specialized expertise.
   
   
4. Advanced Analytics and AI: Machine learning and artificial intelligence technologies are completely dependent upon robust, well-structured data pipelines, which data engineers are responsible for creating and maintaining. The quality of the data sets determines the quality of the machine learning / AI models.
   
   
5. Fragmented Tooling & Ecosystems: Data ecosystems are more highly fragmented than what’s typically seen by software engineers. Orchestrating ETL/ELT pipelines, managing data lakes, building data warehouses, and working with various APIs and storage formats takes a specialized, technical skill set.

The first four factors above are all long-term trends at various stages of evolution. The emergence of machine learning and then more recently generative AI promises that these trends will continue to grow and evolve into the foreseeable future.

So What Does a Data Engineer Do?

A data engineer designs, builds, and maintains the architecture that enables data to flow from where it's created (think: user activity, transactions, IoT sensors, APIs) to where it’s consumed (dashboards, reports, ML models, decision-making tools).

This often includes:

• Ingesting raw data from various sources
• Transforming and cleaning that data into a usable format
• Storing it efficiently in databases or data warehouses
• Ensuring the pipelines are fast, reliable, and scalable
• Collaborating with analysts, scientists, and business teams to understand their data needs

Another way to think about what data engineers do is shown in the Data Engineering lifecycle below. This diagram is from the excellent Fundamentals of Data Engineering by Reis and Housley. In their words, it shows the components and undercurrents of the data engineering lifecycle. In turn, this diagram helps to explain the competencies and skills needed by data engineers as we discuss in the next section. 

Think of data engineers as the architects and builders of the modern data stack—they make sure the data infrastructure is solid, scalable, and efficient.

What Skills Do Data Engineers Need?

Data engineering is a deeply technical field which also requires strong problem-solving skills and a big-picture mindset. Here are the core competencies:

1. Programming and SQL

   1. Python is today’s go-to language for data engineering. It’s used for scripting pipelines, working with data frames, and automating workflows, etc. 
   2. SQL is essential for querying and transforming data, especially in modern data warehouses.
   3. Proficiency in both relational and non-relational database systems.
      
      
2. Data Modeling
   1. Knowing how to structure data using fact and dimension tables or in other architectures is key for making data useful downstream.
      
      

3. ETL/ELT and Pipelines

   1. Building workflows to Extract, Transform, and Load data—using either custom workflows built in languages such as Python or increasingly using tools like Apache Airflow or dbt.
      
      

4. Cloud and Big Data Tools

   1. Familiarity with cloud platforms (AWS, GCP, Azure) and distributed processing tools like Spark is becoming foundational knowledge for data engineers.
      
      

5. Data Warehousing and Storage

   1. Knowledge of how to design and implement centralized repositories for structured, semi-structured, and unstructured data.
   2. Understanding how to use and manage data warehouses like Snowflake, BigQuery, or Redshift.
      
      

6. Data Quality and Monitoring

   1. Making sure data is accurate, complete, and delivered on time—this often involves testing, validation, and alerting.

In today’s data-driven world, the role of a data engineer is more essential than ever. As companies rely on data for everything from customer insights to AI innovation, data engineers are the ones making sure that data is trustworthy, timely, and accessible. Without them, the whole data stack falls apart. For another perspective on the essential nature of data engineering, Charlie Apigian shared his thoughts on the importance of data engineering in our podcast this week (listen to the clip here).

The skills and competencies of data engineers overlap with both the software engineering skills learned and used by graduates of our full-stack web development bootcamp and the data analytics and data science skills acquired by graduates of those two bootcamps. But the skills are applied to different sorts of work than they are on those other career pathways. On the job, the role of the data engineer can be viewed as sitting between the world of applications built and maintained by software developers and the world of analytics and AI projects and applications created by data analysts, machine learning engineers, and data scientists. And as generative AI continues to unfold and become an enabling / embedded technology in applications, data engineers will increasingly be responsible for the data architecture used by AI application developers / AI engineers. 

Tell Me More About the NSS Data Engineering Bootcamp

All of the above feels like enough for everyone to chew on this week. For more information on the NSS bootcamp, watch for the next post in our series coming next week. We’ll talk a bit more about the curriculum, some of the logistical details of the program, and we’ll invite those that are interested to a Data Engineering information session scheduled on Tuesday, April 8, at 6pm.

Sign up below to receive our newsletter and updates on Data Engineering

We'll provide regular updates as we launch this new program.