As technology advances, new roles and specializations are always taking shape. One such field that has become increasingly important is data engineering. But what does a data engineer actually do? And what does it take to build a career in this area?
In our latest episode of Stories from the Hackery, we sat down with two Nashville Software School (NSS) alumni, Teresa Whitesell and Joshua Rio-Ross, to hear about their journeys from our data analytics and data science programs into the field of data engineering. Their stories shed light on how different backgrounds and a passion for problem-solving can lead to rewarding careers building data systems.
Teresa and Joshua came to NSS from different professional backgrounds. Teresa, a Nashville native with an analytical mind, knew she wanted a career in data after gaining some exposure to data analysis on the job. A friend and early NSS graduate recommended our program, leading her to join the first full-time Data Analytics cohort in 2020.
Joshua’s path was more academic; with master's degrees in both mathematics and philosophical theology, he spent ten years teaching math at the college level. Seeking a more sustainable career that still used his mathematical skills, he enrolled in our part-time Data Analytics bootcamp while still teaching, where Teresa served as his junior instructor.
For both, NSS was a place to gain the technical skills needed for a new career. Both went on to complete our Data Science bootcamp as well, adding another layer of skills to their toolbelts. After graduation, both have found their way into data engineering.
One of the key takeaways from the conversation was that data engineering can look different depending on the organization. It’s a field that overlaps with software engineering, data analytics, and data science.
As Teresa explained, her role at HCA has been about creating a new path on a team of infrastructure engineers. "For me, data engineering really means getting the data from the source reliably," she says. "You need data validation, you need to have it automated in a reliable way, getting that into a SQL database for the analytics engineers to pick it up, transform it, and make it useful for the business."
Joshua, who has worked in large enterprise, startup, and consulting environments, describes data engineering as the "process of obtaining and delivering data for a particular use case." He distinguishes it from software engineering by the nature of the end product. Data engineers typically support data-focused outcomes, like an analytics report or a machine learning model, whereas software engineers typically support products.
A key distinction that emerged was the idea of "systems building versus answering questions." Joshua explained, "I think of data engineering as a discipline of systems building... Do you want to build a system?" This is a helpful way for those considering a tech career to think about where their interests lie.
Though neither Teresa nor Joshua started in a dedicated data engineering program, they found that their training in data analytics and data science at NSS provided the perfect foundation.
Understanding the "downstream" needs of data analysts and scientists is critical for a data engineer. "It's very valuable for you to know what the data's going to be used for that you're delivering," Joshua notes. "Don't start doing your work until you know exactly what the people who are gonna use this data are trying to accomplish."
Teresa found that even in her data analytics capstone project at NSS, she was spending most of her time on data engineering tasks without realizing it. "I didn't realize it at the time, but I spent 90% of the time that I had available doing data engineering work because I was getting complex messy data from an API," she recalls. This experience, while challenging, set her up for the work she does today.
While graduates of our software development bootcamp have also gone into data engineering, Joshua’s and Teresa’s backgrounds in analytics and data science have allowed them to bridge the gap between technical and non-technical stakeholders. "I have to be able to meet them where they are," Teresa says of her work with finance teams. "I don't know that I could have been a good data engineer without knowing analytics and data science."
With the rise of generative AI and machine learning, the need for high-quality data has never been more apparent. "Quality data is the only way to get quality output from gen AI or machine learning, or any kind of model," Teresa states. Both she and Joshua emphasized that robust data engineering is the foundation for any successful AI implementation.
To hear more about their experiences and get their advice for anyone considering a career in data engineering, listen to the full conversation on the Stories from the Hackery podcast.
Listen on SoundCloud or watch on YouTube.
00:00 Introduction to Stories from the Hackery
01:27 Meet Teresa Whitesell: From A Desire to Work with Data to a Career in Data Engineering
05:52 Joshua Rio-Ross: From Academics to Data Engineer
12:20 What is Data Engineering
17:55 The Overlap of Data Engineering and Software Engineering
20:48 The Importance of Analytics in Data Engineering
28:16 Systems Building vs. Answering Questions
31:03 Understanding Industry-Agnostic Data Engineering Skills
32:18 NSS’s Data Engineering Bootcamp
33:23 Choosing Between Data Engineering and Other Pathways
36:35 The Role of Data Engineers in Organizations
40:36 The Impact of Generative AI on Data Engineering
53:55 Technology Guilty Pleasures
58:20 Conclusion and Final Thoughts