The Silent Crisis in Data Engineering

In partnership with

Find out why 1M+ professionals read Superhuman AI daily.

In 2 years you will be working for AI

Or an AI will be working for you

Here's how you can future-proof yourself:

1. Join the Superhuman AI newsletter – read by 1M+ people at top companies

2. Master AI tools, tutorials, and news in just 3 minutes a day

3. Become 10X more productive using AI

Join 1,000,000+ pros at companies like Google, Meta, and Amazon that are using AI to get ahead.

Sign up and start learning AI

Hello everyone and welcome to my newsletter where I discuss real-world skills needed for the top data jobs. 👏

In this article I’ll be discussing the scarcity of skilled data engineers. 👀

Not a subscriber? Join the informed.  Over 200K people read my content monthly.

Thank you. 🎉

It’s been the top job in information technology for a long time. Even with all the hype around Ai and all the sundry roles it’s created, one job remains king, the data engineer. 👑

Let’s define the role first. A data engineer is someone who builds and maintains the systems that move and organize data so it’s ready for others—like data analysts and machine learning engineers —to use. That’s a very high-level definition but ideal for this conversation.

Data is the new oil. Companies have been amassing data in every form and storing it wherever they could find a drive to put it on. This picture tells the store.

The Y axis is in Zettabytes. 🤯Companies are pouring billions into data platforms, AI-driven analytics and real-time pipelines. The demand for data engineers has skyrocketed, but the supply of qualified professionals remains stagnant.

For the longest time, companies funneled money into hiring data scientists, expecting them to drive innovation. Unfortunately, most were from academia and had no real-world data skills. Data is a different animal in the real-world.

ChatGPT at Work: Free Resource Bundle

Power up your productivity with Mindstream's exclusive ChatGPT toolkit , designed for professionals who want to work smarter, not harder.

Your free bundle includes:

• ChatGPT Decision Flowchart

• Advanced Prompt Templates

• 2025 AI Productivity Guide

• Task Automation Framework

• Industry-Specific Use Cases

Join thousands of AI-powered professionals by subscribing to our daily newsletter. Get the complete bundle instantly after signup - no extra steps required.

Click here to download your free bundle

If you been following my content for any length of time, you’d know I predicted the fall of the data scientist over a decade ago. For those unfamiliar with this space, that role has collapsed causing billions of dollars in losses. The success rate on data science projects in the real-world is an astounding 4%. Yes, you read that correctly. The number of failed projects outside of big tech sits around 96%.

Without well-architected data pipelines, data scientists are useless. Poor data quality, slow query performance and lack of scalable infrastructure have turned many AI and analytics projects into expensive failures.

Even now, job postings for data engineers are outpacing data scientists and machine learning engineer roles, yet the talent pipeline is thin. Universities still prioritize data science over data engineering and bootcamps are laughable, barely scratching the surface of what’s required to build reliable and scalable data platforms.

Colleges are companies and prioritize all Ai opportunities over data engineering because Ai courses and programs make a lot more money. I’m not blaming them for this, however; this further exacerbates the scarcity of data engineers.

Big tech isn’t helping either. Another reason for this crisis is that tech giants like Google, Amazon and Microsoft are scooping up experienced Data Engineers at exorbitant salaries, leaving startups and mid-sized companies struggling to hire.

The rise of AI-powered ETL tools and automated pipeline solutions have many believing the data engineering role is another one that Ai can automate. Nope. Not going to happen.

AI can enhance productivity, but it cannot replace the deep expertise required to design scalable architectures, optimize query performance and troubleshoot complex system failures. 👏

The Ai lie will make the need for skilled data engineers even more critical, as companies integrate machine learning models directly into their data pipelines.

What can be done? How can the gap be bridged? Well, in the short term it can’t. There are very few to zero entry level jobs in data engineering. So, let’s discuss some longer term options.

• Redesign Education - It’s dying anyhow. Now is a great time to rethink how to teach. Stop all the focus on theory, start teaching real-world skills.

• Level Up Other Data Professionals - This is actually where most data engineers come from. They are often DBAs who simply add working with unstructured data versus the structured data they are familiar with.

• Improve Retention - Yes, the salaries are going to be big but skilled technical employees need more than money. They need a work/life balance and are going to demand c-level like packages including stock options, time off and retention bonuses.

There is good news and for most of you… great news. Many of you are considering careers in data engineering. If there’s a scarcity now, there will be one in five years also. 🙂 The real-world doesn’t respond quickly to change. That means for those of you contemplating a career in the data space, the data engineer is certainly the top contender. 🎉

If you’ve followed me for any length of time you’ll know the exact path. If you’re new to data engineering here are the fast facts. 

• There are no entry level jobs in data engineering.

• You’ll need to start out as a data analyst or other more entry level role.

• Your timeline from scratch to a real-world role is around 5-7 years.

• The top skill for every data engineer is SQL.

• If you’re serious about this role read my article on data engineering .

• This isn’t an easy path but it’s obtainable to those who want to put the work in.

Guess what the top skill for the data engineer is? 🤪

Yep. SQL 👈

I know, I know. I’ve authored over 50K posts in my lifetime on data roles, how many times do you think I wrote that word? (… or acronym) 😢

Say what? GPTs are custom versions of ChatGPT that users can tailor for specific tasks or topics by combining instructions, knowledge, and capabilities.

SQL Server Hyper Focus is ready. 👏The first and currently only customized GPT to help you learn SQL Server interactively. SQL is the top skills for every data role. SQL Server is one of the top relational databases on earth. You must know the basics of relational databases. If you’re going to work on the Microsoft ecosystem, then you must know as much as you possibly can about SQL Server. This GPT is hyper focused on the interview questions you’ll see in your first entry level interview.

Who is my target student for this GPT?

• Entry level data analysts who are going to work in the Microsoft ecosystem.

• Entry level DBAs who are specializing in SQL Server.

• Entry level data engineers who are going to work in the Microsoft ecosystem.

Once you purchase the GPT, simply add it to your ChatGPT PLUS environment.

Here’s what you’ll need to start learning with SQL Server Hyper Focus

⚠️ You will need the plus version of ChatGPT. That’s not included in the price of the GPT.

With this opportunity, you are purchasing the model only. This does not include access to all of the other courses. 

🥳 The cost of the GPT is $39. However, I’ll knock that down to $19 for the first 20 readers. Simply use this code at checkout: maynewsletter2025

Download it Now

It’s dirt cheap and I’ve carefully curated the top SQL Server questions seen on real-world interviews. 

You want elite level salaries, great; companies want elite level data engineers. Stop planning and start learning. This GPT is guaranteed to help you make it past the SQL Server tech interview.

Thanks for reading and have a great day. You can do this. Two hours every day and you’re in the money. 👏