that annoys me is the countless people online, in person, and even in my comments section saying “how AI will replace data scientists.”
I find this frustrating because it often comes from people who aren’t working in the field, and it discourages those who would be great data scientists from pursuing this career path.
Not to mention, I firmly disagree with this view and believe AI will not replace data scientists, at least definitely not within the next decade.
And this is coming from someone who has worked in this field for 5 years across a range of companies, and has seen what the industry was like pre- and post-AI.
I have zero concern about AI taking my job as it stands, and in this article, I want to explain exactly why I think that and put an end to all this scaremongering.
You Need To Learn AI
Before we get into the actual “meat” of the article, let me start off by saying that I am not a complete AI hater.
I use AI daily, and consistently up-skill myself in AI as it is a crazy productivity tool for:
- Writing boilerplate code
- Brainstorming technical ideas
- Creating and drafting documents
- Producing data visualisations and graphs quickly
- An overall great intellectual sparring partner
This technology is here to stay, and you need to learn to use it; otherwise, you will be left behind.
Competency with AI tools will become the norm, just as everyone is expected to use email nowadays or know Microsoft Word.
AI won’t replace data scientists, but an individual with fewer technical skills but greater AI proficiency likely will.
As a data scientist, you need to be well-versed in tools like:
And so many more.
These will become staples in our industry, just like Python has become the lingua franca of machine learning.
It’s inevitable, and you need to get on board the ship as soon as you can.
There Will Be Bigger Problems
Let’s break down the skills AI will need to develop for it to fully replace data scientists:
- Break down ambiguous business problems into framed mathematical systems or algorithms.
- Communicate with non-technical stakeholders and explain certain results with live questions.
- Write error-free production code all the time to ensure all business-critical decisions run smoothly.
- Make both logical and human trade-offs between complexity, architecture design, and the development process.
- Build relationships and trust across a team, a company, and an industry.
If AI mastered all these skills to a level better than a current data scientist, what job would not be gone?
Most of them would go extinct as well.
If this happened, we have far bigger problems to worry about, almost singularity-level problems, and your concern about whether you should go for a data science job will pale in comparison.
The AI singularity is a theoretical future point when artificial intelligence surpasses human intelligence, leading to rapid, uncontrollable, and irreversible technological growth.
If data scientists are replaced, there will likely be bigger fish to fry in our lives than simply worrying about our careers.
Lack Of Mathematical Reasoning
One thing AI greatly lacks is mathematical reasoning.
I am not talking about the layperson maths that most people ask AI like:
- Help me find the gradient of this function.
- Calculate the determinant of this matrix.
- What is the formula for Fibonacci numbers?
What I mean by “mathematical reasoning” is the ability to solve unsolved mathematical problems.
For example, AI currently can’t solve the Riemann Hypothesis because it lacks the creativity and conceptual reasoning to make a major breakthrough in pure mathematics.
The Riemann Hypothesis is a famous unsolved prediction that suggests there is a hidden, underlying order to the seemingly random distribution of prime numbers. It centers on the “zeros” of a complex mathematical tool called the Riemann Zeta Function, proposing that all non-trivial zeros lie on a single vertical line (the “critical line”).
The Riemann Hypothesis is an extreme example as it’s arguably the hardest problem in existence at the moment.
However, it shows that AI hasn’t surpassed humans in mathematical abilities, which is a cornerstone of data science.
Most people forget that these AI models are actually a type of model called large language models (LLMs), specifically designed to predict the next word from a pre-calculated probability distribution.
These models can only output, or base their output, on data they have seen; they can only go off what exists and not necessarily create anything “brand new.”
The data science job requires developing novel solutions to unseen problems. In fact, we actually need data scientists and machine learning practitioners to build these AI models in the first place and maintain them!
AI Still Makes Mistakes
As someone who works with these tools every single day for a range of applications, AI makes so many mistakes it’s ridiculous.
These LLMs often “hallucinate”, which is a term you have likely heard and is when these AI models produce outputs that seem plausible but are actually very incorrect.
This stems from the fact that they are probabilistic models by nature and can potentially “string” words together that make no sense to meet users’ demands or expectations.
Humans also make mistakes, but the difference is that most humans are aware of their mistakes after you correct them. They’re not uber-confident in their initial response either, depending on the scenario.
Whereas AI is quite stubborn, clever, and very certain of the answers it gives you, which psychologically tricks us, humans, into thinking it’s correct.
Imagine how jarring this would be in a work setting.
An AI data scientist would not be able to accurately gauge how outrageous or ridiculous its output is, and so it fails to set expectations when you implement its’ given solution.
It misses that lack of nuance and intangibles us humans have about many data science and machine learning projects.
Limit To Performance
What’s interesting to me is that these AI models are not actually getting substantially better over time.
The reason is twofold:
- The underlying algorithm is still the same; all of these LLMs use the Transformer architecture, so each “new” model isn’t actually that “new.”
- There is a limit on the amount of data they can be trained on, as only so much information exists in the world.
For example, OpenAI’s GPT models have been trained basically on the whole of the internet to a certain extent, there is not much “new” data for it to use.
There is literally a cap on how good they can get.
This data also comes from humans, so it can’t exceed human intelligence; that is its ceiling.
These AI models won’t get any better unless there is a massive scientific breakthrough in the underlying algorithm.
And the fact that they won’t get any better means the current state will remain the same, and AI has not yet replaced data scientists.
Can’t Build Relationships
AI is incapable of relationships, despite how many people are sadly getting emotionally attached to these robots.
Humans are social creatures, and most of the world’s business interactions are done through relationships.
People do business, hire, and work with people they like, even if they may not be the most “technically” qualified.
It’s just how we are wired to act from a biological perspective.
A stakeholder will trust you as a data scientist if you have delivered consistent results for them.
Even if an AI comes up with a “better” solution to their problem, the stakeholder will likely prioritise you due to the intangible human relationship you have built.
Every job relies on human connection. Some parts will be automated, but many will not.
In the case of a data scientist, it would be incredibly hard to automate:
- Data storytelling of a technical problem to a specific stakeholder
- Gathering requirements from a business lead for a problem they want to solve
- Communicating and influencing members of other teams and functions
Any active human part would be impossible to replace.
Has Anything Really Changed?
One of my old line managers once asked me:
Has anything really changed since AI has been released?
Sure, we now have better tools to solve certain problems, and productivity in certain aspects of our jobs has increased, but the data scientist role honestly hasn’t changed that much.
Take a minute and think about what has materially changed in your day-to-day life from AI.
I doubt you could name much, if anything.
AI, in its current form, has been around for more than 4 years, yet society as a whole hasn’t been significantly impacted from where I’m standing.
That is all that needs to be said here.
If, after reading this, you really want to dive deep into learning AI, I recommend my previous post, which gives you a full, in-depth roadmap of everything you need to master AI.
You can check it out below!
Another Thing!
Join my free newsletter where I share weekly tips, insights, and advice from my experience as a practising data scientist and machine learning engineer. Plus, as a subscriber, you’ll get my FREE Resume Template!
Dishing The Data
_Weekly emails helping you land your first job in data science or machine learning_newsletter.egorhowell.com
Connect With Me
Instagram Engineering
Shopify Engineering