“The improvement is 5x” [FULL STOP], it’s designed to make you think, “That’s impressive; it must be worth my time and money.”
However, a standalone statement like that is a red flag, and knowing how to interrogate vague metrics is a foundational skill for anyone who wants to separate real value from clever marketing.
That’s why I invite you to put on your data analyst goggles and uncover the context needed to get insights you can act on.
The popularisation of analytics followed a similar path to psychology, moving from specialist guidance to self-help books and eventually to blog posts, YouTube videos, and inspirational Instagram quotes stamped on T-shirts. The whole idea was to pour our pain into words we could proudly wear on the best piece of clothing out there — the hoodie — and finally explain to the world what an analyst (me!) does, thinking it explains the analytics field, too.
Of course, for the folks who caught me on a laundry day, I had a simple explanation of analytics ready to go, and shared that analytics is “What you see is rarely what you get,” while statistics, on the other hand, is “The truth is out there” (as stated in The X-Files).
Then I added that I’m always enthusiastic about searching for the “truth,” referencing my six years of Ivory Tower experience, but I prefer to swim in analytical waters now.
That’s why I stand by the statement:
Flashy dashboards backed with data storytelling often aim to obfuscate the untrained eye by presenting cherry-picked insights.
Meanwhile, the trained eye knows something more is hiding behind the metrics whenever a statement like this shows up:
“The improvement is 5x”
Followed by the full stop/period/full point/dot or whatever you prefer to call it. I don’t know about you, but it hurts when I look at that sentence knowing someone dared to call it a success metric.
For the sake of the common sense I hope still exists in the current AI limbo, let’s rewrite the above statement the way a metric should be presented by adding…
#1: The dimensions
I know a few of my friends who are still Simon Sinek followers would say, “Start with why.” 🙂
Nope. I started with “what,” and asked, “The improvement of what?”
I dared because the first thing that went through my mind after seeing the word “improvement” hanging like that was:
print(“The improvement is 5x.”)
And that’s no way to get insights anyone can act on, is it?
But imagine for a second the statement was formulated as “The model accuracy improvement is 5x.”
If that had been the case, I would have pictured something different and envisioned a small table with specific performance dimensions, such as model accuracy, and their exact logged measurements.
However, to be sure the “5x” wasn’t pulled out of thin air, the second dimension missing from this imaginary table of mine is the date/datetime. Which would make our statement sound like:
“The monthly model accuracy improvement is 5x.”
Image generated by the author using Gemini.
Now, that’s better, or at least it feels this way, because we can claim we derived the improvement by comparing model accuracy logs across monthly runs.
But, to understand this “5x”, or any improvement whatsoever, another important missing piece of information is…
#2: The baseline
Which is why I will continue with my “what_”_ questions by asking, “The improvement from what?”
A “5x improvement” really does sound amazing until we realise the baseline model accuracy for predicting the exact right outcome out of a hundred possibilities was 1% last month, and now it’s 5%.
If we saw the raw value of 5%, we would know it means the model’s predictions are still wrong 95% of the time, and wouldn’t consider it a kind of improvement that should drive our actions. That’s why we often won’t get presented with the raw numbers, but simply a “5x” because it looks great on a dashboard.
Having this information, we can rewrite our statement again:
“The monthly model accuracy improvement is 5x, growing from a baseline of 1% to 5%.”
That looks better again. Still, seeing a specific time period attached brings me to the next missing piece…
#3: The comparison period
Which leads me to my last “what” question: “The improvement compared to what period?”
Image generated by the author using Gemini.
Our statement still doesn’t tell us what kind of comparison it’s making — an evolution over time, a fixed-cadence comparison (month-over-month, quarter-over-quarter, year-over-year), or an arbitrary period comparison?
To reframe: Is it a direct comparison to the previous month, or across several consecutive months? Maybe a year-over-year comparison, where this month was compared to the same month last year? Or is it just an arbitrary comparison between two hand-picked months?
Assuming we got an answer, we can tweak the previous version once more:
“Comparing May 2026 to April 2026 results, the monthly model accuracy improvement is 5x, growing from a baseline of 1% to 5%.”
Much better. The sentence finally tells us what improved, from what baseline, and over which period. And now, to the delight of some of my friends, I will ask…
The “why” questions
To wrap up, the two “why” questions I would ask when served with a statement like “The improvement is 5x” are:
**#1: Why should this metric concern me?**The metrics that drive the presenter’s decisions aren’t always the ones that drive yours, and the same goes for the level of detail. What’s enough for them to decide often isn’t enough for you to act. So the next time someone presents a statement similar to the one above, ask them who it is for and which actions it’s supposed to drive.
#2: Why were the “what” questions left out?
Yes, starting with “why” is relevant, especially when you are trying to understand a problem for a specific use case. But when it comes to understanding the metric in front of you, someone presented with its own “why” in mind, you need to interrogate it with “what” questions so you don’t get fooled by flashy dashboards and data storytelling.
Fun fact to end on: Chameleons can move each eye independently, one tracking a threat, the other scanning the horizon, giving them about 180° of horizontal and 90° of vertical vision.
Kind of like people with a trained analytical eye, who are great at spotting what’s in front of them: one eye on the metric shown, the other on everything around it that hasn’t been.
Thank you for reading.
This post was originally published on Medium. If you found it valuable, feel free to share it with your network, and connect for more stories on Medium ✍️ and LinkedIn 🖇️.
Instagram Engineering
Shopify Engineering