A Study into the Accuracy of SAP

What This Article Covers

  • Introduction
  • The Accuracy Tracking Table
  • Overall Accuracy Calculation and Future Accuracy Forecasting
  • Research Background
  • Partners in Inaccuracy?
  • A Recognized Advantage Versus SAP
  • Is Accuracy SAP’s Concern?
  • Determining Source Quality
  • Background on Accuracy Measurement
  • Tracking Predictions and Outcomes

Introduction

In most research, the topic is introduced and the background provided, and the data is provided second. However, this research actually serves two purposes. One purpose is to serve as a long-term resource for people and companies interested in fact-checking SAP. For this reason, we have placed the data first, and the explanation second.

  • If you are here just for the fact checking, the table you wish to query is below.
  • You do not need to type in the entire term, just type in the first few letters of the term you seek, and the list will begin to filter immediately.
  • If this is your first time seeing this research, please click the Jump to Research Explanation link below to be taken to the full explanation of the research.

Jump to Research Explanation

A second purpose is for those with more of an academic interest in SAP’s accuracy.

*Scan the list, or use the search box in the upper right above the table.

SAP Versus Brightwork Reseach & Analysis Accuracy

SAP Topic Area (Prediction)SAP's Accuracy PercentageBrightwork's Accuracy PercentageFirst Made by SAP First Made by BrightworkLink to Article with Full Explanation
Netweaver to Become Major Infrastructure "Product." 0%100%20032010See Article Link
SAP to Host its Own Products and not Rely on AWS0%N/A2015N/ASee Article Link
SAP's Run Simple Marketing Program Reflects the Reality of SAP0%100%20152016See Article Link
HANA Will be an Perfect Fit for ERP Systems?0%100%20112017See Article Link
HANA Will Make Run MRP Faster than On Any Other Database0%100%20152016See Article Link
You Can Run S/4HANA on a Smartwatch0%100%20152017See Article Link
SAP Mobility is Number One in Mobility Due to Sybase Acquisition0%N/A2010N/ASee Article Link
HANA is the Fastest Growing Product in the History of Software0%100%20132017See Article Link
Fiori is Commercially Successful, Inevitable and Broadly Used by SAP Customers0%100%20142016See Article Link
With SAP Fiori and SAP Screen Personas, SAP Made Fiori Widely Available..etc..0%100%20132017See Article Link
Apple and SAP Partnership1%99%20162017See Article Link
SAP CRM 7.0 is the Best Product in the Field0%N/A2008N/ASee Article Link
SAP ByDesign Will Beat or Shock Netsuite with a 90 MPH Fastball0%N/A2014N/ASee Article Link
SAP PLM10%90%20042009See Article Link
SAP MDM5%95%20052009See Article Link
SAP SPP5%95%20062010See Article Link
SAP Will Have a Billion Users by 20150%N/A2012N/ASee Article Link
HANA Will Run 100,000x Faster than Any Other Technology0%100%20122016See Article Link
HANA Will Help Employees Work from 10 to 10,000 Times Faster0%N/A2014N/ASee Article Link
Ariba is Useful for Direct Procurement0%100%20122016See Article Link
SAP to Introduce Revolutionary Health Application Based on HANA0%100%20122017See Article Link
S/4HANA Has a Simplified Data Model0%100%20152017See Article Link
SAP ASAP Methodology Sped Implementations0%100%19972015See Article Link
SAP Rapid Deployment Solution Sped Implementations0%100%20122017See Article Link
S/4HANA is Complete0%100%20152017See Article Link
Design Thinking Speeds SAP Implementations0%100%20132017See Article Link
Best Practices are Contained within SAP0%100%1990's2010See Article Link
The SNP Optimizer Provides Good Supply Planning Output0%100%19992012See Article Link
The PP/DS Optimizer Provides Good Production Planing and Scheduling Output0%100%19992010See Article Link
Solution Manager Would Become "the" CMS for SAP Implementations0%100%20012012See Article Link
SAP is Finished on Oracle0%100%20152017See Article Link
SAP's S/4HANA Implementation Numbers20%80%20152017See Article Link
Indirect Access (Type 2) is a Legitimate Claim0%100%20142017See Article Link
Fiori is the Best UI in Enterprise Software0%100%20132017See Article Link
ByDesign to Run on HANA0%100%20172017See Article Link
SAP's ERP Would be The Only System a Company Would Ever Purchase0%100%Since SAP's beginning2014See Article Link
All of an SAP Prospect's Existing Applications are Legacy0%100%Since SAP's beginning2017See Article Link
All Non HANA Databases are Legacy0%100%20142017See Article Link
S/4HANA is so Different from ECC That it is Not Covered as a Normal Free Upgrade0%100%20142016See Article Link
SAP Simplifies The Customer's IT Landscape0%100%Since SAP's beginning2014See Article Link
SAP's Integrated Suite Offers a Lower TCO0%100%Since SAP's beginning2014See Article Link
Reengineering Business Processes to Match SAP Leads to Long Term Competitive Advantages0%100%1980s2017See Article Link
SAP Support is a Good Value0%100%Since SAP's beginning2017See Article Link
In Memory Computing (via HANA) is Something New and Different0%100%20112017See Article Link
The Count of Fiori Apps is High0%100%20152017See Article Link
Hasso Plattner and his PhDs Invented HANA0%100%20112017See Article Link
HANA Has a Lower TCO0%100%20142017See Article Link
Shelf Life Planning Works in APO SNP0%100%20002010See Article Link
Parallel Processing Works with the SNP Optimizer0%100%20002013See Article Link
SAP on Alliance with Google0%100%20152018See Article Link
S/4HANA in the Cloud Can be Easily Extended 0%100%20162018See Article Link
Only SAP HANA is an ACID Database0%100%2017 (est)2018See Article Link
All Non HANA DBs are Legacy0%100%2017 (est)2018See Article Link
Is it SAP Leonardo that Ensures Frozen Ice Cream Delivery?0%100%2017 (est)2018See Article Link
SAP to Open S/4HANA to AnyDBN/ATBDN/A2017See Article Link
Fiori Will Not Survive (To be replaced by new UI initiative)N/ATBDN/A2017See Article Link
Sybase Database Will be a Good Acquisition0%100%20102012See Article Link

*Any cell with N/A means that either SAP or Brightwork did not make a statement about the line item.

Overall Accuracy Calculation and Future Accuracy Forecasting

The following is a straight average of all of the items listed in the table. This is a record of statements and predictions made over decades. In statistical forecasting, the historical forecast error is typically used to produce the estimate of what the future forecast error.

Following this precedent, it should be considered a useful accuracy percentage to apply to future predictions by SAP.

What should be clear is that SAP has a long-term issue with accuracy. This is the only published study into SAP’s accuracy. It is not generally understood how low SAP’s accuracy is by those that work in SAP or even those vendors that compete with SAP. 

A Recognized Advantage Versus SAP

Brightwork had the advantage of making observations years, in some cases decades after SAP initially made its statements/predictions. However, Brightwork performed well even for recent SAP projections. Brightwork has made several predictions which contradict both SAP and the overall market. At this point, not enough time has passed to determine if these predictions can be considered to have come true. Therefore they are marked as TBD.

To be accurate, any entity must value accuracy over other objectives. It should be remembered that companies are normally said to maximize profits, not to maximize the accuracy of their statements to customers. SAP’s score illustrates the extremes this can take. 

Is Accuracy SAP’s Concern?

SAP is trying to drive revenues and is very obviously not attempting to be accurate. Individuals within SAP may or may not believe what they are saying. The study is not engaging in an analysis of what statements SAP representatives know are false when they say them. It only analyzes if the statement turned out to be true.

Research Background

For obvious reasons neither SAP nor the massive army of companies and people that make money from SAP and cover up for them are not fans of the research, we do at Brightwork Research & Analysis.

We have been accused by SAP and SAP IT departments of the following:

  1. Having an ax to grind (which we responded to in this article).
  2. Of biting the hand that fed us (which is the opposite of the previous argument, which is having an ax to grind). This argument (biting the hand that fed us) we agree with, but it does not make our research untrue, merely disloyal. We guarantee no loyalty to any software vendor or to any consulting company or entity, whether we have financially benefited from them or not.

SAP’s inaccuracies are repeated through a network of entities that do not care about accuracy much more than does SAP. This topic is the subject of future research, with IT media entities also being scored for their accuracy on SAP. 

Partners in Inaccuracy?

What SAP and other SAP proponents are careful not to do is to address the conclusions of our research and analysis. SAP prefers to direct people to sources of information like

  • ComputerWeekly
  • Forbes
  • CIO Magazine
  • Gartner and other media entities and IT analysts that SAP pays.
  • Or, to Deloitte and Accenture and other SAP consulting partners, that is entities that have huge businesses built around billing for SAP consulting.

Therefore, according to who SAP recommends that their customers that listen to, the universal feature is that the sources in some way have a financial tie to SAP, which we covered in Which Sources Does SAP Consider Credible?

SAP seems to have a precise definition of what is a credible, source and it seems to line up very carefully with sources that unthinkingly repeat what they say, and where money is involved for the entity that obeys SAP.

So how is a person trying to get the straight story to determine who is credible?

Determining Source Quality

In our view, a compelling way to determine source quality is to check their track record. Therefore, in this article, we will compare SAP’s track record to our own to see which is more reliable.

Background on Accuracy Measurement

We provide a relative accuracy measurement as part of our Honest Vendor Ratings.

We think that this type of measurement of accuracy is essential to decision making in IT. In fact, our book Rethinking Enterprise Software Risk is highly based on evaluating the accuracy of sources.

However, this is the first time we measured, calculated and documented a vendor’s accuracy in anything like this level of detail. It is a lot of work, and we don’t know anyone who does it. First of all, to do something like this, you would need to have financial independence from the source, so that rules out virtually all media entities from doing anything like this. Secondly, there is no way of monetizing something like this. Media and IT analysts primarily make their money by promoting software vendors, and you can’t collect money from vendors by challenging their assertions. It also tends to make a lot of people angry at you to boot.

Tracking Predictions and Outcomes

SAP’s and our predictions can be easily tracked through the published articles where they and we make the prediction. The determination of whether that prediction is true is a bit more tricky. But in most cases, we have informative articles that explain what the eventual outcome was.

Another trick in measurement is that some predictions, for instance, that Brightwork or SAP made, have not come true yet. But it does not mean they will not come true. In those instances, we don’t give a complete rating, but instead, explain the current state of the prediction.

Finally, some predictions cannot be declared as true or false, that is they are not binary. Some predictions worked out to some degree. In many cases, it makes the most sense to provide a percentage.

Here is how we scored the outcomes of predictions.

  1. A zero percent rating would mean that the prediction did not come true, or the exact opposite happened.
  2. A 75% rating is that the prediction mostly came true.
  3. A 100% rating means that the prediction entirely came true.

This is not a “perfect science,” but (unless you work for SAP or an SAP surrogate) we think the readers will find that this is entirely a beneficial exercise. One might question the entity rating themselves against SAP and the major entities that provide information about SAP. However, the results were so noticeable that we have every incentive to grade ourselves harshly.

Something we also have planned in this area is to measure the accuracy of the statements of IT media entities concerning SAP.

Conclusion

There you have it. So given this comparison Brightwork, Research & Analysis beats SAP by a substantial margin in predicting SAP.

The first observation is that SAP has a very low accuracy level for its statements. Almost none of the projections that SAP makes come true.

How are we able to have such a high accuracy level versus SAP?

Well, first of all, SAP is a very aggressive company. Secondly, most of the SAP executives that make these claims are not attempting to be accurate; they are attempting to hit sales goals.

We often receive negative feedback from SAP proponents on the research that we do. SAP proponents don’t seem to have an issue with SAP’s statement or prediction accuracy, but they do seem to have a problem with an independent entity that fact checks what SAP says.

In forecasting, there is a technical definition of bias. Bias is when the forecast is consistently positive or negative. Bias is not the same thing as forecast error. You can be wrong 100% of the time, but still not have a bias if your errors are both positive and negative. (We cover the topic of forecast bias in the article How to Best Understand Forecast Bias.

From the observations in this study, SAP has very close to the highest possible error and the highest possible bias. That is they nearly always overstate or have a positive spin on their projections.

Interestingly, we have been accused of being biased by SAP proponents. However, upon performing this research not only do we not display any bias, but we also barely display any error. In fact, you can’t have a bias if you have almost an error.

Forecast bias is an appealing criticism to make because it is a charge that can be leveled without bringing evidence. When we level the charge of bias, we provide the reason. The most common reason we provide is financial bias or in some cases both financial bias and contractual bias. Such as in the case with partners of SAP that are both financially incentivized to support SAP but also contractually limited in what they can write about SAP as per SAP’s partnership contract, as we cover in the article The Control on Display in the SAP Partnership Contract.

Secondly, in general argumentation such as on LinkedIn forms the term bias is often misused to mean a view which is not in agreement with the other debater. Also, the term bias is applied inaccurately as the term prejudiced. Being prejudiced is to prejudge something. It is not to judge something. The term prejudiced is inadequately applied when used as an ad hominem to describe when a person dislikes or disagrees something they have sufficiently tested or experienced to render a judgment. For example, if one writes a movie review without seeing the movie, then they prejudged the movie. But if they saw the entire movie, and then wrote an unflattering review, it cannot be said that they prejudged the movie. That is not an accurate criticism. They judged the movie.

Overall SAP proponents have a different definition of bias than the actual meaning. The definition of bias according to most SAP proponents is whether you agree with SAP. If you agree with SAP then you have no bias, if you do not agree with SAP, then you have a bias.

Jump Back to the Accuracy Table

  • References

    https://en.wikipedia.org/wiki/SAP_NetWeaver

    https://www.forbes.com/sites/sap/2011/11/15/sap-in-china-hasso-plattner-and-the-hana-revolution/#49f7a2204568

    https://www.cio.com/article/2418277/enterprise-software/sap-co-ceo-defends-price-of-sybase-deal.html

    http://www.businessinsider.com/analyst-says-saps-hana-database-may-not-be-as-successful-as-it-seems-2013-4

    https://www.cnet.com/news/salesforce-coms-benioff-bests-saps-plattner-in-debate/

    https://www.forbes.com/sites/benkepes/2014/07/28/on-sap-and-its-on-again-off-again-mid-market-focus/#46a6874e512c

    https://www.forbes.com/sites/ericsavitz/2012/08/29/sap-co-ceo-bill-mcdermott-live-from-the-churchill-club/#33ec2fd12d44

    https://data.worldbank.org/indicator/SL.TLF.TOTL.IN

    SAP Business ByDesign – alive and kicking

    https://cloudplatform.sap.com/capabilities/mobile/ios-sdk.html

    https://www.cio.com/article/3066748/mobile/apples-sap-deal-means-more-ios-enterprise-apps.html

    https://www.engineering.com/PLMERP/ArticleID/8529/Great-ERP-worse-PLM-What-SAP-PLM-needs-to-sharpen-its-competitive-edge.aspx

    SAP to Expand Cloud Presence With Acquisition of Ariba

    https://www.asug.com/news/asug-briefing-sap-ariba-seeks-to-combine-direct-indirect-spend-on-one-platform

    https://go.forrester.com/blogs/12-05-23-sap_buys_ariba_huh/

    https://news.sap.com/how-to-understand-the-business-benefits-of-s4hana-better/

Who is the Most Accurate Source on SAP?

Want to find out? See... A Study into The Accuracy of SAP