What This Article Covers
- The Accuracy Tracking Table
- Overall Accuracy Calculation and Future Accuracy Forecasting
- Research Background
- Partners in Inaccuracy?
- A Recognized Advantage Versus SAP
- Is Accuracy SAP’s Concern?
- Determining Source Quality
- Background on Accuracy Measurement
- Tracking Predictions and Outcomes
In most research, the topic is introduced and the background provided, and the data is provided second. However, this research actually serves two purposes. One purpose is to serve as a long-term resource for people and companies interested in fact-checking SAP. For this reason, we have placed the data first, and the explanation second.
- If you are here just for the fact checking, the table you wish to query is below.
- You do not need to type in the entire term, just type in the first few letters of the term you seek, and the list will begin to filter immediately.
- If this is your first time seeing this research, please click the Jump to Research Explanation link below to be taken to the full explanation of the research.
A second purpose is for those with more of an academic interest in SAP’s accuracy.
*Scan the list, or use the search box in the upper right above the table.
SAP Versus Brightwork Reseach & Analysis Accuracy
|SAP Topic Area (Prediction)||SAP's Accuracy Percentage||Brightwork's Accuracy Percentage||First Made by SAP||First Made by Brightwork||Link to Article with Full Explanation|
|Netweaver to Become Major Infrastructure "Product."||0%||100%||2003||2010||See Article Link|
|SAP to Host its Own Products and not Rely on AWS||0%||N/A||2015||N/A||See Article Link|
|SAP's Run Simple Marketing Program Reflects the Reality of SAP||0%||100%||2015||2016||See Article Link|
|HANA Will be an Perfect Fit for ERP Systems?||0%||100%||2011||2017||See Article Link|
|HANA Will Make Run MRP Faster than On Any Other Database||0%||100%||2015||2016||See Article Link|
|You Can Run S/4HANA on a Smartwatch||0%||100%||2015||2017||See Article Link|
|SAP Mobility is Number One in Mobility Due to Sybase Acquisition||0%||N/A||2010||N/A||See Article Link|
|HANA is the Fastest Growing Product in the History of Software||0%||100%||2013||2017||See Article Link|
|Fiori is Commercially Successful, Inevitable and Broadly Used by SAP Customers||0%||100%||2014||2016||See Article Link|
|With SAP Fiori and SAP Screen Personas, SAP Made Fiori Widely Available..etc..||0%||100%||2013||2017||See Article Link|
|Apple and SAP Partnership||1%||99%||2016||2017||See Article Link|
|SAP CRM 7.0 is the Best Product in the Field||0%||N/A||2008||N/A||See Article Link|
|SAP ByDesign Will Beat or Shock Netsuite with a 90 MPH Fastball||0%||N/A||2014||N/A||See Article Link|
|SAP PLM||10%||90%||2004||2009||See Article Link|
|SAP MDM||5%||95%||2005||2009||See Article Link|
|SAP SPP||5%||95%||2006||2010||See Article Link|
|SAP Will Have a Billion Users by 2015||0%||N/A||2012||N/A||See Article Link|
|HANA Will Run 100,000x Faster than Any Other Technology||0%||100%||2012||2016||See Article Link|
|HANA Will Help Employees Work from 10 to 10,000 Times Faster||0%||N/A||2014||N/A||See Article Link|
|Ariba is Useful for Direct Procurement||0%||100%||2012||2016||See Article Link|
|SAP to Introduce Revolutionary Health Application Based on HANA||0%||100%||2012||2017||See Article Link|
|S/4HANA Has a Simplified Data Model||0%||100%||2015||2017||See Article Link|
|SAP ASAP Methodology Sped Implementations||0%||100%||1997||2015||See Article Link|
|SAP Rapid Deployment Solution Sped Implementations||0%||100%||2012||2017||See Article Link|
|S/4HANA is Complete||0%||100%||2015||2017||See Article Link|
|Design Thinking Speeds SAP Implementations||0%||100%||2013||2017||See Article Link|
|Best Practices are Contained within SAP||0%||100%||1990's||2010||See Article Link|
|The SNP Optimizer Provides Good Supply Planning Output||0%||100%||1999||2012||See Article Link|
|The PP/DS Optimizer Provides Good Production Planing and Scheduling Output||0%||100%||1999||2010||See Article Link|
|Solution Manager Would Become "the" CMS for SAP Implementations||0%||100%||2001||2012||See Article Link|
|SAP is Finished on Oracle||0%||100%||2015||2017||See Article Link|
|SAP's S/4HANA Implementation Numbers||20%||80%||2015||2017||See Article Link|
|Indirect Access (Type 2) is a Legitimate Claim||0%||100%||2014||2017||See Article Link|
|Fiori is the Best UI in Enterprise Software||0%||100%||2013||2017||See Article Link|
|ByDesign to Run on HANA||0%||100%||2017||2017||See Article Link|
|SAP's ERP Would be The Only System a Company Would Ever Purchase||0%||100%||Since SAP's beginning||2014||See Article Link|
|All of an SAP Prospect's Existing Applications are Legacy||0%||100%||Since SAP's beginning||2017||See Article Link|
|All Non HANA Databases are Legacy||0%||100%||2014||2017||See Article Link|
|S/4HANA is so Different from ECC That it is Not Covered as a Normal Free Upgrade||0%||100%||2014||2016||See Article Link|
|SAP Simplifies The Customer's IT Landscape||0%||100%||Since SAP's beginning||2014||See Article Link|
|SAP's Integrated Suite Offers a Lower TCO||0%||100%||Since SAP's beginning||2014||See Article Link|
|Reengineering Business Processes to Match SAP Leads to Long Term Competitive Advantages||0%||100%||1980s||2017||See Article Link|
|SAP Support is a Good Value||0%||100%||Since SAP's beginning||2017||See Article Link|
|In Memory Computing (via HANA) is Something New and Different||0%||100%||2011||2017||See Article Link|
|The Count of Fiori Apps is High||0%||100%||2015||2017||See Article Link|
|Hasso Plattner and his PhDs Invented HANA||0%||100%||2011||2017||See Article Link|
|HANA Has a Lower TCO||0%||100%||2014||2017||See Article Link|
|Shelf Life Planning Works in APO SNP||0%||100%||2000||2010||See Article Link|
|Parallel Processing Works with the SNP Optimizer||0%||100%||2000||2013||See Article Link|
|SAP on Alliance with Google||0%||100%||2015||2018||See Article Link|
|S/4HANA in the Cloud Can be Easily Extended||0%||100%||2016||2018||See Article Link|
|Only SAP HANA is an ACID Database||0%||100%||2017 (est)||2018||See Article Link|
|All Non HANA DBs are Legacy||0%||100%||2017 (est)||2018||See Article Link|
|Is it SAP Leonardo that Ensures Frozen Ice Cream Delivery?||0%||100%||2017 (est)||2018||See Article Link|
|SAP to Open S/4HANA to AnyDB||N/A||TBD||N/A||2017||See Article Link|
|Fiori Will Not Survive (To be replaced by new UI initiative)||N/A||TBD||N/A||2017||See Article Link|
|Sybase Database Will be a Good Acquisition||0%||100%||2010||2012||See Article Link|
*Any cell with N/A means that either SAP or Brightwork did not make a statement about the line item.
Overall Accuracy Calculation and Future Accuracy Forecasting
The following is a straight average of all of the items listed in the table. This is a record of statements and predictions made over decades. In statistical forecasting, the historical forecast error is typically used to produce the estimate of what the future forecast error.
Following this precedent, it should be considered a useful accuracy percentage to apply to future predictions by SAP.
What should be clear is that SAP has a long-term issue with accuracy. This is the only published study into SAP’s accuracy. It is not generally understood how low SAP’s accuracy is by those that work in SAP or even those vendors that compete with SAP.
A Recognized Advantage Versus SAP
Brightwork had the advantage of making observations years, in some cases decades after SAP initially made its statements/predictions. However, Brightwork performed well even for recent SAP projections. Brightwork has made several predictions which contradict both SAP and the overall market. At this point, not enough time has passed to determine if these predictions can be considered to have come true. Therefore they are marked as TBD.
To be accurate, any entity must value accuracy over other objectives. It should be remembered that companies are normally said to maximize profits, not to maximize the accuracy of their statements to customers. SAP’s score illustrates the extremes this can take.
Is Accuracy SAP’s Concern?
SAP is trying to drive revenues and is very obviously not attempting to be accurate. Individuals within SAP may or may not believe what they are saying. The study is not engaging in an analysis of what statements SAP representatives know are false when they say them. It only analyzes if the statement turned out to be true.
For obvious reasons neither SAP nor the massive army of companies and people that make money from SAP and cover up for them are not fans of the research, we do at Brightwork Research & Analysis.
We have been accused by SAP and SAP IT departments of the following:
- Having an ax to grind (which we responded to in this article).
- Of biting the hand that fed us (which is the opposite of the previous argument, which is having an ax to grind). This argument (biting the hand that fed us) we agree with, but it does not make our research untrue, merely disloyal. We guarantee no loyalty to any software vendor or to any consulting company or entity, whether we have financially benefited from them or not.
SAP’s inaccuracies are repeated through a network of entities that do not care about accuracy much more than does SAP. This topic is the subject of future research, with IT media entities also being scored for their accuracy on SAP.
Partners in Inaccuracy?
What SAP and other SAP proponents are careful not to do is to address the conclusions of our research and analysis. SAP prefers to direct people to sources of information like
- CIO Magazine
- Gartner and other media entities and IT analysts that SAP pays.
- Or, to Deloitte and Accenture and other SAP consulting partners, that is entities that have huge businesses built around billing for SAP consulting.
Therefore, according to who SAP recommends that their customers that listen to, the universal feature is that the sources in some way have a financial tie to SAP, which we covered in Which Sources Does SAP Consider Credible?
SAP seems to have a precise definition of what is a credible, source and it seems to line up very carefully with sources that unthinkingly repeat what they say, and where money is involved for the entity that obeys SAP.
So how is a person trying to get the straight story to determine who is credible?
Determining Source Quality
In our view, a compelling way to determine source quality is to check their track record. Therefore, in this article, we will compare SAP’s track record to our own to see which is more reliable.
Background on Accuracy Measurement
We provide a relative accuracy measurement as part of our Honest Vendor Ratings.
We think that this type of measurement of accuracy is essential to decision making in IT. In fact, our book Rethinking Enterprise Software Risk is highly based on evaluating the accuracy of sources.
However, this is the first time we measured, calculated and documented a vendor’s accuracy in anything like this level of detail. It is a lot of work, and we don’t know anyone who does it. First of all, to do something like this, you would need to have financial independence from the source, so that rules out virtually all media entities from doing anything like this. Secondly, there is no way of monetizing something like this. Media and IT analysts primarily make their money by promoting software vendors, and you can’t collect money from vendors by challenging their assertions. It also tends to make a lot of people angry at you to boot.
Tracking Predictions and Outcomes
SAP’s and our predictions can be easily tracked through the published articles where they and we make the prediction. The determination of whether that prediction is true is a bit more tricky. But in most cases, we have informative articles that explain what the eventual outcome was.
Another trick in measurement is that some predictions, for instance, that Brightwork or SAP made, have not come true yet. But it does not mean they will not come true. In those instances, we don’t give a complete rating, but instead, explain the current state of the prediction.
Finally, some predictions cannot be declared as true or false, that is they are not binary. Some predictions worked out to some degree. In many cases, it makes the most sense to provide a percentage.
Here is how we scored the outcomes of predictions.
- A zero percent rating would mean that the prediction did not come true, or the exact opposite happened.
- A 75% rating is that the prediction mostly came true.
- A 100% rating means that the prediction entirely came true.
This is not a “perfect science,” but (unless you work for SAP or an SAP surrogate) we think the readers will find that this is entirely a beneficial exercise. One might question the entity rating themselves against SAP and the major entities that provide information about SAP. However, the results were so noticeable that we have every incentive to grade ourselves harshly.
Something we also have planned in this area is to measure the accuracy of the statements of IT media entities concerning SAP.
There you have it. So given this comparison Brightwork, Research & Analysis beats SAP by a substantial margin in predicting SAP.
The first observation is that SAP has a very low accuracy level for its statements. Almost none of the projections that SAP makes come true.
How are we able to have such a high accuracy level versus SAP?
Well, first of all, SAP is a very aggressive company. Secondly, most of the SAP executives that make these claims are not attempting to be accurate; they are attempting to hit sales goals.
We often receive negative feedback from SAP proponents on the research that we do. SAP proponents don’t seem to have an issue with SAP’s statement or prediction accuracy, but they do seem to have a problem with an independent entity that fact checks what SAP says.
In forecasting, there is a technical definition of bias. Bias is when the forecast is consistently positive or negative. Bias is not the same thing as forecast error. You can be wrong 100% of the time, but still not have a bias if your errors are both positive and negative. (We cover the topic of forecast bias in the article How to Best Understand Forecast Bias.
From the observations in this study, SAP has very close to the highest possible error and the highest possible bias. That is they nearly always overstate or have a positive spin on their projections.
Interestingly, we have been accused of being biased by SAP proponents. However, upon performing this research not only do we not display any bias, but we also barely display any error. In fact, you can’t have a bias if you have almost an error.
Forecast bias is an appealing criticism to make because it is a charge that can be leveled without bringing evidence. When we level the charge of bias, we provide the reason. The most common reason we provide is financial bias or in some cases both financial bias and contractual bias. Such as in the case with partners of SAP that are both financially incentivized to support SAP but also contractually limited in what they can write about SAP as per SAP’s partnership contract, as we cover in the article The Control on Display in the SAP Partnership Contract.
Secondly, in general argumentation such as on LinkedIn forms the term bias is often misused to mean a view which is not in agreement with the other debater. Also, the term bias is applied inaccurately as the term prejudiced. Being prejudiced is to prejudge something. It is not to judge something. The term prejudiced is inadequately applied when used as an ad hominem to describe when a person dislikes or disagrees something they have sufficiently tested or experienced to render a judgment. For example, if one writes a movie review without seeing the movie, then they prejudged the movie. But if they saw the entire movie, and then wrote an unflattering review, it cannot be said that they prejudged the movie. That is not an accurate criticism. They judged the movie.
Overall SAP proponents have a different definition of bias than the actual meaning. The definition of bias according to most SAP proponents is whether you agree with SAP. If you agree with SAP then you have no bias, if you do not agree with SAP, then you have a bias.
Who is the Most Accurate Source on SAP?
Want to find out? See... A Study into The Accuracy of SAP