Fill Rate Versus Backorder as a Service Measurement

by Shaun Snapp on October 4, 2011

Background

The vast majority of articles on this website that discuss service level tend to focus on fill rate, as this is the most popular service level measurement method in business. However, the majority of early work on inventory optimization and multi echelon planning that began in the late 1950s, and now drives the best of breed service parts planning software applications, was in fact designed around backorders. This is because the research was primarily paid for by the Air Force and carried out by the RAND Corporation, and the focus was squarely on solving the problem of managing military service parts networks. Therefore, it is interesting to compare and contrast two quotations from research papers that focused on minimizing backorders. This first is from Craig Sherbrooke and his METRIC (an acronym for Multi-echelon Technique for Recoverable Item Control) paper written in 1966. This is how Sherbrooke explains his use of backorders over fill rates in in his paper.

Fill rate — defined as the fraction of demands that are immediately fulfilled by supply when the requisitions are received — concentrates nearly all stock at the bases. The result is that when a non fill occurs, the backorder lasts a very long time. Similarly fill rates behaves improperly in allocating investment at a base when the item repair times are substantially different. Consider two items with identical characteristics except that one is base-reparable in a short time, and the other is depot reparable with a much longer repair time. Assume that our investment constraint allows us to purchase only one unit of stock. In that case, the fill rate criterion will select the first item, and the backorder criterion the second.

The fill rate possesses an additional defect. A fill is normally defined as the satisfaction of a demand when placed. But if we allow a time interval T to elapse, such as a couple of days, on the grounds that some delay is acceptable, the policy begins to look substantially different. As longer delays are explored, the policy begins to resemble the minimization of expected backorders.

In summary, the backorder criterion seems to be the most reasonable. The penalty should depend on the length of the backorder and the number of backorders; linearity is the simplest assumption. This is the criterion function most often employed in inventory models. - Craig Sherbrooke

Sherbrooke explains that he considers backorders superior for his purposes due to the following:

  • Fill rates tend to concentrate stock at the bases (bases in Sherbrooke’s papers would correlate to DCs in industry-speak, with the depot being the regional DC or (RDC))
  • Fill rates measure the satisfaction only at the point of initial delay, and do not measure how late a fulfillment actually occurs.

Therefore Sherbrooke designed an algorithm as part of METRIC a penalty which multiplies the length of the backorder by the number of backorders.

Leanard Laforteza states a similar reasoning in his paper for selecting backorders as a measurement for his paper designing a multi echelon system for supplying Marine military deployments.

Fill rate is the percentage of demands that can be met at the time they are placed, while backorders are the number of unfilled demands that exist at a point in time. In commercial retail, if customer demand cannot be satisfied, a customer either goes away or returns at a later time when the item is restocked. the first case can be classified as lost sales while the second case creates a backorder on the supplier or manufacturer. In military applications, especially in most critical equipment, any demand that is not met is backordered. The backorder is outstanding until a resupply for the item is received, or a failed item is fixed and made available for issue. These two principle measures of item performance – fill rate and backorders – are related, but very different. Commercial retailers are more interested in fill rate than in backorders because fill rate measures customer satisfaction at the time each demand is placed. Not only is fill rate easy to calculate, but it also helps retialers form a picture of how well they are meeting customer demand. Experience may tell them that a 90% fill rate on an item is not acceptable and will create customer complaints. On the other hand, backorders are not easy to compute as fill rate. Unlike commercial retail business, the military is not concerned with lost sales. The military measures performance not in terms of sales, but in terms of equipment availability.

In terms of supply support measurement, we recommend tracking backorders. Although fill rate tends to have clearer meaning to commercial suppliers, the rate does not have the same meaning in militar applications. Using the concept of backorders, a unit can determine the status of its supply support not just when the order is placed, but up to the time the item was received. - Leanard D. Laforteza

Here Laforteza does a good job explaining why backorders are more relevant for military application than fill rates. However, as the greater market for MEIO applications is civilian, vendors added fill rates and fill rates are not the dominant method of MEIO implementation. MCA Solutions, a service parts planning vendor with a substantial military client base can measure service level by fill rate or by availability (i.e. the uptime of equipment). However, while it does not measure fill rate by backorder as do Sherbrook’s METRIC or Laforteza’s approach, MCA allows for the flexible setting of backordering for different locations. MCA allows for the following settings:

  1. All locations to be backorderable
  2. Only the root locations to be backorderable
  3. No locations to be backorderable (which is the default).

MCA describes its management of backorders in the following way:

A Location is called backorderable if the unmet demand at that Location gets backordered at that Location and waits until the inventory is available at that Location. A Location is not backorderable (also referred to as lost-sales) if the unmet demand is passed to another Location or outside the supply chain. In backorderable models, preference is given to destinations that do not have enough inventory position to meet their child Location needs. - MCA Solutions

Conclusion

The service level measurement must fit the application. The early MEIO research papers were centered around military application, and thus used backorders, and backorder which is often the number of backorders times the average backorder duration serves as a common service level measure. However, civilian applications require fill rate as the service level measure.

References

“METRIC: A Multi-echelon Technique for Recoverable Item Control,” C.C. Sherbrooke, RAND Corporation, 1966

“Inventory Optimization of Class IX Supply Blocks for Deploying in U.S. Marine Corps Combat Service Support Elements,” Leanard D. Laforteza, Naval Postgraduate School Monterey, California, June 1997

Principles of Operation, MCA Solutions, 2007

{ 0 comments }

Background and Motivation for the Research

When I am often told about the reasons for decisions to go with software that I am familiar with, the logic often does not seem to make sense. In fact the entire process for repeatedly selecting expensive and lower functionality software from the major monopoly vendors turns out not to be based on the main comparison points of software. The main comparison points of enterprise software are the TCO and the application’s functionality. However, companies primarily look for solutions from vendors that they are already working with, and then allow the issue of integration to play a primary decision-making role. Therefore, they primarily ignore TCO (most tend to make decisions without knowing the estimated TCO), focusing more on initial software acquisition cost, and de-emphasize the functionality comparison between applications. A primary way this is done is by having executives, who don’t work with enterprise applications perform the functionality observation through a controlled demo environment and by marginalizing the users, removing them from the decision-making process. This allows applications that are weak or have unreliable functionality to compete with vendors that have excellent and reliable functionality.

The Basis for Estimation

I visit clients often post go-live on SAP APO and have developed a good sample of companies. I know the typical length of an APO implementation, as well the costs of maintaining APO. I also work with a number of best-of-breed vendors. Because I had access to information from several necessary sources, and was able to make times estimations based upon personal experience, I decided to perform a total cost analysis between SPP and a best-of-breed service parts planning vendor. This is just the service parts planning analysis. Here are links to the others.

http://www.scmfocus.com/productionplanningandscheduling/2011/08/09/what-if-you-paid-nothing-for-sap-software-how-saps-tco-compares-for-production-planning/

The Scope of the Analysis

This analysis is limited to the major planning applications. I have developed estimates for costs of APO modules versus best-of-breed applications for the areas which I have first-hand knowledge, which is demand planning, supply planning, service parts planning and production planning and scheduling. I do not perform any similar analysis for other popular enterprise areas such as ERP or analytics.

Why the Best of  Breed Vendor is Not Named

I am not trying to recommend any one vendor in this analysis, so naming the vendor I used would be a distraction. The main point is that SAP’s TCO is in an entirely different cost category. Essentially any best-of-breed vendor I selected would generally compare similarly. Some will be a bit more expensive and some a bit less so, but no best of breed vendor will come anywhere close to SAP’s TCO.

Why SAP License Costs are Set to Zero

SAP license costs are difficult to determine. There is little doubt they have some of the highest average license costs in the enterprise market, but their price fluctuates greatly. In addition, the costs may be bundled with other software. In terms of publicly available rates, SAP has a government price sheet. However, the price sheet is based on an arcane point system that is clearly designed to not allow anyone to independently calculate a price, while meeting the US government requirement that they have a price sheet. I worked with this sheet for around an hour and a half, and then realized, it was not meant to be deciphered. SAP license costs are shrouded in mystery.

However, when I performed the analysis, even without SAP license costs, I found SAP TCO costs to be so high that even without any license costs or SAP support costs (which are based upon the license costs) the best of breed vendors were still easily beating SAP in TCO in all the application areas. Secondly, any article which does not rank SAP as #1 in whatever it is being compared with is open to immediate criticism. (In fact, the easiest way to have a soft life in IT is to skip any analysis and declare SAP the victor. In doing this, you generally are not required to provide any evidence, but simply say something like “SAP support best practices.”) So something that shows SAP’s TCO being higher than anything else will be considered biased. Therefore, to counteract this concern, I decided to tilt the playing field in SAP’s direction by making all of the license costs free. So this analysis assumes you never had to pay anything for SAP’s software or their support.

Doing this does one other thing, it emphasizes the point that the license cost should not be the main focus of the comparison and that other costs predominated in the TCO. Therefore, free software can end up being not the best decision.

Analysis Assumptions

There are a number of assumptions in this analysis. One of the most important is the duration of the implementation. This is one of the trickier things to set. Software companies tend to deemphasize this number, which is why I had to use my experience to adjust the results to what I have seen. SAP implementations take the longest of any enterprise vendor, and there are very good reasons for this, which I get into later in this post. However, for both SAP and the best-of-breed vendor, I have included a range, and the estimated TCO for each in terms of implementation is based upon an average. There is no perfect analysis of this type that can be created because of all the different variables. However, not being able to attain perfection should not get in the way of attempting estimation. One way or another, these types of analyses must be performed and I always think it’s better to take a shot at estimation rather than to throw one’s hands up and say its unknowable.

Total Cost of Ownership

According to this estimate, SAP has a higher total cost of ownership than the best of breed application I compared it against. Having worked in SAP as long as I have, I intuitively I knew it would be higher, but even I was surprised by how much higher it was. Here are some of the reasons.

SAP’s Implementations take Significantly Longer than Best of Breed Implementations

  1. SAP’s software is very difficult to understand and is highly encapsulated. SAP has so many settings which allow the system to behave in different ways that extensive time must be spend in both understanding the settings and understanding the interactions between the settings. The statement that SAP is filled with “best practices,” is actually incorrect, because a best practice approach prescribes that the system define specific ways of doing things, when in fact SAP follows the “comprehensive approach.” This includes a seemingly unlimited number of ways of configuring the system.
  2. Of all the applications I work with, none approach SAP in the number of areas of their applications that don’t work. This includes functionality that never worked, beta functionality that is still listed in the release notes as functional, and functionality that did work at one time but was broken by an upgrade or other cross application factor. In fact no one is even comes close. SAP’s marketing strategy is to cover functionality as broadly as possible so they can always say “we have it.” This same development approach spans across applications, as I observe the same thing in different product lines such as SAP BW. This is one reason SAP’s TCO is probably headed further up in the future. However, this results in product management writing checks that development cannot cash. Testing each area of functionality to ensure (part of what I do by the way) imposes more work and more time on the implementation.
  3. The large consulting companies have built their business model around SAP and extend the time of SAP implementations to maximizes their billing hours. SAP made a strategic decision quite some time ago to let the consulting companies control the speed of implementation in order to be recommended by the major consulting firms, regardless of the fit between the application and the client need.

SAP Resources Are Some of the Most Expensive in IT

  1. There is nothing controversial about this statement, it is well known in IT circles.

SAP Has the Highest Manpower Support Requirement

  1. Getting back to the topic of application complexity and fragility, SAP simply takes more resources to maintain. Something I recently had to work with was one method which was part of functionality that did work, but stopped working as of the release SCM 7.0. First the problem that cropped up due to this needed to be diagnosed and explained (we did not find out about the broken functionality but perceived it through system problems. Once discovered, this functionality had to be change to a method that did work, and the business had to invest time creating a new policy to work with the changed functionality. This was course expensive and time-consuming.

SPP in Particular

Of the four different planning areas that I have created TCO comparisons for, SPP is a special case as it is an immature product with significant needs for on site development. For this reason I have given it the longest implementation timeline of any SAP planning product. It also estimate its support load to be higher than the other SAP planning products because of maturity issues combined with the extra effort to support the custom development that must accompany any SPP project.

Secondly, SPP projects are very risky due to SPP’s lack of maturity. Therefore, the long timeline included here does not include the likelihood of walking away from the implementation at any time during the project.

Integration is Overrated as a Cost

The cost differences between SAP and a best of breed application are enormous, and the frequently used argument, that the company wants an integrated solution, cannot reasonably be used to justify a decision to select SAP. I have not broken out the integration separately, as it is built into the consulting costs, but an adapter of even a few hundred thousand dollars would not tip the TCO in SAP favor. Also, the maintenance of the SAP CIF (the middleware that connects R/3 to APO) is vastly underrated. My experience and with developing custom adapters for connecting best of breed planning applications to SAP, I have become firmly convinced that the cost of maintaining the CIF is more than the cost of developing and maintaining a custom adapter. The CIF, which connects up APO to SAP ERP is unacceptably problematic. For more on the CIF, see this post.

http://www.scmfocus.com/sapplanning/2011/05/19/why-i-no-longer-recommend-using-the-cif/

Implication for ROI

According to most publicly available studies, around 1/2 of projects have a positive return on investment. However, this greatly depends upon the TCO of the solution and the functionality within the application that can be leveraged. SAP planning modules are so expensive compared to alternative solutions, and deliver a lower functionality level than best-of-breed solution, that as a natural consequence they have a lower ROI, and a lower percentage of positive ROI projects. However, the incorrect perception in industry is just the opposite, that SAP is the safe vendor to choose.

Outsourced Support to Reduce Costs?

Companies now often outsource a portion of their support to India, so one might imagine that the support costs listed here could be reduced. This is another frequently held assumption, but does not prove out in reality. A good rule of thumb is that while India based resource are about 1/4rth as expensive it takes more than twice as many individuals to get close to the same amount of support work done. Secondly, there must always be at least one in country resource. Thirdly, this is a mess to manage. There are not only language and time barriers, but it appears some of the companies providing these resources are actually double book the same resource on multiple clients. I have been dealing with this issue for several years now and I end up having to read notes from the support team which are not spelled properly because of language barriers. Outsource operations lack good professional management, and the client resources end up having to take over support organization tasks.

Generally, I am not sure outsourced support works for any area very well, but it particularly unsuited to complex systems such as planning applications. Generally, when support is outsourced, the quality of support drops precipitously, and anyone in IT knows this.

Conclusion

If you confront SAP and large consulting firms to require a good TCO analysis, be prepared for a dispute on the true cost of their software and time required to go live. However, its critical to make your decisions based on actual observations at multiple account, as I have in this article, and not based on hypothetically sales estimates from their sales team on how fast a solution can be brought live. I have done the best job possible here to bring the real world data to my estimates, and I even stacked the deck for SAP by removing all license costs, but SAP still came up with a much higher TCO.

By the way, this was also true in the other application areas I analyzed. The real world data shows across the board that SAP is significantly more expensive in total costs of ownership than best-of-breed solutions.

 

 

 

{ 2 comments }

Understanding the (S, s-1) Inventory Policy

December 31, 2010

Background In several articles distributed across different SCM Focus sites, the (S,s-1) inventory policy is discussed. For this reason, it made sense to create an article which explains what it does. The following was written by Wayne Fu. _________________________________ In Muckstadt’s book, section 1.1.2, it explains briefly about the (s,S) policy. s is normally the [...]

Read the full article →

Important Service Part Multi Echelon Inventory Optimization Books

December 31, 2010

Background It is very interesting to find out about the origins of service parts inventory optimization. It’s rarely discussed, though I thought I would write and article on this topic. Guest Co-Author For this post we have a guest co-author. Wayne Fu, a Senior Product Manager at Servigistics provides a perspective on what he considers [...]

Read the full article →

Heuristic Based Algorithms Explained

December 30, 2010

Background This post documents an email discussion between myself and Wayne Fu regarding heuristic based algorithms. Question for Wayne Fu What is a heuristic based optimization algorithm? I thought that heuristics were one form of problem solving, and optimization was another. How is a heuristic based algorithm is different from a non-heuristic based algorithm? That [...]

Read the full article →

Deloitte Writes “Ok” Paper on Service Parts But Would You Want to Hire Them?

November 13, 2010

Background The paper The Service Revolution by Deloitte “research” is an average white paper which has some interesting numbers about service parts along with quite a lot of fluff to reach its 13 pages. I would recommend it to skim rather than read. It reminded me that I recently wrote an article on the low [...]

Read the full article →

Eric Larkin from Arena Solutions on BOM Management for the Service Market

October 29, 2010

Introduction Eric Larkin, Chief Technology Officer and co-founder or Arena Solutions, a very innovative and powerful BOM (bill of material) management application, describes how Arena Solutions is rolled out on service parts accounts. He describes some of the differences between finished goods implementations of Arena Solutions vs. service part implementations. ____________________________________________ Title: Eric Larkin from Arena [...]

Read the full article →

Microsoft Dynamics AX and Service Management

September 21, 2010

Background Much of the Dynamics AX functionality for supply chain is rather basic. However, there is one supply chain area area which stands out. This is the service management functionality. As Dynamics AX could be connected to a service parts planning engine, I thought it would be interesting to go through some of the screens [...]

Read the full article →

Interviews with Tim Andreae SVP of MCA Solutions

September 18, 2010

Introduction These interviews are with Tim Andreae, SVP of Marketing for MCA Solutions. Tim has extensive experience in strategic work and has been with MCA for over 8 years. His longevity in the planning and service parts space makes these interviews particularly relevant. In these interviews I asked Tim both about the history of MCA [...]

Read the full article →

Understanding SAP The Service Parts Maintenance Solution

September 4, 2010

Service Parts Maintenance is an SAP Multi Module Solution Background I had heard the term Service Parts Maintenance (SPM) in relation to SPP, however, I never paid it very much attention. However, upon re-reading the SPP training documentation I found a very detailed explanation as to what it is. According to the SAP documentation SPM [...]

Read the full article →