June 2014

exponential distributionHave you heard that data must be normally distributed before you can plot the data using a control chart?  Quite often you hear this when talking about an individuals control chart.  This is a myth.  Data do not have to be normally distributed before a control chart can be used – including the individuals control chart.  But, you better not ignore the distribution in deciding how to interpret the control chart.  This month’s publication examines how to handle non-normal data on a control chart – from just plotting the data as “usual”, to transforming the data, and to distribution fitting. 

Not all data are normally distributed.  There are many naturally occurring distributions.  For example, the exponential distribution is often used to describe the time it takes to answer a telephone inquiry, how long a customer has to wait in line to be served or the time to failure for a component with a constant failure rate.  These types of data have many short time periods with occasional long time periods.  These data are not described by a normal distribution. 

So, how can you handle these types of data?  This publication examines four ways you can handle the non-normal data using data from an exponential distribution as an example.   In this issue:

Exponential Example Data

greeet customerTo examine the impact of non-normal data on control charts, 100 random numbers were generated for an exponential distribution with a scale = 1.5.  The scale is what determines the shape of the exponential distribution.  Maybe these data describe how long it takes for a customer to be greeted in a store.   Usually a customer is greeted very quickly. The data are shown in Table 1.

Table 1: Exponential Data

2.262.684.170.032.02
7.770.134.050.045.28
3.671.375.120.210.03
0.910.652.242.670.75
0.110.130.530.60.43
0.360.140.292.951.53
0.061.352.090.542.22
0.311.462.823.540.19
0.910.011.243.430.75
1.010.181.032.652.99
3.211.980.51.70.3
0.240.822.020.162.41
3.841.770.860.162.07
2.282.490.514.061.31
1.750.532.172.041
1.450.40.113.562.15
1.811.670.86.11.3
0.31.023.630.775.25
0.630.810.60.872.44
2.220.150.134.740.76

 

The histogram of the data is shown in Figure 1.  It is definitely not normally distributed.  A normal distribution would be that bell-shaped curve you are familiar with.  The high point on a normal distribution is the average and the distribution is symmetrical around that average. 

Figure 1: Histogram of Exponential Data

non_normal histogram

That is not the case with this distribution.  It is skewed towards zero.  The high point on the distribution is not the average and it is not symmetrical about the average.  For more information on how to construct and interpret a histogram, please see our two part publication on histograms.

From Figure 1, you can visually see that the data are not normally distributed.  You can also construct a normal probability plot to test a distribution for normality.   The normal probability plot for the data is shown in Figure 2.   The assumption is that the data follows a normal distribution.  If this is true, the data should fall on a straight line.  It is easy to see from Figure 2 that the data do not fall on a straight line.  So, again, you conclude that the data are not normally distributed. 

Figure 2: Normal Probability Plot of Exponential Data Set

normal probability plot

So, now what? How can we use control charts with these types of data?  What are our options?  Basically, there are four options to consider:

  • fourUse the individuals control chart
  • X-R control chart
  • Transform the data to a normal distribution and use either an individuals control chart or the X-R control chart
  • Use a non-normal control chart

If you had to guess which approach is best right now, what would you say?  You are right!  Actually, all four methods will work to one degree or another as you will see.

Individuals Control Chart

The first control chart we will try is the individuals control chart.  With this type of chart, you are plotting each individual result on the X control chart and the moving range between consecutive values on the moving range control chart.

The X control chart for the data is shown in Figure 3.  Since the data cannot be less than 0, the lower control limit is not shown. 

Figure 3: X Control Chart for Exponential Data

individuals x chart for non-normal data

The UCL is 5.607 with an average of 1.658.  The two lines between the average and UCL represent the one and two sigma lines.  These are used to help with the zones tests for out of control points.  Only one line is shown below the average since the LCL is less than zero.  For more information, please see our publication on how to interpret control charts

special causeThe red points represent out of control points.  Note that there are two points beyond the UCL.  In addition, there are two runs of 7 in a row below the average.  In addition, there is one spot where there are 4 points in a row in zone B (this one is also below the average) and one spot where there are two out of three consecutive points in zone A (this one is above the average). 

If you look back at the histogram, it is not surprising that you get runs of 7 or more below the average – after all, the distribution is skewed that direction.  The conclusion here is that if you are plotting non-normal data on an individual control chart, do not apply the zones tests.  These tests are designed for a normal (or at least a somewhat symmetrical) distribution.  Using them with these data create false signals of problems.

is it a signal?Removing the zones tests leaves two points that are above the UCL – out of control points.  With our knowledge of variation, we would assume there is a special cause that occurred to create these high values.  Are these false signals?  Remember, you cannot assign a probability to a point being due to a special cause or not – regardless of the data distribution.  So, are they false signals? In the real world, you don’t know.  But wouldn’t you want to investigate what generated these high values? 

Figure 4 shows the moving range for these data.  Not surprisingly, there are a few out of control points associated with the “large” values in the data.

Figure 4: Moving Range Control Chart for Exponential Data

moving range chart non-normal data

The amazing thing is that the individuals control chart can handle the heavily skewed data so well - only two “out of control” points out of 100 points on the X chart.  This demonstrates how robust the moving range is at defining the variation.  The +/- three sigma limits work for a wide variety of distributions.  

X-R Chart and the Central Limit Theorem

Perhaps you have heard that the X-R control chart works because of the central limit theorem.  Another myth. The central limit theorem simply says that the distribution of subgroup averages will be approximately normal – regardless of the underlying distribution as the subgroup size increases. 

Suppose we decide to form subgroups of five and use the  X-R control chart.  Remember that in forming subgroups, you need to consider rational subgrouping.  This is a key to using all control charts.   But, for now, we will ignore rational subgrouping and form subgroups of size 5. Figure 5 shows the X control chart for the subgrouped data (we will skip showing the R control chart)

Figure 5: X-R Control Chart for Exponential Data

xbar chart for non-normal data

Note that this chart is in statistical control.  In addition, there are no false signals based on runs below the average (note: with a larger data set, there probably would be some false signals).  Subgrouping the data did remove the out of control points seen on the X control chart.  So, this is an option to use with non-normal data.  But, you have to have a rational method of subgrouping the data.

Transform the Data

butterfliesAnother approach to handling non-normally distributed data is to transform the data into a normal distribution.  For example, you can use the Box-Cox transformation to attempt to transform the data.   The data were transformed using the Box-Cox transformation. The rounded value of lambda for the exponential data is 0.25.  This means that you transform the data by transforming each X value by X2.5.  The X control chart based on the tranform data is shown in Figure 6. 

This control chart does still have out of control points based on the zone tests, but there are no points beyond the control limits.  So, transforming the data does help “normalize” the data. The biggest drawback to this approach is that the values of the original data are lost due the transformation.  You cannot easily look at the chart and figure out what the values are for the process. 

Figure 6: X Control Chart Based on Box-Cox Transformation

x chart for box-cox transformatin of non-norma data

Non-Normal Control Chart

The fourth option is to develop a control chart based on the distribution itself.  This entails finding out what type of distribution the data follows.  Beware of simply fitting the data to a large number of distributions and picking the “best” one.  You need to understand your process well enough to decide if the distribution makes sense.  Then you have to estimate the parameters of the distribution.

We are using the exponential distribution in this example with a scale = 1.5.  The control limits are found based on the same probability as a normal distribution.  So, the LCL and UCL are set at the 0.00135 and 0.99865 percentiles for the distribution.  For the exponential distribution, this gives LCL = .002 and UCL = 0.99865 (for a scale factor = 1.5).  The only test that easily applies for this type of chart is points beyond the limits. 

The exponential control chart for these data is shown in Figure 7.  All the data are within the control limits.  The process appears to be consistent and predictable.  This type of control chart looks a little “different.”  The main difference is that the control limits are not equidistant from the average.  

Figure 7: Exponential X Control Chart

exponential control chart

There is nothing wrong with using this approach.  It does take some calculations to get the control chart.  But with today’s software, it is relatively painless.

Summary

This publication looked at four ways to handle non-normal data on control charts:

Individuals control chart: This is the simplest thing to do, but beware of using the zones tests with non-normal data as it increases the chances for false signals.  The +/- three sigma control limits encompass most of the data.  And those few points that may be beyond the control limits – they may well be due to special causes.  But then again, they may not.  Probably still worth looking at what happened in those situations.

 X-R control chart: This involves forming subgroups as subgroup averages tend to be normally distributed.  You need to have a rational method of subgrouping the data, but it is one way of reducing potential false signals from non-normal data.

Transform the data: This involves attempting to transform the data into a normal distribution.  This approach will also reduce potential false signals, but you lose the original form of the data.  No one understands what the control chart with the transformed data is telling them except whether it is in or out of control.

Non-normal control chart: This involves finding the distribution, making sure it makes sense for your process, estimating the parameters of the distribution and determining the control limits.  This approach works and maintains the original data.  But it does take more work to develop – even with today’s software.

So, looking for a recommendation?  Stay with the individuals control chart for non-normal data.  Simple and easy to use.   Don’t use the zones tests in this case.  If the individuals control chart fails (a rare case), move to the non-normal control chart based on the underlying distribution.  There is nothing wrong with this approach.  Only subgroup the data if there is a way of rationally subgrouping the data.  Stay away from transforming the data simply because you lose the underlying data. 

Upcoming Release of SPC for Excel Version 5

SPC for Exce LogoWe are preparing to release version 5 of our SPC for Excel software.  This new version is packed with new techniques.   These include:

  • 26 different charting options to monitor your processes
  • Multiple histograms
  • Group histograms
  • Non-normal process capability
  • Box-Cox and Johnson data transformation
  • Distribution fitting
  • Power and sample size calculations
  • Maintain customized formatting on charts when updating

Purchase our version 4 at current pricing between now and October 1 and qualify for a free upgrade to version 5.

Our anticipated release data is October 1, 2014. For more details on SPC for Excel Version 5, please click here

Quick Links

SPC for Excel Software

Visit our home page

SPC Training

SPC Consulting

Ordering Information

Thanks so much for reading our publication. We hope you find it informative and useful. Happy charting and may the data always support your position.

Sincerely,

Dr. Bill McNeese
BPI Consulting, LLC

View Bill McNeese's profile on LinkedIn

Connect with Us

       

Comments (8)

  • anon

    Thank you for another great and interesting Newsletter Bill, and your SPC teaching. Looking forward to Version 5. Didrik

    Jun 29, 2014
  • anon

    now i don't have cognitive dissonance on normality in control charts :)

    Jun 30, 2014
  • anon

    Hi thank you for writing this article- it's very helpful and informative. I just have a quick question- is it unusual for non-normal data to have Individuals and Moving Range graphs in control before transformation, but to have the graphs out of control after transformation? Any advice would be greatly appreciated. Kind regards

    Feb 09, 2015
  • anon

    Hello,

    I find that odd but I would have to see the data to understand what is going on.  Have you seen this?

    Bill

    Feb 11, 2015
  • anon

    Does it will be more pedagogical to suggest the readers to evaluate data distribution (such as shown in Figure 1) and then choose the most appropriate chart (exponential chart for this case/data)?

    Dec 29, 2016
  • anon

    There is nothing wrong with doing that.  Just need to be sure that there is a reason why your process would produce that type of data.  But most of the time, the individuals chart will give you pretty good results as explained above.

    Dec 30, 2016
  • anon

    Hii Bill, Thanks for the great insight into non-normal data. Can you please explain this statement " The control limits are found based on the same probability as a normal distribution.  So, the LCL and UCL are set at the 0.00135 and 0.99865 percentiles for the distribution. " in detail. I want to know how control limits will be calculated based on above mentioned percentiles.

    Jan 04, 2018
  • anon

    If you have a perfect normal distribution, those probabilities represent the the probability of getting a point beyond three sigma limits.  So, you simply use the functions for each different distribution to determine the values that give the same probabilities.

    Jan 04, 2018

Leave a comment

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <h1> <h2> <h3> <h4> <h5> <h6> <img> <hr> <div> <span> <strike> <b> <i> <u> <table> <tbody> <tr> <td> <th>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Image CAPTCHA
Enter the characters shown in the image.