This chapter is from the book
This chapter is from the book
This chapter is from the book
DescTools comes equipped with the Pizza data frame that I used to illustrate, earlier in this chapter, Excel’s Descriptive Statistics add-in. Therefore, this section uses the Pizza data frame to illustrate R’s functions. There’s no need to go through the steps to import data from either another application or from the standard comma-separated values ASCII format.
DescTools includes the very useful Desc function, which you can deploy to obtain descriptive statistics on any of the following:
A single numeric variable
A nominal (or category) variable
Two numeric variables to obtain a correlation
A nominal and a numeric variable to get, for example, mean values of the numeric variable by individual levels of the nominal variable
Two nominal variables to obtain a contingency table
You can get univariate statistics for a single numeric variable by using the Desc function in this way, on the Temperature variable in the Pizza data frame:
Desc(d.pizza$temperature)
Controlling the Type of Notation
Just as a personal matter I find it irritating to go looking for the mean weight of a group of subjects and find that R reports that its value is 1.49e + 02. You know and I know that means 149, and maybe the fractional portion of the mean weight is irrelevant—but on the other hand maybe it’s critically important.
The DescTools package routinely and by default reports results in scientific notation, under which the “e” is understood to mean “times 10 raised to the power of.” This can be a convenient way to represent large numbers, but it also obscures what might be significant digits.
Therefore, I generally enter the following command early in an R session that will use the DescTools package:
options(scipen = 10)
The keyword scipen in this case is set equal to 10. (The option does not belong exclusively to DescTools.) The option applies a penalty—hence the pen part of scipen—to the results of the function so that the use of scientific—hence the sci part—notation is replaced by fixed decimal notation such as 149.41. In this case, DescTools waits until fixed notation would be 10 characters longer than scientific notation before it switches from fixed to scientific.
Figures 2.7 and 2.8 show the difference in displayed results, first without the scipen option and then with it in use.
)
Figure 2.7 You can tell that the default number formats for the summary data aren’t too informative.
)
Figure 2.8 By setting the scipen option to a relatively large value, you override the default scientific notation.
Overriding the scientific notation with fixed decimal notation certainly improves matters, but we’re not there yet. Notice that the thousands separator in the results’ first row is an apostrophe. In the United States it’s conventional to use a comma instead. You can call for the comma in the integer fields of the results—that is, length, n, NAs, unique, and 0s—by means of this command:
options(fmt.abs=structure(list(digits=0, big.mark=”,”), class=“fmt”))
Because the values are all integers, you can call for the number of decimals, here termed digits, to equal 0. The thousands separator is termed big.mark, and here it’s set to the comma.
A quick side trip to explain some more R terminology. In Figures 2.7 and 2.8, the first five values are each counts and are therefore integers. The meanings of the labels are as follows:
Length: The number of records in the data frame d.pizza
N: The number of legitimate values of the variable Temperature
NAs: The number of not applicable values of the variable Temperature
Unique: The number of unique values of Temperature
0s: The number of times that the value 0 appears for Temperature
The effect of setting the big.mark option to a comma for the integer variables appears in Figure 2.9.
)
Figure 2.9 The thousands separator for Length through 0s is now the comma.
Although the Options statement has had its intended effect on the integer variables (no decimals; thousands separator is now the comma) it has had no effect on the statistics such as the mean through the kurtosis, nor on the values of the five smallest (“lowest”) and largest (“highest”) values in the data frame.
That’s because the Options statement set the format for the integers only; the fmt.abs argument restricts its effect to integer variables only. You need a separate Options statement for what DescTools terms numeric variables: the mean, the meanSE, and so on. Notice that although the Options statement for integers called for zero digits, the numeric variables still display three digits following the decimal point. To control the format for numeric (which you might also think of as floating point or double-precision) variables, you need to specify fmt.num instead of fmt.abs. Suppose that you want numeric values to use the comma as the thousands separator and five digits following the decimal point. You could use this command in DescTools:
options(fmt.num=structure(list(digits=5, big.mark=“,”), class=“fmt”))
)
Figure 2.10 The numeric statistics now display five digits following the decimal point.
The Reported Statistics
Now that we can actually see the summary statistics, let’s have another look at Figure 2.10 to see what they actually represent. Recall that you can call for these statistics using DescTools’ Desc function:
The Mean
The average of the numeric values in the variable, so R omits the NA values from the calculation. In the Pizza data frame, I have found that the mean of the Temperature variable, as calculated by R, differs from the mean as calculated by Excel by 0.000000000000071, thirteen zeros followed by a 7 and a 1.
The Standard Error of the Mean (meanSE)
This chapter has already discussed the meaning of this function and touched on its use. R uses the sample version of the standard deviation rather than the population version to calculate the standard error—appropriately, in my view. If you really have your hands on a population of values, there’s no point in calculating an inferential statistic such as the standard error. Excel’s calculation of the standard error differs from R’s at the fourteenth decimal place.
The Seven Quantiles
The seven values that occupy the second row of statistics for a numeric variable analyzed by the Desc function are collectively termed quantiles. They represent the values found at the 5th, 10th, 25th, 50th, 75th, 90th and 95th percentile ranks of the variable’s distribution.
For example, Figure 2.10 shows that the Desc function returns 26.70000 as the 5th percentile, labeled “.05.” To verify that statistic, you could take these steps:
Note that the Temperature variable has an n, or count, of 1,170 legitimate numeric records.
Calculate that five percent of the way through those 1,170 records is 0.05 times 1,170, or 58.5.
Sort the records using Temperature as the sort key, in ascending order.
Note that both the 58th and the 59th of the sorted records have a value of 26.7.
Therefore, the 5th percentile of the distribution of Temperature values is 26.7.
Similarly, the 90th percentile rank of 1,170 records is the 1,053rd record (that’s 0.9 times 1,170). Sorted by Temperature, the 1,053rd record in the data set has a Temperature value of 58.8, just as reported by the Desc function.
The emphasis in Desc’s results on quantiles reflects the DescTools package’s emphasis on “robust” statistics. In this context, the term robust has little to do with the robustness of inferential analyses such as the analysis of variance with respect to the violation of their assumptions. It has to do with a statistic’s stasis when a value in the distribution changes.
The clearest example of this quality is the range of a distribution. You can change any value, or any values, in the distribution other than the largest or the smallest, by any amount, and the range remains fixed. The same is not necessarily true of the quantiles that Desc reports, but the effect is similar. For example, to change the value of the 5th percentile, you would have to add a new value below that percentile or move a value from a spot below that percentile to a spot above it.
Personal opinion: I’m uncomfortable referring to that sort of property as “robust.” The term connotes a generally desirable quality. Certainly, there are times when you prefer to see a distribution’s characteristics unaffected by a record’s minor change in ranked position. But statistics such as the variance or correlation that do change automatically in response to moving a single record are often preferred for that very reason. For more on this matter, see the section later in this chapter on the median absolute deviation, or MAD.
You will often want to call the Desc function with the plotit argument, as follows:
Desc(d.pizza$temperature, plotit = TRUE)
When you use it as above to return univariate statistics, the plotit argument calls for the charts as shown in Figure 2.11.
)
Figure 2.11 The charts are a handy way to judge the nature of the variable’s distribution.
Figure 2.11 shows four charts:
A line chart that shows the relative frequency of different observations, superimposed over a histogram
A box-and-whiskers plot
A cumulative frequency line chart
The line chart, and the histogram it superimposes, are based on groups of observations whose Temperature ranges are five degrees wide. They are the same charts you would get with Excel by inserting a pivot chart that groups the Temperatures into bins of five degree each, and then charting the results as a combination column-line chart.
The box-and-whiskers plot requires a little more explanation. The first and third quartiles—that is, the 25th and 75th percentiles, which John Tukey terms the hinges—define the endpoints of the chart’s central box.
The endpoints of the whiskers are more complex. Tukey, who originated the idea of box-and-whisker plots, recommended that the whiskers extend no more than 1.5 times the size of the interquartile range, often referred to as the IQR, which is the distance from the first quartile to the third quartile. As Desc (and other functions in R and applications other than R) implement the recommendation, a whisker’s endpoint is the actual observation that is
Most distant from the median to one side or the other
No more than 1.5 times the IQR from its side of the box
So in Figure 2.11, the endpoint of the lower whisker is 22.6, and the endpoint of the upper whisker is 64.8. Figure 2.12 shows how these endpoints are calculated.
)
Figure 2.12 The calculations behind the endpoints are not complicated but they are obscure.
Here’s how to calculate the endpoint of the lower and upper whiskers, given the references in Figure 2.12:
The lower and upper edges of the box—the hinges—are at Quartile 1 and Quartile 3, shown in cells C2 and C3. These quartiles correspond to the 25th and 75th percentile ranks, shown in cells D2 and D3.
Cells E2 and E3 show the ranks of the records that occupy the 25% and 75% percentiles. With the records sorted in Temperature order, those ranks are 292.5 and 877.5.
The records at ranks 292 and 293 have the same Temperatures, as do the records at ranks 877 and 878. Therefore, we can take their actual Temperature values as defining the first and third quartiles, or the hinges of the chart’s box. Those actual values are shown in cells F2 and F3.
The interquartile range, IQR, is shown in cell G2. It is the difference between the Temperature values for the third and first quartiles. Multiplying the IQR by 1.5 results in 19.613, shown in cell H2.
Almost there. Subtract 19.613 from the Temperature value for the first quartile to get the lower whisker endpoint. Cell I2 shows that the result is 22.6125. Compare that with the location of the lower whisker’s endpoint in Figure 2.11’s box-and-whisker plot.
Finally, add 19.613 to the Temperature value for the third quartile. The result, 74.913, appears in cell J3. However, that value is greater than the maximum observed value for Temperature. Therefore, we take the maximum observed value as the endpoint for the upper whisker. That maximum value is 64.8, as shown in cell K3.
When observed values exist that are outside the calculated whisker endpoints, R plots them and terms them outliers.
The Range
The range is simply the difference between the maximum and the minimum legitimate numeric values of the variable you’re analyzing. It is an extremely static—some would say robust—measure of the variability in a set of values. You can change every value in the set except the minimum and maximum without changing the range.
Historically, the range has been a useful measure of variability for statistical process control in factory settings where access to computing power was limited. Estimates of control limits could be made quickly with pencil and paper by basing them on the range. More recently, easy access to computers has eliminated that reason to employ the range. It remains a good way to look for errors in data entry. If the range of a patient’s body temperature runs from 98.0 to 990.0, it’s a good bet that someone dropped a decimal point.
The Standard Deviation (SD)
The standard deviation that the Desc function returns is the sample version—that is, it assumes that your data set constitutes a sample from a population, and it divides by its degrees of freedom, or (n − 1), to estimate the standard deviation of that population. This is consistent with the calculation of the standard error of the mean, discussed earlier in this chapter. For the Temperature variable in the Pizza data set, Excel and R return the same value for the standard deviation to the fourteenth decimal place. Beyond that they both return a sequence of zeros.
The Coefficient of Variation (vcoef)
The coefficient of variation, in the context of univariate analysis, is the ratio of a variable’s standard deviation to its mean. The result has no special intuitive meaning by itself. But it can tell you whether two different variables have different degrees of variability. The idea is that the standard deviation and the mean are measured using the same unit. So, by dividing the standard deviation by the mean, you remove the effect of the scale of measurement and you’re left with an indicator of the intrinsic amount of dispersion measured by a variable.
It’s customary to multiply the ratio by 100, in order to work less frequently with ratios less than 1.0. The Desc function does not do this.
Suppose that you wanted to look into the variability in the salaries of computer programmers in 1990 as compared to 2010. Your research shows that the mean salary in 1990 was $40,000 and the standard deviation was $4,000, for a coefficient of variation of 10; in 2010 the mean was $60,000 and the standard deviation was $6,000, and the coefficient of variation is still 10. By that measure, the variability did not change over the 20-year period, even though the standard deviation increased from $4,000 to $6,000, as the scale—in this case, the purchasing power of a dollar—changed over the years.
The same approach is sometimes taken with units instead of variables. For example, suppose that your company manufactures pistons for internal combustion engines. You manufacture one size of piston for automobile engines and a smaller piston for two-stroke engines. The piston ring diameter for the larger engine might be 75 millimeters, with a standard deviation of 0.01 millimeters. Your two-stroke pistons might average 30 millimeters in diameter with a standard deviation of 0.007 millimeters. So the coefficient of variation would be
0.01 / 75 = 0.0001
for the auto engine piston rings, and
0.007 / 30 = 0.0002
for the two-stroke engine. Therefore, even though the auto engine ring diameters have a larger standard deviation than the two-stroke engine rings, when the effect of the unit of measurement is removed from the comparison it turns out that the diameters of the two-stroke engine rings have greater intrinsic variation.
There are various caveats for the use of the coefficient of variation, and accompanying workarounds. For example, you’re usually advised to take the absolute value of the mean or of the ratio, to avoid a negative value for the coefficient of variation. And variables whose mean value is zero, or close to zero, are usually avoided so that you don’t wind up with a division by zero error or a huge coefficient of variation.
The Median Absolute Deviation (mad)
The median absolute deviation, or MAD, is one way to estimate the variability in a set of values. The general approach is to take the absolute values of deviations of individual values from a measure of their central tendency.
Standard measures of variability such as the variance and the standard deviation are built on sums of squares, shorthand for sums of squared deviations. If you were to find the deviations of, say, ten values from their mean and total the deviations, the result would be zero. It would always be zero, regardless of the values themselves or the number of values. So the average deviation is useless as a measure of variability.
Approaches based on sums of squares get around this difficulty by squaring the deviations before summing. The square of a number is always positive, and therefore—given that there’s any variation in the numbers at all—the greater the variation the greater the sum of squares.
You get to the variance by dividing the sum of squares by the number of values. This makes the variance independent of the number of observations: it’s the average squared deviation. And you get to the standard deviation by taking the square root of the variance.
But squaring the deviations can lead to an over-estimate of the variance. One outlier can cause a large deviation, and squaring that large deviation can cause an estimate of the variance that does not stand up in subsequent samples. So an alternative to squaring the deviations that nevertheless returns a positive number as its result is to take their absolute values.
One approach to doing so is to take the deviation of each value in the data set from the set’s mean (that is, its arithmetic average). Get each deviation’s absolute value and find their average.
The approach that the Desc function takes is to use the median instead of the mean. (Unfortunately, the fact that both median and mean begin with the letter “m” causes the acronym to be MAD in either case.) The Desc function calculates the deviations from the median of the original data set, and then finds the median of the absolute values of the deviations.
Certain properties describe different statistics. Unbiasedness, for example, is one desirable property. An unbiased statistic does not routinely underestimate or overestimate the population parameter that it’s meant to estimate. The sample mean is an unbiased statistic. It may not equal the population mean—in fact, it almost certainly doesn’t—but it does not routinely underestimate or overestimate that parameter.
In contrast, the average squared deviation from a sample mean, which is the variance, is a biased statistic. It routinely underestimates the population variance. That’s the reason the sample variance divides the sum of the squared deviations by (n − 1) rather than by n: to correct the bias. (The standard deviation, though, is still slightly biased. The square root of an unbiased statistic is not itself necessarily unbiased.)
Another desirable property of a statistic is consistency. A consistent statistic comes nearer to a population parameter as the size of the sample on which it’s based gets larger. Suppose that the mean age of all residents of the state of Illinois is 55. A random sample of 10 people who live in Illinois might have an average age of 45 years, a 10-year underestimate. Because the mean is a consistent statistic, a random sample of 100 Illinois residents would tend strongly to be much closer to the population parameter—say, 52 instead of 45. That’s a consistent statistic.
Users of the MAD often like to use it as an estimator of the population standard deviation. The median of the absolute deviations from the median is not a consistent estimator of that parameter. It turns out that if you multiply the median of the absolute deviations from the median by a constant, 1.4826, that version of the MAD is a consistent estimator of the population standard deviation.
Figure 2.13 shows the calculation of what the Desc function reports as the MAD.
)
Figure 2.13 I have omitted the NA values from column A’s list of temperatures.
The Interquartile Range (IQR)
This statistic is easily calculated as the distribution’s 75th percentile (or third quartile) less its 25th percentile (or first quartile). The IQR defines the distance between the hinges of the box in the box-and-whisker plot (see Figure 2.11).
The Skewness
Skewness is a measure of a distribution’s departure from symmetry. Skewed distributions tend to have values that bunch up in one tail of the distribution, often resulting in an opposite tail that’s relatively long and thin. If that long, thin tail is on the left side of the distribution, the variable is said to be negatively skewed; if to the right side, positively skewed. The distribution shown in Figure 2.11 is negatively skewed.
Skewness is conventionally calculated as a distribution’s average cubed z-score. That is:
Calculate the distribution’s mean and standard deviation.
Subtract the mean from each value and divide the result by the standard deviation, to get each value as a z-score.
Cube the z-scores.
Find the average of the cubed z-scores to determine the skewness of the original values.
Figure 2.14 illustrates the process. The Desc function calculates the skewness in the same way. (A normal distribution, by the way, has a skewness of 0.0.)
)
Figure 2.14 The negative skewness indicates an asymmetric distribution.
The Kurtosis
Kurtosis measures how much more peaked (or how much flatter) a distribution is vis-à-vis the normal curve. It is calculated in much the same way as skewness, except:
The z-scores are raised to the fourth instead of the third power.
It’s customary to subtract the constant 3 from the average of the raised z-scores.
The average of the z-scores raised to the fourth power in a normal curve is 3. Subtracting 3 from the average means that a normal curve has a kurtosis of zero—a convenient result because a normal curve also has a skewness of zero. A curve flatter than the normal has a negative kurtosis, and a curve taller than normal has a positive kurtosis. The Desc function follows the custom of subtracting 3 from the average of the raised z-scores.
Figure 2.15 illustrates the process of calculating the kurtosis.
)
Figure 2.15 The positive kurtosis indicates that the distribution is slightly more peaked than normal.
Lowest and Highest Five
Mainly to help you interpret any outliers shown in the box-and-whiskers plot, Desc shows you a list of the five highest and five lowest values in the selected variable. If a tie exists, the number of tied records for a given value appears after that value in parentheses.
Running the Desc Function on Nominal Variables
The Desc function has useful information for you concerning nominal, categorical variables as well as the statistics concerning numeric variables that the prior section discussed. It’s primarily frequency and percentage breakdowns, and the amount of information is sparser than with numeric variables.
Figure 2.16 shows an example of how the nominal variable Driver appears when analyzed by the Desc function.
)
Figure 2.16 The analysis for a nominal variable—termed factor by R—is limited to simple counts and percents.
The data provided for a nominal variable forms the basis for the Pareto charts, sorted histograms, shown in Figure 2.17. You get the charts by specifying plotit = TRUE in the Desc command.
)
Figure 2.17 The sorted charts make it easy to identify the most and least frequently occurring values.
The default order is descending by frequency. But you can use an ord argument to change that default. For example:
Desc(d.pizza$driver,ord=“asc”)
returns the values in order of ascending frequency. And:
Desc(d.pizza$driver,ord=“level”)
returns the values in alphabetic order by the driver’s name. (The Desc function uses the term “level” to refer to the different values that a factor can take.)