- Data Validation
- Variable and Model Selection
- Preliminary Analyses
- Building the Multi-Variable Model
- Extracting the Equation
- Final Comments
This chapter is from the book
This chapter is from the book
This chapter is from the book
Building the Multi-Variable Model
I call the technique I've developed to build the multi-variable model "stepwise ANOVA" (analysis of variance). It is very similar to forward stepwise regression except I use an analysis of variance procedure to build models with categorical variables. You will learn more about analysis of variance in Chapter 6. For the moment, you just need to know that this procedure allows us to determine the influence of numerical and categorical variables on the dependent variable, leffort. The model starts "empty" and then the variables most related to leffort are added one by one in order of importance until no other variable can be added to improve the model. The procedure is very labor-intensive because I make the decisions at each step myself; it is not automatically done by the computer. Although I am sure this could be automated, there are some advantages to doing it yourself. As you carry out the steps, you will develop a better understanding of the data. In addition, in the real world, a database often contains many missing values and it is not always clear which variable should be added at each step. Sometimes you need to follow more than one path to find the best model. In the following example, I will show you the simplest case using our 34-project, 6-variable database with no missing values. My goal for this chapter is that you understand the methodology. The four case studies in Chapters 2 through 5 present more complicated analyses, and will focus on interpreting the output.
Example
Determine Best One-Variable Model First, we want to find the best one-variable model. Which variable, lsize, t13, t14, app, or telonuse, explains the most variation in leffort? I run regression procedures for the numerical variables and ANOVA procedures for the categorical variables to determine this. In practice, I do not print all the output. I save it in output listing files and record by hand the key information in a summary sheet. Sidebar 1.2 shows a typical summary sheet. I note the date that I carried out the analysis, the directory where I saved the files, and the names of the data file, the procedure file(s), and the output file(s). I may want to look at them again in the future, and if I don't note their names now, I may never find them again! We are going to be creating lots of procedures and generating lots of output, so it is important to be organized. I also note the name of the dependent variable.
Now I am ready to look at the output file and record the performance of the models. In the summary sheet, I record data only for significant variables. For the regression models, a variable is highly significant if its P>|t| value is 0.05 or less. In this case, I do not record the actual value; I just note the number of observations, the variable's effect on effort, and the adjusted R-squared value. If the significance is borderline, that is, if P>|t| is a number between 0.05 and 0.10, I note its value. If the constant is not significant, I note it in the Comments column. If you are analyzing a very small database, you might like to record these values for every variable—significant or not. Personally, I have found that it is not worth the effort for databases with many variables. If I need this information later, I can easily go back and look at the output file.
For the ANOVA models, I do the same except I look at a variable's Prob>F value to determine if the variable is significant. The effect of a categorical variable depends on the different types. For example, using Telon (telonuse= Yes) will have one effect on leffort and not using Telon (telonuse=No) will have a different effect on leffort. You cannot determine the effect from the ANOVA table.
In Example 1.11, I have highlighted the key numbers in bold. I see that there is a very significant relationship between leffort and lsize (P>|t|= 0.000): lsize explains 64% of the variation in leffort. The coefficient of lsize (Coef.) is a positive number (0.9298). This means that leffort increases with increasing lsize. The model was fit using data from 34 projects. I add this information to the summary sheet (Sidebar 1.2).
Example 1.11
. regress leffort 1size
Source SS df MS Number of obs = 34
Model 22.6919055 1 22.6919055 F(1,32) = 59.67
Residual 12.1687291 32 .380272786 Prob > F = 0.0000
Total 34.8606346 33 1.05638287 R-squared = 0.6509
Adj R-squared = 0.6400
Root MSE = .61666
Leffort Coef. Std. Err. t P>|t| [95% Conf. Interval]
Lsize .9297666 .1203611 7.725 0.000 .6845991 1.174934
_cons 3.007431 .7201766 4.176 0.000 1.54048 4.474383
In Example 1.12, I see that there is not a significant relationship between leffort and t13. Therefore, I do not even look at the coefficient of t13. I note nothing and move on to the next model.
Example 1.12
. regress leffort t13
Source SS df MS Number of obs = 34
Model .421933391 1 .421933391 F(1,32) = 0.39
Residual 34.4387012 32 1.07620941 Prob > F = 0.5357
Total 34.8606346 33 1.05638287 R-squared = 0.0121
Adj R-squared = 0.0188
Root MSE = -1.0374
Leffort Coef. Std. Err. t P>|t| [95% Conf. Interval]
t13 .1322679 .2112423 0.626 0.536 -.2980186 .5625544
_cons 8.082423 .706209 11.445 0.000 6.643923 9.520924
In Example 1.13, I see that there is a very significant relationship between leffort and t14 : t14 explains 36% of the variation in leffort. The coefficient of t14 is negative. This means that leffort decreases with increasing t14. The model was fit using data from 34 projects. I add this information to the summary sheet (Sidebar 1.2).
Example 1.13
. regress leffort t14
Source SS df MS Number of obs = 34
Model 13.1834553 1 13.1834553 F(1,32) = 19.46
Residual 21.6771793 32 .677411853 Prob > F = 0.0001
Total 34.8606346 33 1.05638287 R-squared = 0.3782
Adj R-squared = 0.3587
Root MSE = .82305
Leffort Coef. Std. Err. t P>|t| [95% Conf. Interval]
t14 -.7022183 .1591783 -4.412 0.000 -1.026454 -.3779827
_cons 10.55504 .4845066 21.785 0.000 9.568136 11.54195
In Example 1.14, I see that there is no significant relationship between leffort and app. I note nothing and move on to the next model.
Example 1.14
. anova leffort app
Number of obs = 34 R-squared = 0.0210
Root MSE = 1.06659 Adj R-squared = -0.0769
Source Partial SS df MS F Prob > F
Model .732134098 3 .244044699 0.21 0.8855
App .732134098 3 .244044699 0.21 0.8855
Residual 34.1285005 30 1.13761668
Total 34.8606346 33 1.05638287
In Example 1.15, I see that there is a borderline significant relationship between leffort and telonuse : telonuse explains 8% of the variation in leffort. The model was fit using data from 34 projects. I add this information to the summary sheet (Sidebar 1.2).
Example 1.15
. anova leffort telonuse
Number of obs = 34 R-squared = 0.1094
Root MSE = .984978 Adj R-squared = 0.0816
Source Partial SS df MS F Prob > F
Model 3.81479355 1 3.81479355 3.93 0.0560
Telonuse 3.81479355 1 3.81479355 3.93 0.0560
Residual 31.0458411 32 .970182533
Total 34.8606346 33 1.05638287
Sidebar 1.2 Statistical Output Summary Sheet
Date: 01/03/2001
Directory: C:\my documents\data analysis book\example34\
Data File: bankdata34.dta
Procedure Files: *var.do (* = one, two, three, etc.)
Output Files: *var.log
Dependent Variable: leffort
|
Variables |
Num Obs |
Effect |
Adj R2 |
Significance of Added Variable |
Comments |
|
1-variable models |
|
|
|
|
|
|
*lsize |
34 |
+ |
0.64 |
|
|
|
t14 |
34 |
– |
0.36 |
|
|
|
telonuse |
34 |
|
0.08 |
.056 |
|
|
2-variable models |
|
|
|
|
|
|
with lsize |
|
|
|
|
|
|
t14 |
34 |
– |
0.73 |
|
best model, sign. = 0.0000 |
|
3-variable models |
|
|
|
|
|
|
with lsize, t14 |
|
|
|
|
|
|
none significant |
|
|
|
|
no further improvement possible |
Once I have recorded all of the output in the summary sheet, I select the variable that explains the most variation in leffort. In this step, it is obviously lsize. There is no doubt about it. Then I ask myself: Does the relationship between leffort and lsize make sense? Does it correspond to the graph of leffort as a function of lsize (Figure 1.8)? Yes, it does, so I add lsize to the model and continue with the next step.
){kind=link}
Determine Best Two-Variable Model Next I want to determine which variable, t13, t14, app, or telonuse, in addition to lsize, explains the most variation in leffort. I add lsize to my regression and ANOVA procedures and run them again. I record the output in the same summary sheet (Sidebar1.2). What I am principally concerned with at this stage is if the additional variable is significant. So first I look at P>|t| value of this variable. If it is not significant, I record nothing and move on to the next model.
In Example 1.16, I see that t13 is not significant (0.595).
Example 1.16
. regress leffort lsize t13
Source SS df MS Number of obs = 34
Model 22.8042808 2 11.4021404 F(2,31) = 29.32
Residual 12.0563538 31 .388914638 Prob > F = 0.0000
Total 34.8606346 33 1.05638287 R-squared = 0.6542
Adj R-squared = 0.6318
Root MSE = .62363
Leffort Coef. Std. Err. t P>|t| [95% Conf. Interval]
Lsize .943487 .1243685 7.586 0.000 .6898359 1.197138
t13 -.0697449 .1297491 -0.538 0.595 -.33437 .1948801
_cons 3.151871 .7763016 4.060 0.000 1.568593 4.735149
In Example 1.17, I learn that t14 is significant (0.002): lsize and t14 together explain 73% of the variation in leffort. The coefficient of t14 is a negative number. This means that leffort decreases with increasing t14. This is the same effect that we found in the one-variable model. If the effect was different in this model, that could signal something strange going on between lsize and t13, and I would look into their relationship more closely. lsize and the constant (_cons) are still significant. If they were not, I would note this in the Comments column. Again, this model was built using data from 34 projects.
Example 1.17
. regress leffort lsize t14
Source SS df MS Number of obs = 34
Model 25.9802069 2 12.9901035 F(2,31) = 45.35
Residual 8.88042769 31 .286465409 Prob > F = 0.0000
Total 34.8606346 33 1.05638287 R-squared = 0.7453
Adj R-squared = 0.7288
Root MSE = .53522
Leffort Coef. Std. Err. t P>|t| [95% Conf. Interval]
Lsize .7678266 .1148813 6.684 0.000 .5335247 1.002129
t14 -.3856721 .1138331 -3.388 0.002 -.6178361 -.153508
_cons 5.088876 .8764331 5.806 0.000 3.301379 6.876373
In Examples 1.18 and 1.19, I see that app and telonuse are not significant (0.6938 and 0.8876).
Example 1.18
. anova leffort lsize app, category (app)
Number of obs = 34 R-squared = 0.6677
Root MSE = .63204 Adj R-squared = 0.6218
Source Partial SS df MS F Prob > F
Model 23.2758606 4 5.81896516 14.57 0.0000
Lsize 22.5437265 1 22.5437265 56.43 0.0000
app .583955179 3 .194651726 0.49 0.6938
Residual 11.584774 29 .399474964
Total 34.8606346 33 1.05638287
Example 1.19
. anova leffort lsize telonuse, category (telonuse)
Number of obs = 34 R-squared = 0.6512
Root MSE = .626325 Adj R-squared = 0.6287
Source Partial SS df MS F Prob > F
Model 22.6998727 2 11.3499363 28.93 0.0000
Lsize 18.8850791 1 18.8850791 48.14 0.0000
telonuse .007967193 1 .007967193 0.02 0.8876
Residual 12.1607619 31 .392282644
Total 34.8606346 33 1.05638287
Again, the decision is obvious: The best two-variable model of leffort is lsize and t14. Does the relationship between t14 and leffort make sense? Does it correspond to the graph of leffort as a function of t14? If yes, then we can build on this model.
Determine Best Three-Variable Model Next I want to determine which variable, t13, app, or telonuse, in addition to lsize and t14, explains the most variation in leffort. I add t14 to my regression and ANOVA procedures from the previous step and run them again. I record the output in the same summary sheet (Sidebar 1.2). As in the previous step, what I am principally concerned with at this stage is if the additional variable is significant. If it is not significant, I record nothing and move on to the next model. Let's look at the models (Examples 1.20, 1.21, and 1.22).
Example 1.20
. regress leffort lsize t14 t13
Source SS df MS Number of obs = 34
Model 26.0505804 3 8.68352679 F(3, 30) = 29.57
Residual 8.81005423 30 .293668474 Prob > F = 0.0000
Total 34.8606346 33 1.05638287 R-squared = 0.7473
Adj R-squared = 0.7220
Root MSE = .54191
leffort Coef. Std. Err. t P>|t| [95% Conf. Interval]
lsize .7796095 .118781 6.563 0.000 .5370263 1.022193
t14 -.383488 .1153417 -3.325 0.002 -.6190471 -.1479289
t13 -.055234 .1128317 -0.490 0.628 -.285667 .175199
_cons 5.191477 .9117996 5.694 0.000 3.329334 7.05362
Example 1.21
. anova leffort lsize t14 app, category (app)
Number of obs = 34 R-squared = 0.7478
Root MSE = .560325 Adj R-squared = 0.7028
Source Partial SS df MS F Prob > F
Model 26.0696499 5 5.21392998 16.61 0.0000
lsize 12.3571403 1 12.3571403 39.36 0.0000
t14 2.79378926 1 2.79378926 8.90 0.0059
app .089442988 3 .029814329 0.09 0.9622
Residual 8.7909847 28 .313963739
Total 34.8606346 33 1.05638287
Example 1.22
. anova leffort lsize t14 telonuse, category(telonuse)
Number of obs = 34 R-squared = 0.7487
Root MSE = .540403 Adj R-squared = 0.7236
Source Partial SS df MS F Prob > F
Model 26.099584 3 8.69986134 29.79 0.0000
lsize 12.434034 1 12.434034 42.58 0.0000
t14 3.39971135 1 3.39971135 11.64 0.0019
telonuse .119377093 1 .119377093 0.41 0.5274
Residual 8.7610506 30 .29203502
Total 34.8606346 33 1.05638287
None of the additional variables in the three models (Examples 1.20, 1.21, and 1.22) are significant.
The Final Model The stepwise ANOVA procedure ends as no further improvement in the model is possible. The best model is the two-variable model: leffort as a function of lsize and t14. No categorical variables were significant in this example, so this model is the same model found by the automatic stepwise regression procedure. I check one final time that the relationships in the final model (Example 1.23) make sense. We see that lsize has a positive coefficient. This means that the bigger the application size, the greater the development effort required. Yes, this makes sense to me. I would expect bigger projects to require more effort. The coefficient of t14, staff tool skills, is negative. This means that effort decreases with increasing staff tool skills. Projects with very high staff tool skills required less effort than projects with very low staff tool skills, everything else being constant. Yes, this makes sense to me, too. Print the final model's output and save it.
Example 1.23
. regress leffort lsize t14
Source SS df MS Number of obs = 34
Model 25.9802069 2 12.9901035 F(2, 31) = 45.35
Residual 8.88042769 31 .286465409 Prob > F = 0.0000
Total 34.8606346 33 1.05638287 R-squared = 0.7453
Adj R-squared = 0.7288
Root MSE = .53522
leffort Coef. Std. Err. t P>|t| [95% Conf. Interval]
lsize .7678266 .1148813 6.684 0.000 .5335247 1.002129
t14 -.3856721 .1138331 -3.388 0.002 -.6178361 -.153508
_cons 5.088876 .8764331 5.806 0.000 3.301379 6.876373
On the summary sheet, I note the significance of the final model. This is the Prob > F value at the top of the output. The model is significant at the 0.0000 level. This is Stata's way of indicating a number smaller than 0.00005. This means that there is less than a 0.005% chance that all the variables in the model (lsize and t14) are not related to leffort. (More information about how to interpret regression output can be found in Chapter 6.)
What to Watch Out For
Be sure to use an ANOVA procedure that analyzes the variance of unbalanced data sets, that is, data sets that do not contain the same number of observations for each categorical value. I have yet to see a "balanced" software development database. In Stata, the procedure is called "ANOVA."
Use the transformed variables in the model.
Some models may contain variables with lots of missing values. It might be better to build on the second best model if it is based on more projects.
If the decision is not obvious, follow more than one path to its final model (see Chapters 4 and 5). You will end up with more than one final model.
Always ask yourself at each step if a model makes sense. If a model does not make sense, use the next best model.
Checking the Model
Before we can accept the final model found in the previous step, we must check that the assumptions underlying the statistical tests used have not been violated. In particular, this means checking that:
Independent numerical variables are approximately normally distributed. (We did this in the preliminary analyses.)
Independent variables are not strongly related to each other. (We did this partially during the preliminary analyses; now, we need to complete it.)
The errors in our model should be random and normally distributed. (We still need to do this.)
In addition, we also need to check that no single project or small number of projects has an overly strong influence on the results.
Numerical Variable Checks
We already calculated the correlation coefficients of numerical variables in our preliminary analyses and noted them. Now that I have my final model, I need to check that all the independent numerical variables present in the final model are not strongly linearly related to each other. In other words, I need to check for multicollinearity problems. Why would this cause a problem? If two or more explanatory variables are very highly correlated, it is sometimes not possible for the statistical analysis software to separate their independent effects and you will end up with some strange results. Exactly when this will happen is not predictable. So, it is up to you to check the correlations between all numerical variables. Because my model only depends on lsize and t14, I just need to check their correlation with each other. To avoid multicollinearity problems, I do not allow any two variables with an absolute value of Spearman's rho greater than or equal to 0.75 in the final model together. From our preliminary correlation analysis, we learned that size1 and t14 are slightly negatively correlated; they have a significant Spearman's correlation coefficient of –0.3599. Thus, there are no multicollinearity problems with this model.
You should also be aware that there is always the possibility that a variable outside the analysis is really influencing the results. For example, let's say I have two variables, my weight and the outdoor temperature. I find that my weight increases when it is hot and decreases when it is cold. I develop a model that shows my weight as a function of outdoor temperature. If I did not use my common sense, I could even conclude that the high outdoor temperature causes my weight gain. However, there is an important variable that I did not collect which is the real cause of any weight gain or loss—my ice cream consumption. When it is hot outside, I eat more ice cream, and when it is cold, I eat much less. My ice cream consumption and the outdoor temperature are therefore highly correlated. The model should really be my weight as a function of my ice cream consumption. This model is also more useful because my ice cream consumption is within my control, whereas the outdoor temperature is not. In this case, the outdoor temperature is confounded2 with my ice cream consumption and the only way to detect this is to think about the results. Always ask yourself if your results make sense and if there could be any other explanation for them. Unfortunately, we are less likely to ask questions and more likely to believe a result when it proves our point.
Categorical Variable Checks
Strongly related categorical variables can cause problems similar to those caused by numerical variables. Unfortunately, strong relationships involving categorical variables are much more difficult to detect. We do not have any categorical variables in our final effort model, so we do not need to do these checks for our example. However, if we had found that telonuse and app were both in the model, how would we check that they are not related to each other or to the numerical variables in the model?
To determine if there is a relationship between a categorical variable and a numerical variable, I use an analysis of variance procedure. Let's take app and t14 in Example 1.24. Does app explain any variance in t14?
Example 1.24
. anova t14 app
Number of obs = 34 R-squared = 0.1023
Root MSE = .894427 Adj R-squared = 0.0125
Source Partial SS df MS F Prob > F
Model 2.73529412 3 .911764706 1.14 0.3489
app 2.73529412 3 .911764706 1.14 0.3489
Residual 24.00 30 .80
Total 26.7352941 33 .810160428
Example 1.24 shows that there is no significant relationship between app and t14 (the Prob > F value for app is a number greater than 0.05). I run ANOVA procedures for every categorical/numerical variable combination in the final model. (Note that the numerical variable must be the dependent LHS variable.) If I find a very strong relationship, I will not include the two variables together in the same model. I define "very strong relationship" as one variable explaining more than 75% of the variation in another.
I would like to point out here that we can get a pretty good idea about which variables are related to each other just by looking at the list of variables that are significant at each step as we build the one-variable, two-variable, three-variable, etc. models. In the statistical output sheet, Sidebar 1.2, we see that telonuse is an important variable in the one-variable model. However, once lsize has been added to the model, telonuse does not appear in the two-variable model. This means that there is probably a relationship between telonuse and lsize. Let's check (Example 1.25):
Example 1.25
. anova lsize telonuse Number of obs = 34 R-squared = 0.1543 Root MSE = .832914 Adj R-squared = 0.1279 Source Partial SS df MS F Prob > F Model 4.04976176 1 4.04976176 5.84 0.0216 telonuse 4.04976176 1 4.04976176 5.84 0.0216 Residual 22.1998613 32 .693745665 Total 26.2496231 33 .795443123
Yes, there is a significant relationship between lsize and telonuse. The use of Telon explains about 13% of the variance in lsize. Example 1.26 shows that applications that used Telon were much bigger than applications that did not. So, the larger effort required by applications that used Telon (Example 1.5) may not be due to Telon use per se, but because the applications were bigger. Once size has been added to the effort model, Telon use is no longer important; size is a much more important driver of effort. I learn as I analyze. Had this all been done automatically, I may not have noticed this relationship.
Example 1.26
. table telonuse, c(mean size) Telon Use mean(size) No 455 Yes 1056
It is more difficult to determine if there is an important relationship between two categorical variables. To check this, I first calculate the chi-square statistic to test for independence. From this I learn if there is a significant relationship between two categorical variables, but not the extent of the relationship. (You will learn more about the chi-square test in Chapter 6.) In Example 1.27, I am interested in the Pr value (in bold). Pr is the probability that we are making a mistake if we say that there is a relationship between two variables. If the value of Pr is less than or equal to 0.05, we can accept that there is a relationship between the two variables. Here, Pr = 0.069, so I conclude that there is no significant relationship between the two variables.
Example 1.27
. tabulate app telonuse, chi2
Application Type Telon Use
No Yes Total
CustServ 6 0 6
MIS 3 0 3
TransPro 16 4 20
InfServ 2 3 5
Total 27 7 34
Pearson chi2(3) = 7.0878 Pr = 0.069
If there is a significant relationship, I need to look closely at the two variables and judge for myself if they are so strongly related that there could be a problem. For example, if application type (app) and Telon use (telonuse) had been significantly related, I would first look closely at Example 1.27. There I would learn that no customer service (CustServ) or MIS application used Telon. Of the seven projects that used Telon, there is a split between transaction processing (TransPro) applications (a high-effort category; see Example 1.4) and information service (InfServ) applications (a low-effort category). Thus, the high effort for Telon use (see Example 1.5) is not due to an over-representation of high-effort transaction processing applications. In fact, the majority of projects that did not use Telon are transaction processing applications. I conclude that any relationship between Telon use and effort cannot be explained by the relationship between application type and Telon use; i.e. application type and Telon use are not confounded.
If I find any problems in the final model, I return to the step where I added the correlated/confounded variable to the variables already present in the model, take the second best choice, and rebuild the model from there. I do not carry out any further checks. The model is not valid, so there is no point. We have to start again. (See Chapter 5 for an example of confounded categorical variables.)
Testing the Residuals
In a well-fitted model, there should be no pattern to the errors (residuals) plotted against the fitted values. The term "fitted value" refers to the leffort predicted by our model; the term "residual" is used to express the difference between the actual leffort and the predicted leffort for each project. Your statistical analysis tool should calculate the predicted values and residuals automatically for you. The errors of our model should be random. For example, we should not be consistently overestimating small efforts and underestimating large efforts. It is always a good idea to plot this relationship and take a look. If you see a pattern, it means that there is a problem with your model. If there is a problem with the final model, then try the second best model. If there is a problem with the second best model, then try the third best model, and so on. In Figure 1.12, I see no pattern in the residuals of our final model.
)
Figure 1.12 Residuals vs. fitted values
In addition, the residuals should be normally distributed. We can see in Figure 1.13 that they are approximately normally distributed. You will learn more about residuals in Chapter 6.
)
Figure 1.13 Distribution of residuals
Detecting Influential Observations
How much is our final model affected by any one project or subset of our data? If we dropped one project from our database, would our model be completely different? I certainly hope not. But we can do better than hope; we can check the model's sensitivity to individual observations. Projects with large predicted errors (residuals) and/or projects very different from other project's values for at least one of the independent variables in the model can exert undue influence on the model (leverage).
Cook's distance summarizes information about residuals and leverage into a single statistic. Cook's distance can be calculated for each project by dropping that project and re-estimating the model without it. My statistical analysis tool does this automatically. Projects with values of Cook's distance, D, greater than 4/n should be examined closely (n is the number of observations). In our example, n = 34, so we are interested in projects for which D > 0.118. I find that one project, 51, has a Cook's distance of 0.147 (Example 1.28).
Example 1.28
. list id size effort t14 cooksd if cooksd>4/34
id size effort t14 cooksd
28. 51 1526 5931 3 .1465599
Why do I use Cook's distance? I use it because my statistical analysis tool calculates it automatically after ANOVA procedures. Other statistics, DFITS and Welsch distance, for instance, also summarize residual and leverage information in a single value. Of course, the cut-off values are different for DIFTS and Welsh distance. Do not complicate your life; use the influence statistic that your statistical analysis tool provides.3
Referring back to Figure 1.8, I see that the influence of Project 51 is due to its effort being slightly low for its size compared to other projects, so it must be pulling down the regression line slightly (leverage problem). After looking closely at this project, I see no reason to drop it from the analysis. The data is valid, and given the small number of large projects we have, we cannot say that it is an atypical project. If we had more data, we could, in all likelihood, find more projects like it. In addition, 0.15 is not that far from the 0.12 cut-off value.
If a project was exerting a very high influence, I would first try to understand why. Is the project special in any way? I would look closely at the data and discuss the project with anyone who remembered it. Even if the project is not special, if the Cook's distance is more than three times larger than the cut-off value, I would drop the project and develop an alternative model using the reduced data set. Then I would compare the two models to better understand the impact of the project.