Multiple Imputation in Stata: Estimating

This article is part of the Multiple Imputation in Stata series. For a list of topics covered by this series, see the Introduction.

In most cases, the hard work of using multiple imputation comes in the imputation process. Once the imputations are created and checked, Stata makes estimation using the imputed data relatively easy.

mi estimate

The main command for running estimations on imputed data is mi estimate. It is a prefix command, like svy or by, meaning that it goes in front of whatever estimation command you're running.

The mi estimate command first runs the estimation command on each imputation separately. It then combines the results using Rubin's rules and displays the output. Because the output is created by mi estimate, options that affect output, such as or to display odds ratios, must be applied to mi estimate rather than the estimation command. Thus:

mi estimate, or: logit y x

not:

mi estimate: logit y x, or

mi estimate has a list of estimation commands for which it knows Rubin's rules are appropriate. If a command is not on that list, you can tell mi estimate to apply them anyway with the cmdok ("command ok") option. However, it is your responsibility to ensure that the results will be valid.

Subsamples Based on Imputed Variables

Consider a regression like:

mi estimate: reg wage edu exp if race==1

If race is an imputed variable, then some observations will likely have a one for race in some imputations and not others. Thus the subsample to be used will vary between imputations, and mi estimate will give you an error message.

You have two options at this point. One is to simply tell mi estimate to ignore the problem with the esampvaryok option. The Stata documentation says this may result in "may result in biased or inefficient estimates" but we don't have any guidance at this time as to the seriousness of the problem.

mi estimate, esampvaryok: reg wage edu exp if race==1

The other is to not use observations that have imputed values of the variables used to select the subsample. Hopefully you created indicator variables telling you which observations are missing which variables in the process of determining whether your data are MCAR, MCAR or MNAR with:

misstable sum, gen(miss_)

If so, you can use those variables as part of your subsample selection:

mi estimate: reg wage edu exp if race==1 & !miss_race

Of course this raises the same issues as complete cases analysis, though the effects will likely be smaller.

Dropping Variables

More rarely, you could run into problems with different imputations using different sets of variables. In our experience that's been the result of perfect prediction in some imputations and not others, which suggests problems with the model being run (such as too many categorical covariates for the number of observations available). But it can also arise from the estimation command choosing different base categories. In that case specifying the base category should fix the problem.

Postestimation

Postestimation with imputed data must be done with caution. Rubin's rules require certain assumptions to be valid, notably asymptotic normality, and if a quantity does not meet those assumptions then Rubin's rules cannot provide a valid estimate of it. Fortunately, regression coefficients do meet those assumptions. Some quantities can be estimated if they are transformed to make them approximately normal, such as R-squared values. Others simply cannot, such as likelihood ratio test statistics. See White, Royston, and Wood for a list of quantities that can and cannot be combined using Rubin's Rules.

Unlike standard estimation commands, mi estimate cannot save all the information needed for postestimation tasks in the e() vector. Some tasks require the e() vector from the regression run on each completed data set. If you're planning to do postestimation, tell mi estimate to store the needed information in a small file with the saving() option:

mi estimate, saving(myestimates, replace): ...

This will create the file myestimates.ster in the current directory.

Tests of Coefficients

Hypothesis tests on coefficients can be performed using the mi test command. For testing whether coefficients are equal to zero, the syntax is the same as the regular test command. However, testing transformations or combinations of coefficients is more complicated—type help mi test for more information.

Likelihood ratio tests cannot be performed with multiply imputed data. However, if your goal is to test whether adding covariates improves your basic model, you can test the hypothesis that the coefficients on all those additional covariates are jointly zero.

Prediction

Predicted values can be treated as parameters to be estimated. Linear predictions meet the assumptions of Rubin's rules and thus they can be computed for each imputation and then combined as usual. This is done using the mi predict command, but mi predict needs the additional information contained in the estimates file saved by mi estimate. Thus the full command is:

mi predict myprediction using myestimates

Predicted probabilities do not meet the assumptions of Rubin's rules. However, you can estimate predicted probabilities by first estimating the linear prediction using mi predict and then putting the result through an inverse-logit transformation:

mi predict linear_prediction using myestimates, xb
mi xeq: gen predicted_probability=invlogit(linear_prediction)

The xb option tells mi predict to calculate the linear prediction even if the most recent regression involved probabilities.

Monte Carlo Error and the Number of Imputations

Since multiple imputation includes a random component, repeating the same analysis will give slightly different results each time (unless you set the seed of the random number generator). This is obviously an undesirable property, but acceptable as long as the amount of variation is small enough to be unimportant. The variation due to the random component is called the Monte Carlo error.

mi estimate with the mcerror option will report an estimate of the Monte Carlo error in estimation results. The process for calculating it involves leaving out one imputation at a time. White, Royston, and Wood suggest the following guidelines for what constitutes an acceptable amount of Monte Carlo error:

  1. The Monte Carlo error of a coefficient should be less than or equal to 10% of its standard error
  2. The Monte Carlo error of a coefficient's T-statistic should be less than or equal to 0.1
  3. The Monte Carlo error of a coefficient's P-value should be less than or equal to 0.01 if the true P-value is 0.05, or 0.02 if the true P-value is 0.1

If those conditions are not met, you should increase the number of imputations.

Example

Consider the example data set we imputed in an earlier section. It contains (fictional) wage data, which we will model using the covariates exp, edu, female, urban and race. Given what we found in the prior section, we will also interact female with exp and edu.

Data set to be analyzed (includes imputations)

Do file that carries out the analysis

Complete cases analysis (obtained with mi xeq 0: since the data set contains imputations) gives the following results:

mi xeq 0: reg wage female##(c.exp i.edu) urban i.race

m=0 data:
-> reg wage  female##(c.exp i.edu) urban i.race

      Source |       SS       df       MS              Number of obs =    1779
-------------+------------------------------           F( 12,  1766) =   98.93
       Model |  1.0350e+12    12  8.6247e+10           Prob > F      =  0.0000
    Residual |  1.5396e+12  1766   871809267           R-squared     =  0.4020
-------------+------------------------------           Adj R-squared =  0.3979
       Total |  2.5746e+12  1778  1.4480e+09           Root MSE      =   29526

------------------------------------------------------------------------------
        wage |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
    1.female |  -7033.218   4180.626    -1.68   0.093    -15232.71    1166.277
         exp |   2004.479    102.498    19.56   0.000     1803.449    2205.509
             |
         edu |
          2  |   10679.82   2634.031     4.05   0.000     5513.673    15845.96
          3  |   28279.73   2844.885     9.94   0.000     22700.03    33859.42
          4  |   51097.61   4591.212    11.13   0.000     42092.83    60102.39
             |
female#c.exp |
          1  |   -511.406   150.6243    -3.40   0.001    -806.8267   -215.9853
             |
  female#edu |
        1 2  |    -5736.4   4041.507    -1.42   0.156    -13663.04     2190.24
        1 3  |  -3876.886   4208.948    -0.92   0.357    -12131.93    4378.159
        1 4  |  -12072.54   5845.627    -2.07   0.039    -23537.62   -607.4622
             |
       urban |   4076.262   1577.229     2.58   0.010     982.8305    7169.694
             |
        race |
          1  |  -4409.319    1739.41    -2.53   0.011    -7820.838   -997.8001
          2  |  -4952.449   1790.243    -2.77   0.006    -8463.667   -1441.232
             |
       _cons |   31591.61   3200.808     9.87   0.000     25313.84    37869.38
------------------------------------------------------------------------------

Compare with results using the imputations:

mi estimate, saving(miexan,replace): reg wage female##(c.exp i.edu) urban i.race

Multiple-imputation estimates                     Imputations     =          5
Linear regression                                 Number of obs   =       3000
                                                  Average RVI     =     0.3261
                                                  Largest FMI     =     0.3672
                                                  Complete DF     =       2987
DF adjustment:   Small sample                     DF:     min     =      35.46
                                                          avg     =     206.66
                                                          max     =     710.00
Model F test:       Equal FMI                     F(  12,  516.3) =     122.88
Within VCE type:          OLS                     Prob > F        =     0.0000

------------------------------------------------------------------------------
        wage |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
    1.female |  -7637.933   3554.531    -2.15   0.034    -14676.67   -599.1996
         exp |   2003.071   82.22401    24.36   0.000     1841.639    2164.502
             |
         edu |
          2  |   12388.95   2190.569     5.66   0.000     8063.544    16714.36
          3  |   28619.32   2443.457    11.71   0.000     23764.93    33473.71
          4  |   51773.01   4248.138    12.19   0.000     43152.79    60393.23
             |
female#c.exp |
          1  |  -459.7754    130.917    -3.51   0.001    -719.6377   -199.9131
             |
  female#edu |
        1 2  |   -5981.89   3390.213    -1.76   0.080     -12676.1    712.3196
        1 3  |   -4640.03   3554.687    -1.31   0.194    -11672.93    2392.866
        1 4  |  -12926.75   5274.621    -2.45   0.018    -23517.89   -2335.615
             |
       urban |   4467.026   1465.326     3.05   0.004     1508.758    7425.294
             |
        race |
          1  |  -3221.866   1394.161    -2.31   0.021    -5960.173    -483.559
          2  |  -5977.193   1579.916    -3.78   0.000    -9123.292   -2831.093
             |
       _cons |   30617.76   2545.795    12.03   0.000     25614.32     35621.2
------------------------------------------------------------------------------

The 95% confidence intervals are smaller, which is just enough to put the P-value of female under the .05 cutoff for "significance."

These results were calculated with just five imputations, which we suggested as a starting point. How much Monte Carlo error does this leave?

mi estimate, mcerr: reg wage female##(c.exp i.edu) urban i.race

Multiple-imputation estimates                     Imputations     =          5
Linear regression                                 Number of obs   =       3000
                                                  Average RVI     =     0.3261
                                                  Largest FMI     =     0.3672
                                                  Complete DF     =       2987
DF adjustment:   Small sample                     DF:     min     =      35.46
                                                          avg     =     206.66
                                                          max     =     710.00
Model F test:       Equal FMI                     F(  12,  516.3) =     122.88
Within VCE type:          OLS                     Prob > F        =     0.0000

------------------------------------------------------------------------------
        wage |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
    1.female |  -7637.933   3554.531    -2.15   0.034    -14676.67   -599.1996
             |   614.5666   219.6112     0.22   0.018     810.9298    823.4979
             |
         exp |   2003.071   82.22401    24.36   0.000     1841.639    2164.502
             |    8.54593   1.876621     0.61   0.000     10.89688    7.715809
             |
             |
         edu |
          2  |   12388.95   2190.569     5.66   0.000     8063.544    16714.36
             |   347.7053   136.7152     0.24   0.000     182.1151    637.3576
             |
          3  |   28619.32   2443.457    11.71   0.000     23764.93    33473.71
             |   453.6681   191.0128     0.92   0.000     628.9118    692.7885
             |
          4  |   51773.01   4248.138    12.19   0.000     43152.79    60393.23
             |   1000.615   225.8875     0.44   0.000     494.3933    1592.194
             |
             |
female#c.exp |
          1  |  -459.7754    130.917    -3.51   0.001    -719.6377   -199.9131
             |   23.88987   6.507623     0.26   0.001     27.60468    30.27696
             |
             |
  female#edu |
        1 2  |   -5981.89   3390.213    -1.76   0.080     -12676.1    712.3196
             |   538.2219   166.8896     0.15   0.024     785.7183    522.4397
             |
        1 3  |   -4640.03   3554.687    -1.31   0.194    -11672.93    2392.866
             |   600.5342   309.3857     0.30   0.090     278.1348    1288.453
             |
        1 4  |  -12926.75   5274.621    -2.45   0.018    -23517.89   -2335.615
             |   1134.893   383.1226     0.21   0.012     1831.385    1154.305
             |
             |
       urban |   4467.026   1465.326     3.05   0.004     1508.758    7425.294
             |   331.6953   169.3962     0.58   0.006     715.5021    340.4445
             |
             |
        race |
          1  |  -3221.866   1394.161    -2.31   0.021    -5960.173    -483.559
             |   155.3758   26.42719     0.10   0.006     183.4623    145.6764
             |
          2  |  -5977.193   1579.916    -3.78   0.000    -9123.292   -2831.093
             |   305.4687   167.2478     0.61   0.001     251.5403    676.1527
             |
             |
       _cons |   30617.76   2545.795    12.03   0.000     25614.32     35621.2
             |   307.1376   84.98822     0.48   0.000     423.9462     281.053
------------------------------------------------------------------------------
Note: values displayed beneath estimates are Monte Carlo error estimates.

A brief glance at the estimates for female shows this does not meet the suggested criteria: the Monte Carlo error on the coefficient is about 17% of the standard error rather than 10%, and the Monte Carlo error on the P-value is .018 when we'd want it to be less than .01 if we believe the true P-value is .05 or less. This suggests you should use more imputations.

Even if it turned out that the Monte Carlo error was acceptable, we'd still recommend using more imputations in this case. About 40% of the observations are missing values, so White, Royston, and Wood would suggest 40 imputations. While that may seem like a lot, the entire process from imputation to analysis (including many diagnostics) still ran in less than 15 minutes in our testing. Thus there's no reason not to use at least that many imputations when you're ready to produce final results.

Here are the results with 40 imputations, and you'll see that the Monte Carlo error now meets the guidelines:

Multiple-imputation estimates                     Imputations     =         40
Linear regression                                 Number of obs   =       3000
                                                  Average RVI     =     0.2340
                                                  Largest FMI     =     0.2505
                                                  Complete DF     =       2987
DF adjustment:   Small sample                     DF:     min     =     494.37
                                                          avg     =     723.63
                                                          max     =    1046.30
Model F test:       Equal FMI                     F(  12, 2407.3) =     134.54
Within VCE type:          OLS                     Prob > F        =     0.0000

------------------------------------------------------------------------------
        wage |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
    1.female |  -8444.446   3568.764    -2.37   0.018    -15450.36   -1438.528
             |   242.2941   70.94422     0.08   0.004      294.817    266.1893
             |
         exp |   1981.635   90.55428    21.88   0.000      1803.76    2159.509
             |   6.812604   2.379668     0.61   0.000     9.595854    6.819707
             |
             |
         edu |
          2  |   11896.82   2277.964     5.22   0.000     7423.364    16370.28
             |   164.9826    51.1388     0.14   0.000     200.7386    187.6542
             |
          3  |   28702.41   2405.316    11.93   0.000      23980.9    33423.92
             |   160.1252   56.74204     0.31   0.000     218.7132    170.9788
             |
          4  |   52000.64   3991.622    13.03   0.000        44158    59843.27
             |   310.4559   94.26391     0.35   0.000     426.0258    288.6724
             |
             |
female#c.exp |
          1  |  -435.3699   126.7969    -3.43   0.001     -684.175   -186.5648
             |   7.621008   1.908626     0.08   0.000     8.635785    8.400978
             |
             |
  female#edu |
        1 2  |  -5278.077   3463.037    -1.52   0.128    -12076.34    1520.188
             |   234.4022   91.09243     0.07   0.017     330.3471    259.2769
             |
        1 3  |  -4498.973   3525.953    -1.28   0.202    -11418.57    2420.619
             |   220.3948   79.45707     0.07   0.025     265.3468    277.5948
             |
        1 4  |   -11832.2   4963.477    -2.38   0.017    -21576.09   -2088.301
             |   336.8887   98.06887     0.08   0.004     383.8081    395.9982
             |
             |
       urban |   4301.578     1351.3     3.18   0.002     1648.864    6954.293
             |   91.36621   21.07064     0.09   0.000     104.6887    96.41936
             |
             |
        race |
          1  |  -3478.967   1540.658    -2.26   0.024    -6505.774   -452.1594
             |   118.6348   35.97534     0.11   0.006     120.0791    155.5802
             |
          2  |  -5657.502   1575.349    -3.59   0.000    -8751.359   -2563.646
             |   115.2371   35.82667     0.12   0.000     120.9807    149.2301
             |
             |
       _cons |   31156.05   2678.619    11.63   0.000     25898.25    36413.85
             |   176.8282   54.19707     0.23   0.000     177.3412    233.6436
------------------------------------------------------------------------------
Note: values displayed beneath estimates are Monte Carlo error estimates.

One reasonable question is whether the interactions are actually required, and with unimputed data one might use a likelihood ratio test to answer it. With imputed data you'll instead test whether the coefficients on the interaction terms are jointly equal to zero:

mi test 1.female#c.exp 1.female#2.edu 1.female#3.edu 1.female#4.edu

note: assuming equal fractions of missing information

 ( 1)  1.female#c.exp = 0
 ( 2)  1.female#2.edu = 0
 ( 3)  1.female#3.edu = 0
 ( 4)  1.female#4.edu = 0

       F(  4, 156.0) =    4.63
            Prob > F =    0.0015

Given that some of the terms are significantly different from zero on their own, it's no surprise that the joint test rejects the hypothesis that they are all zero.

Finally, if we wanted to calculate predicted wages, we would use the following:

mi predict wagehat using miexan

Note the use of the miexan.ster file (without needing to specify the extension) created by the initial mi estimate command. It contains the coefficients from the regressions on each completed data set, which are needed to form the individual predictions combined by mi predict.

Next: Examples

Previous: Managing Multiply Imputed Data

Last Revised: 10/12/2012