Reliability of CES State and Area estimates

The Current Employment Statistics (CES) survey, like other sample surveys, is subject to two types of error—sampling and nonsampling. The magnitude of sampling error, or variance, is directly related to the size of the sample and the percentage of universe coverage achieved by the sample. Measurements of error associated with sample estimates for Statewide industries are provided below in Table 1 and for total nonfarm at the area level in Table 2.

Estimates for states and areas are produced using two methods. The majority of state and area estimates are produced using direct sample based estimation. However, published area and industry combinations (domains) that do not have a large enough sample to support estimation using only sample responses have been estimated using a small domain model.

Variance of the CES estimates. The sampling errors provided in this technical note are based on Generalized Variance Functions (GVFs).The Generalized Variance Function (GVF) connects the variance of an estimator to a set of variables the variance depends on, such as, the size of employment, the number of respondents, and the fraction of the population represented by the sample. Variances estimated directly from the sample may be unstable, especially in smaller domains. A model-based GVF is a way to stabilize direct estimates of variances.

The parameters of the model used for GVF are estimated by fitting the model to a set of direct replication-based variances obtained from the Repeatedly Grouped Balanced Half Samples (RGBHS) procedure. RGBHS is a modification of the Balanced Half Samples (BHS) technique used for the National estimates (see the Variance Estimation section in Chapter 2 of the Handbook of Methods). This modified technique involves the repeated calculation of BHS estimates R times. After each calculation, the sample units are randomly assigned to two groups in order to produce new BHS estimates. More information on the Generalized Variance Function can be found at https://www.bls.gov/osmr/research-papers/2013/pdf/st130050.pdf.

Under normal circumstances, where the average response rates are met and there are no reporting anomalies, the sampling errors represent a good indication of the magnitude of error in the estimates from sampling only. They do not represent components of error in the estimates that are due to reasons other than sampling; nonsampling error is captured in the benchmark revision.

The CES State and Area benchmark revision. To help control both sampling and nonsampling error, CES estimates are benchmarked annually to universe employment counts. These counts are derived primarily from employment data reported on unemployment insurance (UI) tax reports that nearly all employers are required to file with state employment security agencies. Benchmark levels replace the original sample-based estimates usually from April of the previous benchmark year to September of the current benchmark year for each month.

Once the new September level for the current benchmark level is made, the appropriate sample links are applied to the new level, and the estimates are recalculated from October forward. These sample links capture the over-the-month change of sample estimates, and are calculated by dividing employment reported by survey respondents for that month by employment reported by those same respondents for the previous month. The links used during the benchmark process may differ slightly from those used to derive original estimates, because they include data from respondents who reported too late for inclusion in the previously published estimates.

Appropriate uses of sampling variances in CES. Variance statistics are useful for comparison purposes, but they do have some limitations. Variances reflect the error component of the estimates that is due to surveying only a subset of the population, rather than conducting a complete count of the entire population. However, they do not reflect nonsampling errors, such as reporting errors, bias due to nonresponse, or the model error for series denoted as model-based. The overall performance of the all-employee estimates is still measured in terms of the benchmark revisions. Benchmark revisions are discussed in the last section of this technical note. For other data types published by the CES program, average hourly earnings and average weekly hours, variances are the best measure of accuracy. The variances of the over-the-month change estimates are very useful in determining when changes are significant at some level of confidence.

Illustration of the use of standard error tables. Tables 1 and 2 provide a reference for standard errors of CES State and area employment estimates. The standard errors of differences between estimates in 2 non-overlapping industries are calculated as

The errors are presented as standard errors.

Table 1 provides a reference for the standard errors of 1-, 3-, and 12-month changes in employment for NAICS supersectors by state. The errors are presented as standard errors of the changes.

Table 2 provides a reference for standard errors of CES area employment estimates for the total nonfarm industry. For all states and areas, the total nonfarm series is not estimated directly but is rather a sum of multiple detailed industries. The stratification of these industries varies across areas depending on the size and sample composition of each area. The column marked ‘Sample Based %’ is calculated using total private employment due to sampling differences in all government series.

Example:

Suppose that the over-the-month change in manufacturing employment from January to February in a given state is 1,500. The table indicates that the standard error for a 1-month change in employment for manufacturing is 1,200. The interval estimate of the over-the-month change in employment that will include the true over-the-month change with 90-percent confidence is calculated:

1,500 +/- 1.645(1,200) = 1,500 +/- 1,974 = [-474, 3,474]

The true value of the over-the-month change is in the interval -474 to 3,474. Because this interval includes 0 (no change), the change of 1,500 shown is not significant at the 90-percent confidence level. Alternatively, the estimated change of 1,500 does not exceed 1,974 (1.645 x 1,200); therefore, one could conclude from these data that the change is not significant at the 90-percent confidence level.