Analysis of the Current Employment Statistics program using customer outreach survey results

The Current Employment Statistics (CES) program of the Bureau of Labor Statistics (BLS) was selected by the BLS Division of Management Services to participate in an internal program review. As part of this review, the CES program—which produces monthly estimates on employment, hours, and earnings using payroll records from a sample of business establishments—conducted an outreach survey in early 2014 of its stakeholders to gauge customer satisfaction of the program’s outputs and measurements. This article discusses the survey design and the classification of data users, and focuses on the analysis of data users’ survey responses.

Survey design

The outreach survey was designed in late 2013 by upper-level CES management and the statistical research methods staff in the BLS Office of Survey Methods Research. The survey was made available to the public on January 6, 2014. It included 34 questions asking users about CES data and outputs as well as the CES program’s customer service. (See appendix table A-1 for a list of survey questions.) The survey was available for public response for approximately 6 weeks; it closed on February 19, 2014. In that time, a total of 8,867 people responded to the survey.

CES prompted both known CES data users and CES data providers¹ via email to take the survey. In addition, a link to the survey was provided on the public CES home page, www.bls.gov/ces. So that we could meaningfully analyze the responses, people who responded to the survey were divided into CES data users and CES data providers.² Because the outreach survey participants were not directly asked whether they were data users or data providers, they were classified into a category on the basis of their responses to the survey questions. We performed additional analysis on the two groups of outreach survey respondents to verify that the results were what was expected given the classification. This classification resulted in 3,083 data users and 5,784 data providers.

Profiling data users

With more than 3,000 survey respondents classified as data users (regardless of whether they use CES data or only other data), we wanted a further distinction between users to aid in analyzing the survey results. So, data users were classified as either basic, intermediate, or super users (explained below) and additionally as state-and-area data users, national data users, or users of both state-and-area and national data.

Classification of data users. The basic, intermediate, or super user classification—or BIS—is a distinct classification, with each user assigned to only one of these classifications. Users answered questions that indicated whether they had characteristics of a basic, intermediate, or super user of CES data. The set of questions used to classify the data users were about types of information accessed, data revisions, historical data, and the use of similar data not produced by BLS. (See appendix table A-2 for detailed question-and-answer classification information.)

We finalized the BIS user classification by assigning the maximum classification identified across each of the question–answer pairings. BIS user status, for each outreach respondent classified as a data user, was identified by each question-answer pairing by the following scale:

• 1 indicates basic users
• 2 indicates intermediate users
• 3 indicates super users

Data users are split fairly evenly across the three different classifications. (See table 1.)

State-and-area and national user classification. In addition to being classified as basic, intermediate, or super users, users were classified into state-and-area data users and national data users. These classifications, unlike the BIS classifications, are not distinct; a user can be classified as a state-and-area data user and as a national data user. State users were identified by keywords in any open-ended response portion of questions and by responses to this question: “Which level of geographic detail do you use for CES data?” National users were identified solely by their responses to the question.

A total of 1,564 data users were classified as state-and-area data users and 625 data users were classified as national data users. Throughout the analysis, state-and-area and national user opinions are presented separately when the two diverge.

Analyzing stakeholders’ survey responses

An analysis of responses to the first few survey questions underscores that the surveyed data users use many of the CES data products available to them. The large majority of data users are interested in employment data, with data about women employees used more frequently than data about production employees. Hours and earnings data are used moderately, while diffusion indexes and indexes of aggregate hours are not used often and are mostly used by super users. (See figure 1.) Additionally, major and industry sectors are the most used levels of industry detail across all users.

View Chart Data

Figure 1. Responses to the Current Employment Statistics (CES) customer outreach survey question, “Which BLS CES data have you used?”

Type of data	Type of data user
	All users	Basic users	Intermediate users	Super users
Employment	2,280	468	972	840
Production and nonsupervisory employees	568	126	223	219
Women employees	1,230	270	561	399
Average hourly or weekly earnings	625	90	230	305
Average weekly hours	468	74	151	243
Indexes of aggregate weekly hours	123	7	35	81
Diffusion indexes	29	0	1	28
I have not used BLS CES data	559	484	75	0
Source: U.S. Bureau of Labor Statistics.

View Chart Data

 

Figure 2. Responses to the Current Employment Statistics (CES) customer outreach survey question, “Which level of geographic detail do you use for CES data?”

Type of data	Type of data user
	All users	Basic users	Intermediate users	Super users
National	626	90	212	324
State	1,256	241	519	496
Metropolitan Statistical Area	395	34	141	220
Metropolitan division	236	39	63	134
Other	307	58	139	110
Source: U.S. Bureau of Labor Statistics.

In terms of geographic detail, statewide data are the most frequently used; national data are used at about half the frequency of statewide data, and metropolitan-area data are used even less. (See figure 2.) Numerous users commented that they would prefer more granular geographic data, such as at a county or zip code level.

News releases and BLS databases are the most frequently used CES publications or files by all data users. By the criteria we employed to classify users, basic users rarely use other publications or files provided by CES. In addition to using news releases and BLS databases, intermediate users frequently use tables and charts as well as the Current Employment Statistics Highlights. Super users use all products frequently with the exception of the strike report. (See table 2.) Some users noted that they use external sources to get CES data, such as the ADP National Employment Report and the St. Louis Federal Reserve Board’s Federal Reserve Economic Data (FRED) database.³

Data users largely want CES data for personal interest reasons. Intermediate users employ the data for short- and long-term forecasting, economic research, and educational uses. Along with the uses identified by intermediate users, super users often use CES data in business cycle analysis, financial markets modeling and forecasting, economic policy analysis and planning, and news reporting. (See table 3.)

How and how often are users accessing CES data? The overwhelming majority of data users access CES data from the BLS website. Intermediate and super users also access data via email or phone requests to BLS staff and state labor market information websites. Very few users identified “private data supplier/ repackager” or social media as ways they access the data.

By definition, super users access CES data far more frequently than their intermediate and basic user counterparts. Only 11 percent of super users had not accessed the data from BLS website in the last 12 months. About 58 percent of super users had accessed the data at least six times in the last year. Approximately 80 percent of intermediate users accessed the data at least once in the last year, but none accessed the data more than 12 times. Basic users accessed CES data very infrequently, none more than five times per year and many noting they hadn’t accessed CES data at all in the past 12 months.

Most users do not need or want to be notified when BLS releases CES data and publications, and those who do want or need to be notified prefer to be notified by BLS News Service emails. CES currently offers an email service to users when updates to the CES program are made; users can sign up on the CES home page by adding their email address in the “Subscribe to the CES update” box on the left side below the navigation links.⁴

Which measures of changes over time are useful? CES data, which is published by level rather than percent, can be used to calculate changes, such as over-the-month changes. Determining which measurements of changes over time are considered useful by data users is important for answering whether CES is providing the most useful calculations. The CES program asked data users about the analytical usefulness of the following measures: over-the-month change in level, over-the-month change in percent, over-the-year change in level, over-the-year change in percent, and current level.

The majority of super users find each of these measures at least somewhat useful. Intermediate users are split between the usefulness of these measures, and most basic users find the measures either not useful or they report having no basis on which to decide.

Additionally, users mentioned they are interested in measures of the change in the data from the most recent peak or trough (or another anchor point) as well as quarterly levels, quarterly changes, and average changes over the past few months.

Are users satisfied with revisions to CES data? To reflect the receipt of additional sample data and other information, the CES program revises each month of data several times. To provide a preliminary estimate of employment, the program publishes initial monthly estimates before all the reports have been sent in by participating businesses. When more reports have been received, a revised or updated set of statistics is published. Additional revisions are made to adjust the data for seasonal variations and the annual benchmark process, which ties the survey results to the Quarterly Census of Employment and Wages.⁵

As would be expected, the intermediate and super users are more familiar with the revisions to the CES data than basic users, and super users are the most knowledgeable. (See figure 3.) All basic users either skipped this question or answered that they were “not at all knowledgeable.”

View Chart Data

Figure 3. Super data users’ responses to the Current Employment Statistics (CES) customer outreach survey question, “How would you rate your knowledge of the revisions made to CES data?”

Response	Number
Very knowledgeable	150
Somewhat knowledgeable	467
A little knowledgeable	124
Not at all knowledgeable	89
Source: U.S. Bureau of Labor Statistics.

Because basic users are not knowledgeable about the revisions to CES data, the following analysis will focus on the intermediate and super users’ responses to the revisions questions.

Far more data users, both intermediate and super, noted that revisions improve the quality of CES data than the number who noted that revisions lower the quality. Of those that said revisions improve the quality (352 intermediate users and 472 super users), a large portion of the intermediate users (about 70 percent) said the revisions either somewhat improve quality or improve quality a great deal and the vast majority (about 80 percent) of super users noted that the revisions either somewhat improve quality or improve quality a great deal. Both state-and-area users and national users have similar opinions of how the revisions improve data quality.

Of the data users who said it lowers the quality (27 intermediate users and 30 super users), about 75 percent of intermediate users and about 95 percent of super users noted that it lowers the quality somewhat or a great deal. More state-and-area users stated that revisions lower data quality a great deal, somewhat, or a little than did national users.

Data users responded similarly for both seasonally adjusted and not seasonally adjusted data when asked about the acceptability and appropriateness of the number of times each set of data were revised. Most intermediate users neither agree nor disagree that the number of seasonally adjusted and not seasonally adjusted revisions is acceptable and appropriate. The majority of the remainder either agree or strongly agree, with only a few disagreeing or strongly disagreeing that the number of revisions is appropriate. Over half of super users agree or strongly agree that the number of both seasonally and not seasonally adjusted revisions is acceptable and appropriate as is. A small minority of super users disagree that the revisions are acceptable. (See table 4.)

Data users were then asked to identify whether fewer or more revisions to the seasonally adjusted data would be helpful and whether revisions of any kind would be problematic for their analyses. The majority of both the intermediate and super users neither agreed nor disagreed with having additional revisions made to seasonally adjusted data. That said, super users are more interested than intermediate users in having more revisions to CES seasonally adjusted data to assist in data analysis. Similarly, for both groups of users, the majority of users do not have an opinion either way when it comes to fewer seasonally adjusted revisions. However, more super users are interested in fewer revisions than are intermediate users. (See table 5.) Very few users in either classification noted that revisions of any kind would be problematic to their analysis.

Overall, CES data users with knowledge of revisions find that the revisions do improve data quality. There was no overwhelming call to have more or fewer revisions to the data; some users would prefer more and some would prefer fewer, but across most responses, users did not have strong opinions either way.

Would users be interested in additional versions of CES data? Currently, the CES program provides data users with only the currently published versions of the data in the LABSTAT databases. In order for data users to access previously published versions of the data (that is, the data prior to the most recent revisions), users must look through old press releases and gather that information release by release.

To determine whether data users would find easier access to previous versions of data useful, the question was posed in the survey. Both intermediate and super users identified that access to data as originally published would be a little useful, somewhat useful, or very useful (about 57 percent of intermediate users, about 76 percent of super users). About 6 percent of intermediate users and about 19 percent of super users responding said that previous versions of data would be very useful. (See table 6.) Clearly, data users are interested in easier access to previous versions of CES data, and BLS plans to release vintage CES data to address this need in 2015.

Are users satisfied with accessing data and publications? The CES program provides data to its data users via numerous products, including

• data in databases
• research articles
• news releases
• documentation of methods and processes

CES attempts to provide each of these outputs to users in a way that is easy to access and understand. In order to evaluate whether CES is meeting its goal, the outreach survey asked data users about each of these outputs.

Intermediate and super users identified that data are available for the geographic areas they require; basic users did not have a basis on which to decide. Similar results were reported for available industry sectors, ease of downloading, file format availability, and finding data release schedules.

Super users noted that articles and research summaries are available on topics that interest them; basic users did not have a basis on which to decide and intermediate users were unsure. Intermediate users were more unsure (neither agree nor disagree) of whether these articles and research summaries are easy to understand than were the super users.

The uncertainty seen among intermediate users in the previous analysis is different across the state-and-area and national classification. Looking at the intermediate users within each of these classifications, we see that state-and-area users are more uncertain of the availability of articles and research summaries than are the national users. The difference between those who agree that articles and research summaries are available and those who neither agree nor disagree is much larger for state-and-area users than national users. A similar pattern is seen regarding the understanding of these articles and research summaries.

Like with articles and research summaries, far more intermediate and super users agree that news releases and documentation explaining industry employment concepts, sources, and methods are easy to understand and provide useful information than disagree, though intermediate users are more unsure (neither agree nor disagree) in both cases than super users are. Basic users report not having a basis on which to decide.

Data users who don’t use CES data. In order to increase the number of CES users, we must understand why some people don’t use the data. Compared with the entirety of data users who responded to this survey, few marked that they do not use CES data and the majority of these are classified as basic users. Of those who responded to this survey and don’t use CES data, some said that the “data are difficult to locate/BLS CES-related web pages are difficult to navigate” or data are not available for the industry or sector they require. Many also said that they were unaware of the data or the data have no application to their work. (See figure 4.)

View Chart Data

Figure 4. Responses to the Current Employment Statistics (CES) customer outreach survey question, “Why don't you currently use BLS CES data?”

Reason	User status
	All users	Basic users	Intermediate users	Super users
I was unaware of the data	41	30	11	0
Data have no application to my work	32	26	6	0
Data are not available for the industry	45	29	16	0
Revisions to data are problematic	7	4	3	0
Data are not explained and/or presented clearly enough	39	28	11	0
Data are difficult to locate	49	29	19	1
Data are not in the file format I need	9	9	0	0
Note: responses are from users of statistical data who do not use CES data. Source: U.S. Bureau of Labor Statistics.

Are users satisfied with the customer service? The CES office tries its best to respond to user questions by phone and email in a courteous and timely manner. Economists are assigned shifts throughout the month to help answer data users’ questions as they come in. To gauge whether the program is succeeding at its goal of good customer service, users were asked about their experience with CES staff.

Of the users who answered they had contacted CES staff for help with CES data (which is about a quarter of the users who answered the question), the vast majority contacted CES via telephone call. The website (“contact us” link) and direct email were used fairly equally, though at a much lower rate than telephone call within each user classification. (See figure 5.) The overwhelming majority of those who contacted BLS staff said the response was timely and that they were satisfied or very satisfied with the reply. (See figure 6.)

View Chart Data

Figure 5. Responses to the Current Employment Statistics (CES) customer outreach survey question, “How have you contacted BLS staff about CES data?”

Method of contact	User status
	All users	Basic users	Intermediate users	Super users
Website “contact us” link	127	18	38	71
Direct email	120	17	27	76
Telephone call	462	91	172	199
Other method	7	1	4	2
Source: U.S. Bureau of Labor Statistics.

View Chart Data

Figure 6. Responses to the Current Employment Statistics (CES) customer outreach survey question, “How satisfied were you with the reply (when you contacted BLS staff about CES data)?”

Level of satisfaction	User status
	All users	Basic users	Intermediate users	Super users
Very satisfied	278	44	106	128
Satisfied	192	37	67	88
Neither satisfied nor dissatisfied	29	8	11	10
Dissatisfied	14	6	5	3
Very dissatisfied	16	10	4	2
Source: U.S. Bureau of Labor Statistics.

In an open-ended response, the majority of responses were positive, with users mentioning that CES staff were knowledgeable, helpful, and polite during the interaction; few users mentioned negative interactions with staff.

Do CES users also use other industry employment data? About 15 percent of users who responded to this question identified that they do use other sources of industry employment data. The following is a list of other industry employment data sources that were mentioned: ADP, Job Openings and Labor Turnover Survey, Current Population Survey (particularly the unemployment rate), Business Employment Dynamics, Occupational Employment Statistics, Moody’s, U.S. Bureau of Economic Analysis, U.S. Bureau of the Census, Labor Market Information, Economic Research Institute, Federal Reserve, Global Insight, Industrial Supply Association, Quarterly Census of Employment and Wages, Local Area Unemployment Statistics, Bloomberg, FRED, County Business Patterns, Haver, Towers Watson, Hay, and Sullivan Cotter.

What is the affiliation of the users? A large majority of the data-user respondents to this survey consider themselves part of private business. In addition, numerous data users categorized themselves as part of “other.” Data users from state and local government and industry and trade association classifications were also well represented. (See table 7.)

Open-ended feedback. The final question on the survey asked data users to provide additional comments and open-ended feedback. Numerous data users mentioned that the BLS website is difficult to use, especially when it comes to downloading data for multiple series into Microsoft Excel. Some users suggested that templates, tools, or both be provided for obtaining large amounts of data. Furthermore, users are interested in accessing the microdata and in obtaining more information about the month-to-month revisions: what the revisions were, when they occurred, and why the data were revised.

Additionally, some users expressed their desire for more geographic breakouts and for vintage data. Regarding state and area series, users mentioned, for instance, separating mining and logging from construction in all states, adding residential versus nonresidential construction, and benchmarking data quarterly.

Many users commented that they were very satisfied with the work done in CES and thanked BLS and CES for performing the outreach survey. CES has taken all responses into consideration regarding how the program can be improved in the future.

Data providers. In an effort to understand data providers’ knowledge of the CES program, prompts were sent to them to participate in this customer outreach survey. Our analysis finds that the overwhelming majority of CES data providers are unfamiliar with the CES program and its outputs. This, paired with other feedback received from the survey, is being used to improve the data providers’ experience.⁶