ADVISORY	:	TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 3-03
TO	:	ALL STATE WORKFORCE LIASONS ALL STATE WORKFORCE AGENCIES
FROM	:	EMILY STOVER DeROCCO Assistant Secretary
SUBJECT	:	Data Validation Policy for Employment and Training Programs

1. Purpose. To inform states and National Program grantees of the Employment and Training Administration’s data validation policy, and the plans to implement data validation requirements for the following programs: Workforce Investment Act Title IB, Labor Exchange, Trade Adjustment Assistance, Migrant and Seasonal Farmworkers, Native American Employment and Training, and Senior Community Service Employment Program. The data validation policy and supplemental guidance on implementing data validation will be conveyed to National Program grantees through the issuance of program bulletins.

2. References. Training and Employment Notice (TEN) No. 14-02, "Data Validation Initiative;" TEN No. 8-02, "Implementation of Common Performance Measures for Job Training and Employment Programs;" and President’s Management Agenda ( http://www.whitehouse.gov/omb/budget/fy2002/mgmt.pdf).

3. Background. The accuracy and reliability of program reports submitted by states and grantees using Federal funds are fundamental elements of good public administration, and are necessary tools for maintaining and demonstrating system integrity. The President’s Management Agenda to improve the management and performance of the Federal government has emphasized the importance of complete information for program monitoring and improving program results.

   States and grantees receiving funding under the Workforce Investment Act (WIA) Title IB, Labor Exchange, Trade Adjustment Act (TAA), and Senior Community Service Employment Program (SCSEP) are required to maintain and report accurate program and financial information [WIA Section 185 (29 USC 2935) and WIA regulations 20 CFR 667.300(e)(2); Wagner Peyser Act Section 10 (29 USC 49), Older Americans Act Section 503(f)(3) and (4) (42 USC 3056a (f)(3) and (4)), and TAA regulations 20 CFR 617.57]. Further, all states and grantees receiving funding from the Employment and Training Administration (ETA) are required to submit reports or participant records to the agency and attest to the accuracy of the data.

   Recent performance audits conducted by the Department of Labor’s Office of the Inspector General, however, found that the accuracy of reported performance outcomes cannot be assured due to insufficient local, state, and Federal oversight. To address this concern and meet the agency’s goal for accurate and reliable data, ETA committed to the development and implementation of a data validation process in order to ensure the accuracy of data collected and reported on program activities and outcomes.

   ETA has already begun implementing data validation requirements for the Unemployment Insurance program. ETA will separately issue requirements for reporting validation results for state programs under WIA Title IB, Labor Exchange, and TAA, and for the National Program grantees operating Migrant and Seasonal Farmworker (MSFW) programs, Native American Employment and Training programs, and SCSEP in fall 2003.

   States and grantees should be aware that there are legislative and policy changes on the horizon that will impact ETA’s performance accountability system. First, ETA plans to implement a set of common performance measures for job training and employment programs in CY 2004. Further information on the common measures can be found in TEN No. 8-02. Second, WIA was authorized for only five years, and the congressional process to pass reauthorizing legislation is underway. These legislative and policy changes will not significantly alter ETA’s data validation policy or the validation process. The validation tools developed by ETA, discussed later in this guidance, will be adapted to reflect changes to data collection and WIA reauthorization.

4. Employment and Training Administration Data Validation Policy.

   1. General. In order to ensure the accuracy and reliability of reported information, states and grantees are required to validate the data submitted to ETA. Failure to demonstrate the validity of reported data will be deemed to be a failure to report, and subject to corrective action or sanction, as appropriate.

   2. Scope. There are two basic sources of reporting error. First, if the data collected are incorrect or data entry errors occur, then the outcome information will not be accurate. Second, even if the data collected are correct, if the state’s or grantee’s reporting system does not meet Federal standards, it could calculate the performance outcomes incorrectly. Therefore, the ETA data validation requirement consists of two parts:

      1. Report validation evaluates the validity of aggregate reports submitted to ETA by checking the accuracy of the reporting software used to calculate the reports. Report validation is accomplished by processing an entire file of participant records into validation counts and comparing the validation counts to those reported by the state or grantee.

      2. Data element validation assesses the accuracy of participant data records. Data element validation is performed by reviewing samples of participant records against source documentation to ensure compliance with Federal definitions.

   3. Frequency. Data validation will be required annually as follows:

      1. Report validation must be completed prior to the submission of required reports to ETA (exceptions to this for the initial year of validation are noted in Sections 6 and 7).

      2. Data element validation must be completed within 120 days after required annual reports or participant records are submitted to ETA. Exact deadlines for the completion of data validation will vary somewhat by program.

   4. Accuracy Standards. For the data validation initiative to be effective and to allow for continuous improvement, ETA will establish acceptable levels for the accuracy of reports and data elements. These accuracy standards will be established in phases. The initial validation year will focus on detecting and resolving any issues with state and grantee data and reporting systems. Error rates collected in the second year will be analyzed, and based on this information, standards for accuracy will be established prior to the third year of validation.

      Once accuracy standards are established, states and grantees will be held accountable for meeting those standards and will be required to address any issues concerning data accuracy. States and grantees that fail to meet accuracy standards will receive technical assistance from ETA and will develop and implement a corrective action plan. Data that do not meet accuracy standards will not be acceptable for measuring performance, and may keep the state or grantee from being eligible for incentives that are awarded based on performance data. Significant or unresolved deviation from accuracy standards may be deemed a failure to report.

5. Description of ETA Data Validation Process. The ETA data validation process is intended to accomplish the following goals:

   • Ensure that critical performance data are accurate.

   • Detect and identify specific problems with a state’s or grantee’s reporting process, including software and data issues, to enable the state or grantee to correct the problems.

   • Help states and grantees analyze the causes of performance successes and failures by displaying participant data organized by performance outcomes. In addition, the process will allow states and grantees to select appropriate validation samples necessary to compute statistically significant error rates.

   ETA has developed a set of validation tools discussed in Section 8 – instructional handbooks, software, and user guides – that states and grantees can use to validate data. The ETA software can also be used to generate the aggregate information required in reports submitted to ETA. States and grantees that use the software provided by ETA to generate this aggregate information will not be required to perform report validation.

   States and grantees may use an alternative methodology and tools as long as the methodology meets criteria for sampling and confidence intervals. States and grantees that do not use the validation tools provided by ETA will be required to document that the alternative methodology is statistically valid.

   At present, two states have participated in a pilot and ETA is beginning to work with additional states that are interested in being early implementers of the data validation process. Early implementing states have been testing the validation methodology using PY 2001 data. All states and grantees are encouraged to begin testing validation as well.

6. Implementation for State Programs. Reporting instructions for data validation for state programs will be issued separately. The timeframes for implementation are as follows:

   • Workforce Investment Act Title IB. For PY 2002, each state is required to submit an annual file of WIASRD records by December 1, 2003, and an annual report (ETA 9091) by December 1, 2003. States will be required to complete report validation and data element validation using a file of exiters and participants reported on the 9091 by April 1, 2004. If states use the ETA validation software to generate the aggregate information included in their annual 9091 report, they will be considered to have conducted report validation.

   • Labor Exchange. Each state is required to submit the ETA 9002 and VETS 200 reports on a quarterly basis, 45 days after the end of the report quarter. The ETA 9002 and VETS 200 reports due on November 14, 2003, must be validated by April 1, 2004. The Labor Exchange requires report validation. A minimal data element validation, in which 25 files are compared to state-level data, must also be conducted to ensure that the files needed to perform report validation are properly constructed. States that use the ETA provided software to generate the aggregate information included in their ETA 9002 and VETS 200 reports will be considered to have conducted report validation.

   • Trade Adjustment Assistance. Each state is required to submit its fourth quarter file of TAPR records for FY 2003 by November 15, 2003. States are required to validate by April 1, 2004, a sample from the cumulative file of four quarters of FY 2003 TAPR records (created by adding the four quarterly files together). TAA requires only the submission of individual participant records, and not a report containing aggregate performance information, and therefore states are not required to conduct report validation.

   In subsequent years, states will be required to complete report validation for WIA Title IB and Labor Exchange prior to the submission of the reports listed above to ETA. However, in this initial year of validation, report validation will be completed in the same timeframe as data element validation – by April 1, 2004, which is approximately 120 days after the submission of the reports to ETA.

7. Implementation for National Grant Programs. Reporting instructions for data validation for national grant programs will be issued separately. The timeframes for implementation are as follows:

   • Migrant and Seasonal Farmworker Program. National Farmworker Jobs Program (NFJP) grantees are not required to submit an annual report but are required to submit their fourth quarter PY 2002 WIASPR records. All information except placement retention was due by August 15, 2003. Retention information is due by February 15, 2004. Data element validation of all PY 2002 data must be completed by June 15, 2004. NFJP grantees are only required to submit individual participant records, and not a report containing aggregate performance information, and therefore are not required to conduct report validation.

   • Native American Employment and Training Program. Grantees are required to submit their annual PY 2003 ETA 9085 Supplemental Youth Services Report by June 30, 2004 (90 days after the March 31st program year end date). Grantees are required to submit their annual ETA 9084 Comprehensive Services Report covering PY 2003 by September 30, 2004 (90 days after the June 30th program year end date). Grantees must complete data element validation for the 9085 and 9084 reports by January 31, 2005.

   • Senior Community Service Employment Program. SCSEP grantees are required to submit their annual ETA 5054 reports by December 31, 2004. Grantees must complete report validation prior to the submission of the ETA 5054 report. Data element validation must be completed by May 2, 2005. The ETA validation software will also generate the aggregate information included in the ETA 5054 report; if grantees use the software for this purpose, then they will be considered to have conducted report validation.

8. Data Validation Tools. ETA has developed instructional handbooks, standardized software, and user guides that states and grantees can use to perform data validation.

   • Handbooks provide detailed information on the validation methodology, including sampling specifications and data element validation instructions for each data element to be validated.

   • Software developed by ETA generates samples, worksheets, and reports on data accuracy. For report validation, the software will validate the accuracy of aggregate reports that are generated by the state’s or grantee’s reporting software and will produce an error rate for each reported count. The software can also be used to generate the aggregate information required in reports submitted to ETA. For data element validation, the software generates a sample of the participant records and data elements for the state or grantee to validate. The software produces worksheets on which the validator records information after checking the source documentation in the sampled case files. The software calculates error rates for each data element, with confidence intervals of 3.5 percent for large states/grantees and 4 percent for small states/grantees.

   • User guides developed for each ETA validation software application guide states and grantees through the process of installing the application, building and loading a validation file, and completing report and data element validation.

   ETA’s validation handbooks, software applications, and user guides for each program can be found on ETA’s Web site at http://www.doleta.gov/usworkforce/performance/. The validation tools for the WIA Title IB, Labor Exchange, and TAA programs are currently available; tools for the national grant programs will be posted as finalized.

9. Training and Technical Assistance. ETA is providing data validation training for states in regional sessions during the summer of 2003. Training for National Program grantees will be provided separately for each program. Training for grantees of the MSFW and Native American Employment and Training programs will be held during winter 2003/2004, and training will be provided for SCSEP grantees during spring 2004.

   States and grantees are encouraged to obtain technical assistance on validation procedures and the use of the validation tools by contacting ETA’s data validation contractor. To avoid unnecessary delays and problems in the implementation of data validation, the contractor is available to facilitate the validation process and to address any difficulties states and grantees encounter. E-mail and phone contact information is available on the ETA Web site listed above, in the Help menu of each software application, and in the user guides.

10. Monitoring. To ensure that states and grantees are able to successfully implement data validation procedures and standards, ETA will monitor the validation effort on a regular schedule. The record retention requirements that will apply to records of all validation activities, including files, worksheets, reports, and source documentation, will be articulated in the reporting requirements to be issued separately.

11. Action Required. States and grantees are expected to:

    • Distribute this advisory to the appropriate program staff.

    • Communicate interest in beginning validation implementation by contacting the appropriate regional office or national grant program office.

    • Notify the appropriate regional office or national grant program office if they intend to use a methodology to perform data validation other than the methodology developed by ETA. States interested in alternative methodology for validation of WIA Title IB, Labor Exchange, or TAA programs should notify the appropriate regional office by September 30, 2003.

    • Complete data validation within the timeframes established in Sections 6 and 7 of this guidance.

12. Inquiries. Questions regarding the data validation initiative should be directed to the appropriate regional office or national grant program office.