IV. CONDUCTING THE REVIEW

A. Tripartite Quality Review System.

This section provides a description of the tripartite nonmonetary determination quality review system, the procedures for conducting the reviews, and the method for reconciling scores. Each quarter the samples selected by the SESA will be reviewed by a team comprised of nonmonetary determination evaluation experts using the tripartite quality review system.

The core requirements of the tripartite quality review process include the following:

A number of variations of the participants in the tripartite process can be found in Appendix B, and others are also possible, including cooperation of States from other Regions. The process has been planned to allow for maximum flexibility, within the core requirements as explained above.

1. Identifying Review Teams.

In at least one quarter each year, an on-site review must be performed for each State by a review team comprised of one QPI expert from the State being reviewed (Host State), one QPI expert from another State in the same Region (Other State), and one QPI expert from the Regional Office. This team composition, if resources permit, may be used for each quarter. In some instances, National Office staff may replace or accompany the Regional Office participant.

The tripartite quality review team may conduct on-site or off-site reviews for the other three quarters using three QPI experts, preferably with staff from the Host State and other State(s) within the same Region. See Appendix B for options for conducting the quarterly tripartite reviews (on or off-site) including examples of review team configurations and schedules. Selecting the review option requires advance consultation between the States and Regional Office.

2. Assigning Cases.

Sampled cases selected for review should be assigned according to the tripartite review option being used for the quarter. For on-site reviews, the sampled cases should be held in the Host State until the date of review. If the review is off-site, copies of the sampled cases should be mailed directly to the other State or Regional Office reviewers.

3. Reviewing and Scoring Cases.

The scores of the first reviewer should not be disclosed to the second reviewer prior to his/her independent review of the same cases.

Once the two reviews are completed, the two reviewers must compare their results element by element. The two reviewers must agree on the outcome of each element evaluated before an official score is entered into the database. If the reviewers do not agree, the case must be provided to the tie-breaker for an independent evaluation and reconciliation with one of the other reviewers.

The total score for determining nonmonetary determination quality is based on a 100 point scoring system. Six quality elements are evaluated. Points per element range from 0-50. The score of certain elements directly affects the score on other related elements. For example, if the adjudicator failed to obtain or make a reasonable attempt to obtain relevant and critical information from the claimant/employer/others, the appropriate element is scored "not obtained" (0 points out of a maximum of 10 points). Because the missing information is critical, the proper application of Law and Policy is questionable, at best, and a score of only 30 out of a potential maximum of 50 points for law and policy is allowed.

4. Reconciling Scores.

When two reviewers disagree on the outcome for any one of the elements evaluated, and they cannot reconcile the outcomes, that case will be independently reviewed by the third reviewer. The third reviewer must not be informed of the scores of the first and second reviewers. When the third reviewer completes his/her review, all three reviewers should discuss their results for each disputed element and their reasons for the results. This process provides each reviewer with the opportunity to convince (based on supportable evidence from the case materials) the other reviewers to alter their results. At least two of the three reviewers must be in complete agreement on the results for each of the elements. The State receives the score of the majority as the official score for the case. The Host State will enter the official scores for each reviewed case into the UI Required Reports (UIRR) database for transmittal to the National Office, and, at that point, the results are regarded as final.

5. Automatic Calculation of Score.

Review results for each case are entered on a hard copy score sheet by the review team. Once the case outcomes are resolved through the tripartite review, the official outcome is entered into data entry screens on the SESA's SUNsparc machine. It is not necessary to manually calculate the quality score for each case reviewed. When all the data is entered for a completed case and the case is saved in the database, a review edit module is initiated to ensure that the data for each element is acceptable. If any unacceptable entries exist, warnings will be displayed. Cases cannot be transmitted until all errors have been corrected. The database is then updated with the completed case data. At the time all case data are transmitted to the National Office UIRR database, a weighted overall score is calculated for the review period and displayed on the SESA screen. Because the weighting formula requires the exact number of separation and non-separation determinations reported for time lapse, the transmit program must be able to access the SESA's electronic ar9052 reports for each month in the review quarter.

Cases where the adjudication material is not found will NOT be factored into the quarterly score for quality. This data will be analyzed to determine if a State may be experiencing records management problems and to assess the potential impact such cases may have had on the quarterly score if the case material had been available to the reviewers.

If the total number of cases where the case material is not found, cases outside the scope of this review, and/or invalid cases exceeded the predetermined threshold for calculating statistically reliable results, a message will be generated stating that the scores for the quarter are inconclusive.

6. Retain All Case Reviews.

Current requirements for SESA retention of reported data apply. Hard copies of the Data Collection Instruments from all reviewers should be retained by the SESA for future reference. This information, particularly the "Comments" sections, will be helpful in identifying and resolving any inconsistencies in scoring outcomes and in reviewing data validity questions.

7. Use of Sample Data.

Nonmonetary determination performance will be tracked over time to determine, among other things, trends in performance, problems with particular facets of the nonmonetary process, timely implementation of nonmonetary determinations, etc. Each quarter's results will be compared to prior periods of performance to determine if improvement has occurred, particularly if interventions were introduced by the SESA to correct identified performance deficiencies. The data may also be used by State and Federal managers to determine if factors such as fluctuations in the business cycle, changes in personnel, changes in administrative procedures, technological changes, or other conditions affect nonmonetary determinations performance.

B. Completing the Data Collection Instrument.

Each case will be reviewed and completed in its entirety, with two exceptions:

1. when case material cannot be found for a sampled nonmonetary determination; or

2. when a case is selected that should not have been included in the sample frame because it is established that the case is either:

(a) invalid because it does not meet the definition of a nonmonetary determination as described in the ETA 207 reporting instructions contained in ET Handbook 401:

"A decision made by the initial authority based on facts related to an 'issue' detected:

• which had the potential to affect the claimant's past, present, or future benefit rights; and
• for which a determination of eligibility was made."
  
  

(b) outside the scope of the review, e.g., nonmonetary redeterminations, BPC crossmatches on uncontested earnings, DUA, TRA, EB.

Although nonmonetary redeterminations are not evaluated, they are considered valid cases for purposes of estimating the number and percentage of cases meeting the data validity criteria. If a nonmonetary redetermination is selected in the sample, the reviewer will enter N in element 4, and "01" in element 5, to signify that the case is a nonmonetary redetermination. No further review of that case is necessary.

Cases where the case material is not found, invalid cases, and those outside the scope of the quality review are NOT included in the calculation of quarterly nonmonetary determinations quality scores. Built into this calculation is a function that determines the threshold which the number of cases in these situations cannot exceed in order for the quarter's results to be statistically reliable. The frequency of occurrence when the case material is not found may indicate a problem with a SESA's records management system. Similarly, significant numbers of invalid cases drawn in the sample may signify a SESA problem with identifying issues that do not meet data validation criteria, i.e., are not countable for workload.

The UI automated system will generate a "show score" screen of the numbers and percentages of cases where the case material is not found and the number and percentage of invalid cases. The screen will display:

1. Total cases drawn in the sample.

2. The numbers of cases where the case material was not found.

3. The number of cases that are outside the scope of the review, including nonmonetary redeterminations, and are invalid.

4. The total number of cases scored.

5. The scores for each stratum (separation and nonseparation determinations).

6. The overall weighted score for all valid nonmonetary determinations reviewed and scored.

7. A message stating that the scores for the quarter are inconclusive because the total number of cases where case material is not found, cases outside the scope of the review, and/or invalid cases exceeded the threshold for calculating statistically reliable results.

8. For data validation:

(a) the number of invalid cases in the sample; and

(b) the percentage of sampled cases that are invalid.

All of this information is accessible in the UI database, where it is stored in the ar9056t, the "transmit" table.