ATTACHMENT

Report: Study of the Measure of Nonmonetary Determination Quality in the Unemployment Insurance (UI) Program

Background. As part of the UI Performs five-year review, the Employment and Training Administration (ETA) convened a federal/state team to study and recommend changes to the Benefits Timeliness and Quality (BTQ) Nonmonetary Determination Quality Review. Twelve state subject matter experts, four staff from the National Office, and staff from each of the six ETA Regional Offices participated. Regional Office staff invited the state subject matter experts.

In preparation for the meetings, National Office staff summarized comments received in the course of the five-year review of UI Performs and canvassed the participants for additional issues/discussion points

The group convened for two three-day sessions in January and February 2004. The work included a review of the data collection instrument (DCI) and instructions used in the quality review of nonmonetary determinations, operational guidance that ETA gives the states regarding claim determination requirements, and other matters concerning the adjudication process. Particular effort went toward exploring ways to refine the measurement instrument, how determinations are scored, and instructions to the reviewers to ensure consistent and accurate review results. In the review of UI Performs only two alternatives to the scoring methodology were put forward. Washington State deserves special thanks for the time and effort of its staff in developing those proposals. Neither alternative approach was ultimately recommended and ETA remains open to suggestions in the area. The group’s recommendations on all decisions reflect the opinions of the majority of the group members. The group decided that strongly held conflicting opinions would be included in this report in addition to the majority opinions.

Scope. Discussion topics included:

Policy Considerations

Technical Changes

This review revealed that a substantial amount of the controversy surrounding the nonmonetary determination review process is the result of poor communications and unclear instructions. Clarifying and updating ET handbook 301 will be instrumental in promoting uniformity in the reviews and reducing confusion.

The following topics were briefly discussed but no conclusions/recommendations were reached. The National Office agreed to explore these topics more fully in the future.

Recommendations.

1. Policy Considerations

1. Minimum Criteria to Satisfy Reasonable Attempts Requirements
   Background: State agency experts review a sample of completed nonmonetary determinations each quarter to assess whether adjudicators obtained necessary information, applied law and policy correctly, and wrote an intelligible nonmonetary determination to send to the claimant and employer. The result of this expert review produces the nonmonetary determinations quality score, which is a Tier I measure in UI Performs.

   BTQ nonmonetary determination reviewers are instructed to give full credit to sampled cases if the adjudicators documented a reasonable attempt to obtain necessary information, even if they were unsuccessful in obtaining the information. These instructions derive from longstanding guidance that balances the desire for completeness against time considerations. Adjudicators must obtain sufficient information about UI claims to reasonably insure the proper payment of benefits; however, the investigation should not be so time-consuming as to delay unduly the payment of benefits. In 1996, ETA tried to resolve the tension between completeness and promptness by operationally defining a reasonable attempt to obtain information which states:

   Any deadline set for receipt of information before a determination is issued based on available evidence, must be reasonable. Generally, this would be the number of days normally allotted … for other activities, e.g., 5 days, 7 days, or 10 days, whether information is being requested in writing or by telephone, and in no case should be less than 48 hours.

   This definition became known as the 48-hour rule. ETA went on to delineate the required documentation of reasonable attempts made by telephone to include:

   • The time and date of the attempted contact;
   • Names and titles (if appropriate) of individuals with whom messages were left; and
   • The fact that the individual was informed of the consequences of failure to respond.
     (The consequence for failure to respond is that the decision will be made on the basis of information at hand.)

   From the definition of reasonable attempts to obtain information and the delineation of the documentation needed to support the attempts two issues arose.

   Issue 1: Through anecdotal information, ETA became aware that some cases were failing the quality review because the adjudicator completed the determination without needed information just short of the full 48 hours. ETA was unable to quantify the extent to which this situation occurs, but the possibility remains for otherwise acceptable cases to fail the quality review on this basis alone.

   Recommendation: Reasonable attempts should continue to be defined for nationally uniform application; however, the minimum time frame should change from 48 hours to close-of-business (COB) the next business day.

   Rationale: COB the next business day:

   • Permits a reasonable amount of time for claimant/employer/third party response in light of current state operations; and

   • Creates a clear demarcation of the period of time provided for response, thereby mitigating concerns that strict adherence to the 48-hour rule causes otherwise acceptable determinations to fail.

   Note: Although the study group generally agreed that COB the next business day was an acceptable timeframe for response, 40% of the group felt the 48-hour requirement should remain in place without change.

   Issue 2: ETA was informed of a perception that some cases were failing the quality review because the adjudicators failed to document that they had included the consequences of non response in telephone messages left for claimants, employers, or third parties.

   Recommendation: Eliminate the requirement that the consequences of failing to respond must be documented.

   Rationale: Informing claimants/employers/third parties of the consequences of non response is a standard practice for state adjudicators; therefore, documenting the fact that consequences were given in every instance is unnecessary. Failure to document whether consequences were given is not itself an indication of a poor quality determination.

2. Holiday Pay and Deductible Income Factfinding Requirements
   Background: When a claimant reports receiving a payment other than earnings from an employer that may be deducted from his/her weekly benefit amount, the adjudicator is required to contact the employer to verify the type of income, the amount, and the period to which the payment is allocated. Information provided by the claimant is not sufficient because the claimant may not be certain of the exact amount of the income, or the week to which it applies. Because of the large number of nonmonetary determinations involving earned income, the Claim Determination Standard published in the Employment Security Manual allows states to accept a claimant’s word as to the amount of earned income and the week to which it applies.

   Issue: When state law dictates the week to which holiday pay must be allocated, and the amount is treated as if it were earnings, the requirement to contact the employer serves little purpose.

   Recommendation: States that must allocate holiday pay to the week of the holiday would not be required to contact the employer for verification. The claimant would not be required to produce proof, for example a check stub, of the amount. His/her word would be sufficient.

   Rationale: State law obviates the need for verification when it establishes the week to which holiday pay applies. Further, holiday pay has the same characteristics as earnings, and like earnings, should not require verification.

3. Guidelines to Determine Quality of a Fully Automated Nonmonetary Determination
   Background: In an effort to be more efficient, some states have implemented automated systems that issue nonmonetary determinations on certain limited issues solely on the basis of claimants’ responses without adjudicator intervention. Issues concerning a claimant’s availability for work, or search for work, are often adjudicated in this manner in those states.

   Issue: State UI law in some, but not all, instances requires an adjudicator’s interpretation. Fully automated determinations raise concerns that the information gathered without assistance from an adjudicator may be insufficient to make high quality determinations, jeopardizing the proper payment of benefits. Consider the following examples. The claimant’s response to "Were you available for work?" requires an interpretation of the meaning of availability under state law; but the claimant’s yes or no response to "Did you look for work?" provides a conclusion that does not allow for interpretation.

   Recommendation: Automated nonmonetary determinations must meet all quality guidelines outlined in Chapter 6 of Handbook 301. The state should ensure that:

   • The fact finding contains all relevant and critical facts related to the issue. The automated system confirms the claimant’s response and gives the claimant an opportunity to change the response.

   • The automated system advises the claimant that his/her response raises an issue that will affect UI entitlement.

   • Facts must lead to only one conclusion on the issue; an adjudicator must intervene if they do not.

   Rationale: Claimants' rights must be protected as states seek efficiencies through the use of automated systems. State agencies have responsibility for interpreting state UI eligibility requirements and cannot shift the burden to the claimant.

4. Claimant’s Failure to Report or Contact the Agency as Instructed
   Background: This proposal applies to states that have a provision in law that allows the agency to consider good cause for failure to report for the following: participation in an eligibility interview, registration with the Job Service, reporting required under the WPRS (worker profiling) program, and reporting to provide information necessary for adjudication of an issue affecting rights to benefits. Currently, states that have good cause provisions in their law must contact the claimant to investigate the reason for the claimant’s failure to report before issuing a failure to report nonmonetary determination. States that do not have a good cause provision have never been required to contact claimants, since any information provided would have no bearing on the outcome.

   Issue: Once the claimant has been given written notice of his/her rights and responsibilities and clearly instructed to report, states consider the requirement to investigate good cause for failure to report excessive and time consuming.

   Recommendation: Claimant information should be considered adequate when evaluating the quality of the determination if a claimant is directed to report to the state agency, and

   • The notice advises the claimant of the date and time to report;

   • The notice advises the claimant of the consequences of failure to report;

   • The notice provides the claimant with the necessary information and the opportunity to contact the agency to explain reasons for failure to report and/or to reschedule; and

   • The notice advises that the agency will consider whether the claimant had good cause for failure to report as directed.

   Rationale: The claimant has been advised of his/her rights and responsibilities; therefore, the agency has met its responsibility.

5. Case Materials Not Found
   Background: Currently, if the state is unable to review a determination for quality because staff cannot locate the case materials, or the case materials are illegible, the case is disregarded. The quality score is then based on the reduced sample.

   Issue: Missing or illegible cases, which cannot be scored, present a potentially serious problem from a sampling viewpoint. It cannot be assumed that the ratio of pass to fail for these lost cases would be the same as the cases that were scored. The estimated percentage of cases passing the quality evaluations could, therefore, be biased. Also, excluding cases from scoring produces a less accurate estimate of the percentage of cases passing the quality evaluation because the sample size has been reduced.

   Recommendation: Fail any case for which the case materials are missing or illegible.

   Rationale: States should be able to produce legible case material for review. The consequences of not doing so are serious because these are legal documents and sometimes required for court proceedings. This recommendation will provide greater incentive for states to improve filing and retrieval systems.

   Note: A number of the group members felt that the issue of missing or illegible cases was unrelated to the quality of the nonmonetary determination. Other members of the group believed that we should continue the current practice of dropping the cases from the sample, or that we should require states to draw additional cases in subsequent quarters to make up for the missing ones.

6. Consistency in the Nonmonetary Quality Review Process
   Background: For review scores to be reliable indicators of the quality of nonmonetary determinations nationwide, different reviewers should typically arrive at the same result. To achieve this reliability, two experts review each case with a third expert assigned to break impasses. The process is referred to as the tripartite review. ET Handbook No. 301 gives several options that states can use to conduct the tripartite review.

   Issue: There are concerns that differing levels of review stringency are applied across the country when state and regional office nonmonetary determination reviewers do not have the benefit of an outside perspective.

   Recommendation: States must participate in one-cross regional review annually.

   Rationale: Cross-regional reviews help to ensure that the scoring process is as unbiased as possible and free of local or regional uniqueness. The reviews also provide a platform for sharing information about better practices.

2. Technical Changes

1. Streamline the DCI
   Background: The following recommendations concern the DCI, which is the worksheet that reviewers complete during the BTQ nonmonetary review. The current DCI has 27 elements: some are obsolete; others are confusing and create reporting errors. Recommendations would delete some, but add others as described below.

   Element #5
   Recommendation: Delete code 29 - Separation "Other"
   Report all claimant initiated separation issues as quits, and all employer initiated separations as discharges. Issues that had been erroneously reported in the other category would be reported as nonseparation issues.

   Rationale: All issues arising from a claimant’s separation from employment that affect his eligibility for UI benefits can be categorized as voluntary quit or discharged for misconduct. Having an "Other" code creates confusion and contributes to misreporting.

   Recommendation: Combine Suitable Work with Failure to Apply/Accept Referral into code 60. Eliminate code 71.

   Rationale: States’ laws do not differentiate between a claimant’s refusal of suitable work or his/her failure to apply for or accept referral to a job opening. Separating the two issues in the quality review serves no point and is a source of confusion.

   Recommendation: Delete code 89. Separate the issues currently reported under code 89 and give each its own code number. Reporting requirements will become code 31. Misrepresentation will become code 86. The "Other" category will be eliminated.

   Rationale: A separate code for Reporting Requirements and Willful Misrepresentation was inadvertently omitted when the BTQ nonmonetary determinations quality review was implemented. The "Other" category will be eliminated because all countable nonseparation issues should be included in an appropriate code. The states were queried regarding their use of the nonseparation "Other" code, and it was determined that having an "Other" code creates confusion.

   Element #11 – Week Claimed?
   Recommendation: Eliminate. This data element is obsolete.

   Rationale: In 1996, the ETA eliminated the requirement that the claimant must certify for a week of benefits before the state agency could issue a reportable nonmonetary determination. (See UIPL 10-96 Implementation of New Time Lapse and Quality Measures for Unemployment Insurance (UI) Benefits Operations.)

   Elements#15, 16, and 17 – Issue Detection Date
   Recommendation: Eliminate these data elements if the proposal to discontinue its use in the nonmonetary determination time lapse measure is accepted.

   Rationale: These elements validate the starting parameter for the nonmon time lapse measure. If that measure is changed as a result of the five-year review, then the data elements will no longer be needed.

   Element #21 – Rebuttal Opportunity Provided
   Recommendation: Eliminate.

   Rationale: Rebuttal is a subset of factfinding and should be a part of other scored items on the DCI. Having a separate item created unnecessary confusion.

   Element #24 – Appeal Information Provided
   Recommendation: Combine with written determination element.

   Rationale: The adequacy of the appeal information should be evaluated as part of the written determination. According to Part V of the Employment Security Manual, "…. (appeal) information must be included either in the notice of determination or in separate informational material referred to in the notice…."

   Element #25 – Date of Determination
   Recommendation: In the new numbering of the DCI, old element #25 will become new element #4. Element #5 will be changed to "Correct Date on Determination? (Y/N)." Element #6 will be changed to "Corrected Date on Determination."

   Rationale: The reviewer should verify this date early in the review, because if the date falls outside the review period, then the reviewer should stop the review. These questions validate the accuracy of the date on which the determination was issued.

   Elements #26 and 27 – Determination Fully Implemented
   Recommendation: Eliminate.

   Rationale: These data elements were rarely used. Implementation of the determination has not proven to be a problem.

   Establish Error Codes
   Recommendation: Establish error codes that indicate causes for failure and store them in a database for easy analysis. For example, error codes would identify insufficient facts obtained, or minimum attempt not made.

   Rationale: Codes are needed to pinpoint specific problem areas so the states can focus on training, and the Federal partner can focus on technical assistance. Reviewers would continue to document various problems on the back of the DCI, but a coding scheme would make analysis of error causes easier and could help track improvements.

2. Scoring System
   Background: The system used to score the quality review of nonmonetary determinations has been in place since the late 1970s. It assigns numerical values to a series of questions, but is, in fact, a pass/fail system. In order for a case to pass, each of the questions must receive full value except for the question concerning the quality of the written determination. The case can pass with a score of 95 points if the written determination is scored inadequate, but will fail if the written determination is scored completely wrong. The instructions to reviewers say that cases scoring 80 points or higher are passing. In fact, however, 80 points is not a possible score. Possible passing scores are 100 points and 95 points. Failing scores are 75 points and lower. The study group discussed other possible methods for scoring, but ultimately agreed to retain the current system with additional clarifications. Also, the elimination of the rebuttal element from the DCI would necessitate some redistribution of those points. Changes and clarifications are described below.

   Recommendation: Maintain the current scoring methodology, but clearly describe a passing score as either 95 points or 100 points.

   Rationale: Because cases cannot be assigned a score of 80 points, the current description is misleading. The change accurately describes the scoring system.

   Recommendation: Change the point system on DCI elements for: 18. Claimant Information; 19. Employer Information; 20. Info/Facts from Others; and 22. Law/Policy.
   Point system for Elements 18, 19, 20:

   • Adequate = 15
   • Inadequate = 10
   • Not Obtained = 0
   • N/A = 15
   Point system for Element 22:
   • Meets = 45
   • Questionable = 30
   • Does not meet = 0

   Rationale: This distributes the points that were assigned to the rebuttal element, which will be dropped, so that the total is still 100 points.

   Recommendation: Whether or not claimants are apprised of appeal rights will be scored as part of the written determination. If appeal rights are inadequate then the written determination cannot be scored as adequate, but the case would not fail solely because appeal rights were inadequate. The appeal rights element is being eliminated as a separate element and incorporated with the written determination. See Element #24 above.

   Rationale: The Employment Security Manual, Part V, requires that adequate information about appeal rights must be included on any determination sent to interested parties.

3. Clarifications to ET Handbook No. 301
   The study group made the following recommendations to clarify the quality review instructions.

   Recommendations:

   • Voluntary Quit: Add language stating that if complete fact finding with a claimant on a voluntary quit issue establishes a quit without good cause, employer information is not required if the disqualification penalty for voluntary quit and discharge is the same. If there is a more severe penalty for "gross" misconduct, employer information is required. Also, any time a decision to pay is made, employer information is required
   • Discharge: clarify that the consequences of repeating an act can be implied in a valid warning during the disciplinary process.
   • Refusal of Work/Referral: add language stating that if the facts clearly establish good cause (for an uncontrollable reason), it is not necessary to examine the suitability of the work.
   • Refusal of Work/Referral: clarify that if the claimant reports a job refusal, but the employer indicates there is no job offer, no issue exists.
   • Reporting Requirements: add language to update the entire section to reflect applicable technology/automation advancements (example e-mail).
   • Approved Training: Clarify the definition of approved training in the quality review instructions.
   • Educational Employees Between/Within Terms: reinstate omitted language about mutuality of assurance if claimant voluntarily quits. (Oct 1979 attachment to 'Draft language and commentary to implement the unemployment compensation amendments of 1976 PL 94-566) – This was inadvertently omitted from the last revision of ET Handbook No. 301.
   • Unemployment Status: Prepare guide sheet.
   • Administrative Penalty: Prepare guide sheet.
   • Harmonize ET Handbook No. 401 and ET Handbook No. 301 so as to minimize state programming.