Chapter 8

Measuring Outcomes

Measuring client outcomes in the legal hotline setting requires a somewhat different focus than measuring results achieved in a full-service unit. Jan May at AARP Legal Counsel for the Elderly (LCE) in Washington D.C. has devised a method of tracking outcomes via the program’s case management software. LCE created two pull-down lists on the case closure screen of the client data base- one extended list for casehandlers and a shorter list of about 30 outcomes for the Hotline. The hotline outcome codes have short narrative descriptors such as “obtained verbal explanation of legal rights, responsibilities, or procedures” or “obtained assistance in filling out a form”. The extended services outcome codes contain descriptions such as “obtained federal bankruptcy protection” or “established paternity for a child”. The hotline outcomes list provides hotline attorneys with a quick and efficient way to record results without having to scroll through the 135 alternatives available on the extended service closure list. Furthermore, LCE has created various MicroSoft Access reports which provide statistical information on the outcomes achieved. Cases and outcomes can be sorted by casehandler, program unit, or legal services problem code. These statistics should prove invaluable for grant proposals and reporting, and identifying accomplishments of individuals, projects, and the program as a whole.

Two important outcome measures have been undertaken this year. The Project for the Future of Equal Justice completed its Pre-Test of the Hotline Outcomes Assessment Study which involved interviewing approximately 60 hotline clients in Washington and Connecticut regarding the outcome of their legal problem. The pre-test ascertained that telephone follow-up is a feasible way of conducting outcome studies. The majority of the small sample in the Pre-Test reported a favorable resolution. The full-report can be downloaded at www.povertylaw  or www.equaljustice.org. The second outcomes study involved five senior statewide legal hotlines which sent all clients a survey form eliciting information about the outcome of their problem during a three month period. The survey forms can be accessed at www.povertylaw. Over 1000 surveys were returned. The results are currently being tabulated by the Center for Policy Research. Results will be circulated via the websites mentioned above and in the Legal Hotline Quarterly.

The most common device used by hotlines to determine how well they are doing remains the client satisfaction survey. This method usually involves mailing a survey form to clients at some point following their contact with the hotline. The surveys typically ask whether the client was satisfied with the service, whether the problem was resolved, whether the hotline staff were helpful, courteous, and knowledgeable, and whether they would use the service again or recommend it to a friend.

There is considerable variation in the actual implementation of the survey. Some programs mail surveys to virtually all clients and some to a smaller sample. The Legal Advocacy and Resource Center in Boston (LARC) surveys 10% of their clients. Pro Seniors Hotline in Cincinnati mails a survey to a random sample of 30% of its clients every month. The Health Rights Hotline in California calls 20-25% of their clients for a follow-up interview to assess the effectiveness of its hotline assistance. Knoxville Legal Aid (Tennessee) sends everyone who receives legal advice a letter confirming the advice given and advising the client of the grievance procedure to follow if not satisfied. Every client who receives actual representation gets a formal client satisfaction survey. The Massachusetts Justice Project also includes a questionnaire with every closing letter. The Senior Legal Hotline in Sacramento sends a survey to virtually every client as soon as the case is closed coupled with a request for a donation.

There does not appear to be any consensus either on the timing of the survey or whether it is more effective to call clients or mail out the survey. Some hotlines mail surveys once a year, others from time to time. Some do monthly mailings, others time the mailing to the completion of the service. The Massachusetts Justice Project tried sending out surveys immediately following the service but found there was a better response when they waited 1-2 months. They also found that mailing surveys was more efficient than telephoning. LARC has been mailing its surveys for more than three years and has achieved a return rate of 18%. For comparison, they also surveyed clients by telephone but found it less satisfactory; although speaking to the clients provided more detail about their hotline experience, this was offset by the difficulty of reaching many clients at all. Statewide Legal Services of Connecticut tried a telephone survey with somewhat better success. In the summer of 1998 they called 251 clients from the previous summer and reached about 20%. CARPLS of Chicago undertook a major evaluation project in which they telephoned 1,423 former hotline clients and reached 23.4% on the first call (only one attempt was made).

Hotlines have used different types of personnel to contact the clients. Sometimes the hotline attorneys themselves conduct the surveys but other programs prefer to use volunteers, outside consultants, or other specially recruited staff. Because, by their nature, the surveys are very labor intensive, there are costs associated with any choice; even volunteers require training and supervision to make them effective.

The information gathered by the surveys tends to focus on the client’s satisfaction with the service provided rather than being directly related to the outcome of the case. Typical questions address matters such as ease of access to the hotline, whether the hotline attorney was courteous and knowledgeable, how useful were the self-help materials (if applicable), etc. The Health Rights Hotline even asks about the helpfulness of referral agencies. Some questionnaires ask directly about the resolution of the problem, while others may elicit such information indirectly in a comment section. Clients who respond to surveys generally report positive experiences. Typically, 85-97% of those who send back the questionnaires are pleased with the services. Those few who report dissatisfaction tend to be clients whose problems do not have a satisfactory solution.

The fundamental problem with most client satisfaction surveys is the low return rate. Rates as low as 6% have been reported and LARC’s 18% seems to be very good. The Lawyer Referral for the Elderly Program of the State Bar of New Mexico reports a return rate of 75%, but since they canvas only hotline callers who have also received some services in-house, this percentage may not be truly reflective of the hotline component. Despite their shortcomings, these surveys at least yield a general impression of the programs’ effectiveness. The responses are also useful for other purposes. For instance, the Legal Hotline for Older Michiganians reports that they use comments made by satisfied clients in their annual reports, newsletters, appeals for donations, etc. Many hotlines find them similarly helpful.

Client satisfaction surveys is a topic that deserves more attention. As a growing population needs more legal services, funded by a shrinking pool of resources, there will be increased demands for demonstrable efficiency in the delivery of services. Thus, legal service programs will be forced to defend their use of scarce funds. Hotlines will need to do more to measure the success of their programs. They must master the mechanics of conducting surveys and learn to design better survey instruments. Future survey protocols might be designed to focus on identifying a program’s target population so that the follow-up questions could be specifically directed to the clients who are in the intended groups. Perhaps hotlines could share the results of their evaluations to reduce duplication of efforts. We would be happy to provide a forum for the exchange of ideas and information.