PHInational.org

Gathering the Evidence that Makes Progress Possible

This is the third in a series of PHI Expert Interviews, which bring you insights from four senior PHI staff. They’re an impressive group - among the nation’s leading experts on long-term care’s direct-care workforce - and collectively they’ve spent decades studying the challenges facing the workforce and how to address them. We think you’ll be interested in what they’ve learned. 

Marcia Mayfield, PHI’s director of evaluation, helps PHI document its successes for policymakers, employers, funders, and anyone else who needs to know what works and what doesn’t. As she explains it, her evaluation team does three things:

• Helps PHI learn from what it has done, to make its work more effective;
• Documents PHI’s work and measures its impact, “both for our own purposes and to share what we’ve learned with others in the field”; and
• Develops evaluation tools and approaches for use by anyone interested in improving direct-care jobs. For example, providers can use a business investment calculator due out this fall to calculate their turnover costs, comparing that figure to the cost of various retention or culture change initiatives.

Hired last year by PHI  after 12 years as an evaluator for an international women’s  health organization, Marcia says her goal at PHI is “to demonstrate in a measurable way that what we’re doing works. We essentially have to make the business case for the initiatives we’re promoting.”

PHI works on two tracks, so she does too. Her training and organizational development (TOD) work measures whether and how PHI helps employers improve workforce practices. And her policy work measures the progress made by PHI and its allies in improving public policy to support direct-care workers.

Training and Organizational Development: Documenting process as well as outcomes

PHI’s Center for Coaching Supervision and Leadership (CCSL) is in the midst of a four-year initiative to introduce PHI’s coaching supervision model in nursing homes and home health agencies. Marcia’s team is an integral part of figuring out what is and isn’t working.

They’re collecting both quantitative and qualitative data to determine whether coaching supervision affects things like turnover, absences, and job satisfaction. And they’re evaluating the process itself, documenting how the program is being implemented.

Process evaluation, says Marcia, is an important and often overlooked step. “In CCSL, for instance, there was research that said if you improve supervision you improve retention, but there’s not a lot of evidence about how to do it, or how to sustain such an effort. So it’s important to document program implementation as you’re going along, rather than just collecting outcome data.”

On the most basic level, Marcia says, documenting the process provides evidence that people really did what they say they did. It also makes it easier to identify and change things that aren’t working well. For example, CCSL’s evaluation helped clarify who should attend the train-the-trainer sessions. “In the first year, people were selected who had a huge amount of other responsibilities within the organization, so pulling them away to do training was very difficult,” says Marcia. “In the second year, the CCSL team was much clearer about what was required – the skills and the amount of work and the commitment to the project. They also increased the number of people trained as trainers, to create depth in the organization and to mitigate the problem of trainer turnover, which had been an issue the first year.”

Policy: evaluating the interim steps

Changes in organizational practices may come slowly, but they seem lightning-fast compared to the process of passing a piece of legislation or changing a regulation. And so, with policy evaluations, the team starts by measuring what’s being done to prime the political pump.

“For Health Care for Health Care Workers our ultimate goal is for direct-care workers to receive needed health care coverage, but before legislative policies can be enacted work must be done just to increase awareness,” says Marcia. “So we look for things like increased media coverage and hits on our website, then measure whether that raises awareness.

“Alliances are also important, so we’re measuring things like how many partnerships we’re forming, who’s writing letters to legislators, and so on,” she adds. “You’re building evidence that there is a base of support for your issue, which often is a prerequisite for needed policy change.”

Making evaluation part of the process

Whether she’s measuring a policy or a TOD initiative, Marcia’s process is the same.  “At the start of all projects, we develop a logic model that engages staff and other stakeholders in thinking about the connections between inputs – the resources you put in and the activities you do and so on – and the results you expect, not just immediate outputs but intermediate and long-term outcomes as well.”

Any new program or project would benefit from that kind of evaluation, Marcia says. “It helps make sure that what you’re doing will lead to some kind of measurable outcome. And if there’s a certain outcome the staff want to see but their inputs are not addressing that in a straightforward way, it helps them engage in a discussion about what else might be needed.”

Last but not least, evaluation helps the field move forward, using the wheels other people have forged rather than constantly stopping to reinvent them. “You want to look at the existing empirical data and evidence, read the existing literature, before you start something new,” says Marcia. “You want to know what has worked in the past.”

Interview by Elise Nakhnikian, Senior Online Editor
enakhnikian@phinational.org