Using Data to Measure Judicial Innovation Impact

On August 12, IAALS and the Berkeley Judicial Institute (BJI) hosted Measuring What Matters: Evaluating the Impact of Innovations, a webinar designed to equip court leaders with practical strategies for taking a data-driven approach to evaluating the impact of their initiatives. Moderated by Danielle Kalil, Director of Civil Justice and the Judiciary at IAALS, the conversation between Kevin S. Burke, Senior Judge, Minnesota District Courts; Margaret Hagan, Executive Director, Legal Design Lab, Stanford Law School; Victor D. Quintanilla, Professor of Law & Val Nolan Faculty Fellow, Indiana University Maurer School of Law; and Beth Wiggins, Director of Research Division, Federal Judicial Center, focused on how to identify key performance indicators, effectively collect and analyze relevant data, and use those insights to inform future decisions and demonstrate the value of your innovative work. 

Why evaluate an innovation at all? Judge Burke pointed out how the courts are comfortable doing things the way they have always been done, but they must break out of that comfort in order to improve the courts for the people who use them. The panelists agreed that evaluation builds trust, sharpens the design of innovation, and makes it clear what you want to achieve for all involved in and affected by an innovation. 

“Building an evaluation component early into the process sharpens the innovation itself because it forces everybody to think hard about why they think what they're thinking. It gets everybody to be more specific about what they actually want to achieve and what their precise concerns are,” said Wiggins. 

Defining the success of your innovation is as important as understanding why you are evaluating it in the first place. The panelists talked about how knowing the specific problem you are trying to solve when you begin can guide you as you are evaluating the results. Understanding who your stakeholders are, what you want a change to do, and if people actually use your change are all effective ways to measure if a change is working. 

“I like to divide up into three areas of success. One is the immediate thing. What does success look like? Does this thing live up to the standards that we want it to? Success two is measuring do people actually use the thing? Can they find it? Does it get them to that next step?” said Hagan. “And then we want to measure the downstream impact on the court's efficiency, on the rule of law and just procedure, and on people's outcomes.”

There will be hurdles along the way. Hagan talked about how difficult it can be to gather and connect relevant data, and how crucial it is to do that. Judge Burke encouraged courts to find other comparable courts to use as a measuring stick for themselves to fight complacency. Quintanilla touched on how courts need to build a learning culture so they can learn from and share information with one another.     

“Cooperation is really helpful. We should be talking with one another and learning from the bright spots that do exist, and there are a lot of them,” said Quintanilla.

The panelists shared that although this process can be difficult, it is necessary to continue to improve the courts for everyone who uses them. Wiggins directed courts to simply begin with a problem they see as a priority. Do not be overwhelmed by all the things you could do or the lack of resources you may be facing. Start with one problem and do what you can with what you have. The panelists encouraged courts to be open to dialogue and sharing data, whether that is with other courts, the users of the courts, or partners in the community, like universities.

“I would say to a court leader, be curious… [and] be inclusive. It's really important that you get the people who are actually doing the work engaged in why we're doing this, and why it's important that we collect what we collect,” said Judge Burke. 

This webinar is the sixth in this series focused on judicial leadership.