Utah Sandbox Evaluation
On August 15, 2020, the Utah Supreme Court issued Standing Order No. 15, which established a regulatory Sandbox for new kinds of legal services and service providers—those disallowed under the traditional regulatory framework—could emerge and develop. The stated objective of this new regulatory framework is “to ensure consumers have access to a well-developed, high-quality, innovative, affordable, and competitive market for legal services.” Utah’s regulatory Sandbox is built largely on the model IAALS developed and published in 2019.
Rigorous data collection and evaluation is critical in understanding whether Utah’s regulatory Sandbox is meeting its stated goals. To that end, IAALS is acting as an independent third-party evaluator for the Sandbox.
We are conducting an interim evaluation consisting of four reports, covering August 2020 through February 2025. Each report can be downloaded below.
This is part of a broader, longer term evaluation effort that we expect to publish towards the end of the Sandbox’s pilot period, which is set to expire in 2027.
Objectives
- Process Evaluation: Assessing the implementation of the Sandbox by articulating a theory of change and developing a logic model to understand the relationships between inputs, activities, and expected outcomes. This critical evaluative step provides a structured framework to examine whether the Sandbox’s design and execution align with its intended objectives and regulatory principles.
- Outcomes Evaluation: Analyzing the Sandbox in key areas, including the entity authorizations, types of entities operating in the Sandbox, the services provided, the quality of those services, the objectives of Sandbox entities, and additional insights gathered from their experiences. This evaluation examines measurable indicators such as service volume, consumer satisfaction, and observed impacts on the legal marketplace. Understanding these outcomes helps assess the Sandbox’s role in improving legal service accessibility and innovation.
- Social Return on Investment (SROI) Analysis: Conducting an assessment similar to a cost-benefit analysis to quantify the value of the Sandbox’s outcomes, including estimating the monetary value associated with key impacts. This analysis helps determine the broader economic and social benefits generated by the initiative relative to its costs.
Interim Evaluation Reports & Takeaways
Reports:
IAALS' interim evaluation report series consists of four separate publications:
- Sandbox Background and Evaluation Design Overview
Presents important background and contextual information about the Sandbox along with an overview of the purpose and evaluative approaches used. - Process Evaluation
Provides a detailed examination of how the Sandbox is designed and implemented, including a fully articulated logic model and a comprehensive review of changes to Sandbox policy to date. - Outcomes Evaluation
Presents findings related to the outcomes and intended impacts of the Sandbox, including the entity authorizations, types of entities operating in the Sandbox, services provided, the quality of those services, the objectives of Sandbox entities, and additional insights gathered from their experiences. - Social Return on Investment (SROI) Analysis
Provides empirically driven insights into the value the Sandbox has generated—including social value—in comparison with the amount invested.
Takeaways:
Since October 2020, the Sandbox has grown from 31 applications and 11 authorized entities to 105 applications and 51 authorizations by early 2024. Growth has been concentrated among low-risk models, with no high-innovation entities approved. While participants express optimism and the Utah Supreme Court highlights the program’s flexibility, delays, evolving policies, and unclear data requirements continue to create friction. Resource constraints may limit the Sandbox’s ability to support more ambitious innovation.
Services expanded from 612 to over 76,000, driven largely by moderate innovation entities (~58%). However, delivery models largely continue to involve lawyers (rising from 72% to 79%). Service areas shifted away from end of life planning and marriage/family and toward military and immigration, with business services remaining steady, and underserved populations are being reached. Still, high entry costs, complexity, and concerns about perceived court endorsement may constrain broader impact.
Initial indicators from the Sandbox point to strong service quality and minimal consumer harm. Three service quality audits—focused on moderate innovation entities—found no material or substantial harm and concluded that services were at least satisfactory. Complaint data aligns with these findings: as of January 2024, just 14 complaints were reported across more than 76,000 services, with only nine involving defined consumer harms—fewer than 0.01% of all services. Together, these results suggest that the Sandbox is delivering services safely at scale. At the same time, expanding the scope and consistency of quality measurement would help build a more comprehensive and durable evidence base as the program continues to grow.
Entities and leadership are optimistic but cite staffing limits and uncertainty about long-term pathways. Strengths include committed leadership and reforms like the Innovation Requirement, but challenges persist: reliance on volunteers, regulatory burden, and questions about courts as regulators. Clearer structures are needed for long-term sustainability.
The Sandbox has invested in public communications, but clarity, accessibility, and consistency remain issues. Materials are often too complex, outdated, or fragmented, and the shift away from detailed reporting has reduced transparency. These gaps limit public understanding and trust.
Entities face cost burdens from delays and multi-jurisdictional compliance, compounded by limited regulatory resources. Leadership also highlights unsustainable reliance on volunteers. In response, the Utah Supreme Court moved to a fee-based model and narrowed the program’s scope—steps that improve sustainability but may affect accessibility and impact.
Data collection is core to the model, yet reporting has become less detailed and more difficult to sustain. Entities find requirements unclear, and leadership acknowledges challenges in defining meaningful metrics. While the Utah Supreme Court emphasizes data-driven regulation, gaps remain in measuring consumer benefit and comparing outcomes.
Entity goals closely align with the Court’s priorities—client-centered service, access to justice, innovation, and growth. About 76% of entities’ goals were met or exceeded expectations, suggesting real progress, though some execution challenges remain.
The Sandbox’s Social Return on Investment (SROI) analysis estimates that the program generated between $1.17 and $1.71 in social value for every $1.00 invested during its first three years of operation. In SROI analysis, ratios above 1.0 generally indicate that a program is generating more social value than the resources required to support it. While these returns are relatively modest, this is consistent with what would typically be expected during the early stages of a new regulatory initiative, when development, oversight, and implementation costs are highest and many longer-term impacts have not yet fully emerged. Even within this early-stage context, the findings suggest that the Sandbox is already producing measurable economic benefits for both consumers and the justice system. Because the analysis relies on conservative assumptions and excludes many longer-term and non-monetized impacts, such as broader access-to-justice benefits, improved procedural engagement, and downstream social effects, the findings likely represent a bounded estimate of the Sandbox’s overall public value.