Differences—and Similarities—Across Respondent Groups

The following sections explore variations in the helpfulness of hiring criteria based on certain respondent demographic characteristics.(15) Specifically, we examine:(16)

  • Practice setting;
  • Firm size (for those in private practice);
  • Years of experience; and
  • Geographic region.

Notably, across all of these groups, responses were much more similar than different—respondents tended to view each criterion as helpful, regardless of their specific demographic characteristics. Still, analysis yielded a large number of statistically significant results. A statistically significant result tells us that there is a relationship between the variables—a specific hiring criterion and the analyzed demographic group—which likely reflects a true response pattern in the population, rather than emerging from the sample by chance. However, when respondents in one demographic group consistently indicate that a given hiring criterion is more (or less) helpful than do those in another group and when there are a large number of responses, a result may be statistically significant even when there is not much divergence between the groups. For example, a difference of one or two percentage points between two groups may be statistically significant, but such a small difference is not large enough to indicate a meaningful difference upon which law schools and law students might base a change in curricula or focus.

Accordingly, and in addition to statistical significance alone, we have determined that we will consider a result practically significant if the difference between demographic groups is large enough to influence our thinking in practice. Specifically, for the purposes of this report, we have defined a practically significant difference as one ten percentage points or larger between any two demographic groups.(17)

We also wanted to explore and compare the ten hiring criteria within each demographic group which respondents identified as the most helpful in determining whether a new lawyer possesses the needed foundations. Together with the discussion of practical significance, we are able to see the full picture of how different demographic groups view the helpfulness of each criterion differently—or similarly.


15.  We used chi-square analyses to test the relationships between demographic characteristics and helpfulness of hiring criteria. To adjust for the large number of statistical tests conducted, we used a statistical significance level of p < .003.

16.  One potential concern in analysis was whether respondents with a role in hiring new lawyers would respond differently from those who do not currently have such a role. Thus, in addition to the four demographic characteristics presented in this report, we analyzed the hiring criteria data by respondent role in hiring. The results of this analysis showed no practically significant relationship between hiring role and any of the seventeen hiring criteria. We can be, therefore, confident that responses are consistent between those who hire new lawyers and those who do not.

17.  All results are presented in the appendix, available at http://iaals.du.edu/sites/default/files/documents/publications/hiring_whole_lawyer_appendix.pdf.