Maintaining a Critical Eye When Assessing Research Findings

We live in the information age. Reclaiming Futures sites and other jurisdictions engaged in similar focused reform efforts have access to information from multiple sources and disciplines such as: web-sites; journal articles, news; juvenile justice, behavioral health, psychology, sociology, education, social work, medicine and so on.

So how do we critically assess these sources of information? Let’s discuss by using an example.

Recently the National Program Office (NPO) was sent a news release (see link below). This topic is relevant and it’s from a credible source. If you read the news release, without asking questions or seeking additional information, you may conclude that acknowledging racial disparities within the juvenile justice system is negatively affecting young people of color. However, what the news release does not tell us is:

• How people were selected? How were the White people recruited from the San Francisco train station? Where was the recruitment location in New York?
• How were participants informed about the purpose and procedures of the study? Why were they being invited to participate?
• Is providing participants with one example of a local policy enough to make the conclusion?
• Is recruiting from two geographically and demographically different locations enough to make the conclusions?
• Is it fair to the people in the study to say that being exposed to mug shots determines their policy opinions?

Given these questions, we went to the original source (see link to the article). While the article provides additional information, other questions were raised. For example:

• Was this study approved by an institutional review board?
• How long did study participation take (i.e., minutes; hours)?
• What was the environment for the interview (i.e., distractions)?
• Do the overall and group sample sizes and statistical methods provide enough power to make the conclusions?
• What were the limitations to this study?

The research question posed - what is the “relationship between racial disparities in incarceration and people’s acceptance of punitive policies” (page 1) is an important one. A study that helps us understand this question would be pertinent to Reclaiming Futures work and the work of other organizations concerned with causes and negative impacts of racial disparity.

The Office of Juvenile Justice and Delinquency Prevention (OJJDP) has been collecting data on racial disparities in the juvenile justice system since 1990 (National Disproportionate Minority Contact Databook: http://www.ojjdp.gov/ojstatbb/dmcdb/). Other organizations such as Annie E. Casey and The John D. and Catherine T. MacArthur Foundation have developed programs and policies to help reduce racial disparities. We still have a long way to go, but there have been measureable decreases in racial disparities in our juvenile justice systems since 1990. This might be attributed to utilizing a long-term approach, based on data, to inform and develop policy and programs. Perhaps most relevant to our work is the question - If all things are equal, do youth of racial and ethnic minority groups receive the same punishment and treatment as White youth? If the answer is no (as many practitioners and researchers think that it is), then we still have more work to do.

Not all studies meet the highest standards of randomized clinical trials and many other less costly and equally powerful methods exist (e.g., program evaluation; propensity score matching). The University of Michigan’s, Research Connections provides guidance for assessing the quality of social science and policy research (http://www.researchconnections.org/content/childcare/understand/research-quality.html ).

Each of us must have a critical eye to the information we receive and determining what is most useful and reliable for our work. This does not mean that the information must always support our mission and values. In fact, well designed studies that provide alternative findings may encourage us to ask additional questions, continue improving, and educating our communities.

Link to news release: http://news.stanford.edu/news/2014/august/prison-black-laws-080614.html

Link to article: http://pss.sagepub.com/content/early/2014/08/05/0956797614540307.full.pdf+html