<p>Is there much precedent or openness to Q&amp;A on the public sphere issues involving statistical methods?  This question has three parts: a motivating example, a statistical perspective, and a perspective as it relates to utility of such questions on CV.SE.</p>

<p>The motivating example comes from reading <a href="http://www.nytimes.com/2012/02/24/education/ratings-of-new-york-city-teachers-to-be-released-friday.html?hp" rel="nofollow">this article</a> in the New York Times.  The gist is that someone, somewhere has produced a scoring of NYC teachers, based on a "value added model", and the results of those evaluations will be made public.  I'd like to put aside issues regarding evaluation of teachers (e.g. <a href="http://www.nytimes.com/2012/02/23/opinion/for-teachers-shame-is-no-solution.html?_r=1&amp;ref=opinion" rel="nofollow">Bill Gates has addressed this</a>) and any more general matters regarding education, and focus on the statistical issues.  This topic is in the public sphere: public resources and attention are shaped by the results of a statistical project.</p>

<p>The statistical perspective is that there are several things that can be addressed: what <em>is</em> the method that is employed?  What are its assumptions?  Where can one access the data?  What are alternative methods?  How can/did the authors check the assumptions?  What happens if the assumptions are wrong?  How sensitive is the model to errors in the data, etc.</p>

<p>The goal isn't to make a judgement about either the utility of the practice nor to attack or defend the research.  Instead, the statistical questions, like all research and consulting projects, are based on utility and reproduceability.  Are the statistical questions well posed, do the data allow us to answer the questions, were the questions answered, and what may be done in a subsequent iteration (either of analyses, data collection, or application of the methodology in another context)?</p>

<p>As for the CV perspective, there are some questions that seem quite at home here, such as: What is the statistical model that's used?  What are its assumptions?  There are other questions that are not so good, and are more opinion based: What's wrong with this paper?  What's good about this paper?</p>

<p>I believe that outright opinion questions not appropriate.  Could they be reformulated?  Perhaps.  An example: what is the most suspect assumption of this application?  Time and again, I take a project, identify the key assumptions, and then determine which ones would not hold up to scrutiny in either the same context, using data from a different sample, or in a related context.  In other words, which assumptions <em>only</em> work this one time, for this one data set?  Those are particularly suspect assumptions.  Many other assumptions are tolerably false.</p>

<p>Now, my opinion:</p>

<p>I believe that such questions could make CV quite attractive to a larger set of users and serve a serious educational and public benefit: people could learn about statistical methodology and engage in understanding a use of statistics in the public sphere.  I am not interested in potential questions that relate merely to reports on surveys and census results: these can be tar pits of debate over sampling methodologies.  Instead, I'm concerned with more methodological and model-based types of questions, where assumptions about data and models play a role.</p>

<p>I have used such analyses of published works in the past to teach students statistical methods, and it has worked very well.  At some point, perhaps it would be of interest to expand to other published works, but I think that public sphere questions are a good start.</p>
