<p>As the many different responses in this thread already attest, it would be helpful to address some preliminary issues before asking this question.  The important ones include:</p>

<ul>
<li><p>The study question itself ought to be well defined.  Exactly what is a "reputation effect"?  (I have followed all the referenced threads, and have quickly reviewed them again, but can find no description or definition of this term.  Am I the only one who doesn't know precisely what it means?)</p></li>
<li><p>Guidance from previous work can be helpful.  E.g., does anyone know of any specific examples of "reputation effects"?  Is there some background literature describing a theory of such effects?  If so, what does it suggest about effect sizes, important covariates, explanatory factors, appropriate ways to measure them, and so on?</p></li>
<li><p>What do we hope the scope of the study will be?  Do we aim do draw conclusions about all "SE Communities" or only about the ones whose data will be studied?  Is there any intent or hope of drawing more general conclusions about "reputation" itself?</p></li>
<li><p>Because this appears to be research <em>ab initio</em>--that is, not building on or relying on previous studies--we will need a preliminary EDA phase.  It may be difficult to recommend a good sampling plan until some EDA is accomplished.  Indeed, a minimum of three phases of sampling seems to be required: some sampling--not necessarily even random--to conduct EDA, followed by more formal randomized sampling to obtain a small representative subset of data whose aim is to estimate effect sizes and variances well enough to determine how much more data will be needed to test the study questions to given degrees of confidence.</p></li>
<li><p>...which implies we need to establish some targets for confidence in the study results.</p></li>
<li><p>Those targets may have to be adjusted to account for multiple comparisons.</p></li>
<li><p>We cannot do any such adjusting until a set of specific hypotheses has been established, which is one of the aims of the EDA.</p></li>
<li><p>Can we be more specific about the actual cost (or its equivalent in volunteer effort) of sampling the data?  This would help us make tradeoffs between the desire for high confidence (achieved by larger sample sizes) and the pain of gathering the data needed to achieve such confidence.  If necessary, addressing this question can be delayed until after the EDA or preliminary sampling phase.</p></li>
</ul>

<p>Once these questions have been answered it should be straightforward do specify a sampling protocol.  I don't imagine these questions will be difficult to answer, but the answers ought to be given and documented somewhere.  Such formalities seem especially important in light of the collaborative, distributed nature of the proposed research, so that a common understanding is established (and not just assumed).</p>

<p>Apropos this last point, it's starting to look like developing a study protocol and all its documentation solely as a set of threads on this site will quickly become unwieldy.  Who wants to wade through a long set of links to questions/comments/answers in order to look up basic information?  It would be <em>very</em> useful to have a central location in which a small number of key documents are dynamically maintained.  These would eventually include a statement of the study goals (including background information), the study protocol, a data dictionary, and the various datasets themselves.</p>
