<p><strong>Update</strong></p>

<p>I have asked the mods to lock this thread. I will post a question which will be a poll on what specific project question we want to investigate as part of the first project.</p>

<p><strong>End of update</strong></p>

<p>[<strong>Important Meta Note</strong> </p>

<p>Please do not vote-up this question as this question is a housekeeping type of question designed to move the polystats project to the next phase of the project. However, do vote up answers depending on their quality. I will add dummy edits at some regular frequency to push this question to the top from time to time. You may also wish to see this meta thread on the issue of <a href="https://stats.meta.stackexchange.com/q/428/28">rep and polystats project</a>.]</p>

<p>The consensus (based on the votes/answers) to the first question <a href="https://stats.stackexchange.com/q/2512/28">What should be our first Polystats Project</a> seems to be indicate that our first project would use Stackexchange data to answer several type of questions. The following summarizes the project ideas of that thread:</p>

<ul>
<li><p>Are there reputation effects in SE communities and to what extent are reputation effects contingent on various variables such as size of the community, number of views per day, no of questions per day etc?</p></li>
<li><p>Quantify the quality of answers and questions (what makes a question/answer a good one?)</p></li>
<li><p>Classify how the communities differ.</p></li>
<li><p>Is there any evidence for herd behavior? (i.e., Is the probability of a question/answer being voted up/down dependent on the existing up/down totals?)</p></li>
</ul>

<p>The above is my interpretation of the suggestions made in that thread. While we can debate the merits of the above questions on various dimensions (e.g., which ones are interesting, doable etc), I would like the focus of this question to be on our data collection strategy.</p>

<p>Currently, there exist three main SE sites (Stackoverflow, Superuser and ServerFault) and a whole bunch of beta sites including our site (I am not counting meta.SE and SE 1.0 sites). It seems to me that the beta sites are not well established enough for us to answer any of the questions we have in a reasonable way. Thus, I propose that we restrict attention to Stackoverflow, Superuser and Server Fault.</p>

<p>However, the amount of data that is available on these sites is staggering. The following is a snapshot of the current statistics for these three sites:</p>

<p><img src="https://i.stack.imgur.com/pD8jR.pngBlockquote" alt="alt text"></p>

<p>Clearly, we cannot download all the data to work with and need some form of sampling strategy. </p>

<p>Given the above background, my questions are:</p>

<ol>
<li><p>Do you agree with the restriction to Stackoverflow, Superuser and Serverfault?</p></li>
<li><p>What data sampling strategy would you recommend that we follow?</p></li>
<li><p>Do you think that all of us should work with the same dataset (to ensure that we can reproduce each other's results/bugs)? If so, where/how should we store it?</p></li>
<li><p>What variables do you like to see in the data dump? The following shows the data that we can potentially download using the API/data-dump provided by SE. Table names are in capital letters with corresponding fields below the table name.</p></li>
</ol>

<blockquote>
  <blockquote>
    <p>
    POSTS
    Id
    PostTypeId
    AcceptedAnswerId
    CreationDate
    Score
    ViewCount
    Body
    OwnerUserId
    LastEditorUserId
    LastEditDate
    LastActivityDate
    Title
    Tags
    AnswerCount
    CommentCount
    FavoriteCount
    ClosedDate
    ParentId
    CommunityOwnedDate
    USERS
    Id
    Reputation
    EmailHash
    CreationDate
    DisplayName
    LastAccessDate
    WebsiteUrl
    Location
    Age
    AboutMe
    Views
    UpVotes
    DownVotes
    COMMENTS
    Id
    PostId
    Score
    Text
    CreationDate
    UserId
    BADGES
    Id
    UserId
    Name
    Date
    POSTTAGS
    PostId
    TagId
    TAGS
    Id
    TagName
    VOTES
    Id
    PostId
    VoteTypeId
    CreationDate
    BountyAmount
    UserId
    VoteTypes
    Id
    Name
    </p>
  </blockquote>
</blockquote>
