<p>First, a meta point- We probably should not have moved to this phase without finalizing the project question. So, I will wait a few days for any more opinions to come in and then get this thread locked. The next question will focus on which one of the four ideas we should work on our first project. Ideally, all of us should commit to working on the most voted idea (even if it is not our personal favorite) to maximize participation and to divide-and-conquer the project. After all, the point of Polystats Projects = 'massively collaborative data analysis'.</p>
<p>Here are my answers to my four questions from the top:</p>
<ol>
<li><p>Depending on the project selected we should use data from the SO, SU and SF OR the data from the beta sites OR perhaps all sites. I think the answer to this is dependent on the question we select.</p>
</li>
<li><p>IMO, if we select SO, SU and SF we definitely need a sampling strategy. If we want to maximize contribution then it would make sense to work with 'small' datasets as the community will have different computing limitations (e.g., some folks may not have the RAM to load all the data in memory etc). We want to be as inclusive as possible and hence large datasets are a negative as far as this issue is concerned.</p>
<p>Perhaps, we can do some sort of <a href="http://en.wikipedia.org/wiki/Stratified_sampling" rel="nofollow noreferrer">stratified sampling</a> with the strata being: time period, No of votes for qns during a time period (high, medium, low etc), rep of users during a time period (again high, medium, low etc).</p>
</li>
<li><p>Yes, we should work with the same dataset. Not only does this help with reproducibility but it would also ensure that we are on the same page. One advantage here is that if we use different methods (machine learning, stats, data mining etc) to investigate the same question then we can attribute any differences in conclusions to the methods used rather than any idiosyncrasies of the data we used.</p>
<p>Some form of delimited, text file is ideal (perhaps, csv). Storage is much less of an issue as long as it is a stable link.</p>
</li>
<li><p>What data we need to extract is dependent on the questions we seek to investigate. So, I will defer answering this question.</p>
</li>
</ol>
