<p><strong>TL;DR:</strong> Frank and <a href="https://stats.meta.stackexchange.com/questions/2732/feature-selection?cb=1#comment8347_2732">Scortchi</a> have a point for prediction - but the <a href="https://stats.stackexchange.com/q/66448/17230">question Scortchi proposes</a> doesn't really address people's concerns about assessing a variable's relevance. I think this is a pertinent question to which we don't yet have a good answer. Apart from point 9 in <a href="https://stats.stackexchange.com/a/20856/1352">gung's answer here</a>, which says that no technique will save the analyst from having to actually think about his data.</p>

<hr>

<p>I think there may be an unstated philosophical/epistemological/teleological point here: <em>WHY do people actually want to model?</em></p>

<p>If they want to model for prediction, then things are easier. I fully agree with Frank and <a href="https://stats.meta.stackexchange.com/questions/2732/feature-selection?cb=1#comment8347_2732">Scortchi</a> here.</p>

<p>However, often people model data to "understand a predictor's relevance", to use terms that are not loaded with technical meaning. This usually takes the form of inferential statistics (and can lead to an unhealthy obsession with p-values), but things like variable importance in random forests aim in the same direction. As do formal causal models.</p>

<p>I would say that this is a legitimate aim of data analysis, both in an academic and an applied context. However, understanding a predictor's relevance runs into problems that we don't see if all we aim for is a good overall response prediction.</p>

<p>In this second situation, we could argue for "tapering effect sizes" in the sense of <a href="http://rads.stackoverflow.com/amzn/click/B008UBJ0VQ" rel="nofollow noreferrer">Burnham &amp; Anderson</a>: <em>everything</em> has an impact, it's just that effect sizes taper off to insignificance. This would argue for keeping everything in the model... except for the bias-variance tradeoff (see <a href="https://stats.meta.stackexchange.com/questions/2732/feature-selection?cb=1#comment8347_2732">@Scortchi's suggestion</a> to link to <a href="https://stats.stackexchange.com/q/20836/1352">this question</a>) and non-identifiability. So we need some kind of dimension reduction or regularisation.</p>

<p>Unfortunately, dimension reduction mangles predictors beyond recognition, so assessing a variable's relevance after, say, PCA, will be <em>hard</em>. And regularization can introduce an element of arbitrariness that can be hard to defend - if a predictor's relevance depends on your Bayesian priors, this is a weak point in your argumentation. Optimal choice of lasso or elastic net parameters can at least remove this element of arbitrariness, and then we can look at standardized parameter estimates in an elastic net, so this is something I like, although I don't know of any actual research on assessing variable relevance via elastic nets (pointers would be most welcome).</p>

<p>Unfortunately (again), this seems like we would need to push people towards one very specific kind of model: elastic net-based regularization. But what do we answer those people that are using some other kind of model, for whatever reason, and find that it does not work well with the full set of predictors?</p>

<p>Alternatively, we can argue that you only have understood a process well if you can predict it well. So, to assess a variable's importance, build prediction models with and without that variable (and any transforms of the variable you want), test the models out-of-sample, and see whether prediction accuracy goes down. (Diebold-Mariano type tests can be used for disciplines that can't get away from p-values.) This is actually my preferred position.</p>

<p>However, there is the problem that nominally out-of-sample tests actually aren't, because people will run this sequence iteratively in building their models. In addition, it is <em>hard</em> to get an approach like this accepted in disciplines that traditionally don't tick like this, like psychology. Not to speak of the <em>fascinating</em> discussions you can have about how to actually measure a prediction's accuracy, where the arbitrariness discussed above creeps in again.</p>
