<p>I've observed that we tend to get a number of questions in the vein of &quot;I'm training a neural network and the model is not effective (for some definition of effective). Please help me fix it.&quot; (I can provide examples of these types of questions, but I don't think it would be helpful to highlight the authors at this time.)</p>
<p>These questions are not exactly programming questions, since OP has <em>syntactically-correct</em> code; instead, OP's problem is that the model isn't very effective at whatever task they're working on. (But there could be <em>semantic</em> errors, in which case the question is, in my view, a programming question -- but whether or not this is the case is not obvious without a detailed study of the code and what task OP is trying to accomplish with the model. Both are often unclear.)</p>
<p>In some sense, they are on-topic because they are about how to carry out modeling which is, essentially, a core component of modern machine learning.</p>
<p>On the other hand, since all neural networks are bespoke and parameterized with a truly massive number of knobs (hyperparameters) to twiddle, these questions don't seem to generalize very well. And these threads tend to take on the form of a frustrating IT support dialogue:</p>
<blockquote>
<p>Answerer: &quot;Try adjusting the learning rate.&quot;</p>
<p>OP: &quot;That didn't work.&quot;</p>
<p>Answerer: &quot;Try using a smaller hidden layer.&quot;</p>
<p>OP: &quot;That didn't work.&quot;</p>
<p>Answerer: &quot;Try using a different activation function.&quot;</p>
<p>OP: &quot;That didn't work.&quot;</p>
</blockquote>
<p>We could undertake to write a comprehensive answer enumerating the various knobs to attempt improving the model, perhaps in the form of &quot;symptom/diagnosis/treatment.&quot; I don't know how effective this would be, since OP will often insist &quot;but I tried that!&quot; Moreover, it's usually true that many knobs will need to be turned <em>in combination</em>.</p>
<p>By way of example, in working on a hobby neural network project, it took me more than 150 different model configurations to get a model that worked to my satisfaction, and this involved tuning a number of different model attributes <em>in combination</em> (and many hours of computing time).</p>
<p>Is there a way that we can give helpful answers to the rising tide of these rather under-specified questions? Tuning neural networks is important, and we have a large volume of questions about how to do it, but I have yet to see a good answer to any one question.</p>
