<p>I <em>know</em> that accuracy is an improper scoring rule, thus it shouldn't be used to choose among classifiers. In particular, it should <em>not</em> be used to choose the best hyperparameters for a NN. However, since the popularization of Deep Learning, lots of people have been using validation accuracy for NN model selection. All of them, at some point, rediscover the fact that even if the validation loss is increasing, the validation accuracy may keep increasing or oscillate:</p>

<p><a href="https://stats.stackexchange.com/questions/368431/can-it-be-over-fitting-when-validation-loss-and-validation-accuracy-is-both-incr">Can it be over fitting when validation loss and validation accuracy is both increasing?</a></p>

<p>I think at least some of them would be interested in an authoritative answer to the question: "How can validation accuracy increase or oscillate, if the validation loss keeps increasing?". Do we have a canonical Q&amp;A on this? It would not only be interesting in itself, but also in order to close duplicates.</p>
