<p>A few months ago, I read an interesting comment by user shimao to a question, which linked to a YouTube video where some lecturer (a mathematician, I think) showed how ReLU activations <strong>don't</strong> solve the vanishing gradient problem (contrary to "popular" belief), but ResNets do, and went on to prove the second result for linear residual connection networks. I tried to find the question, but searching for things like "ReLU solve vanishing gradient" or even just "vanishing gradient" doesn't help. I think the question was not too old. Can you help me find it? As the link was in a comment, finding it is particularly hard.  </p>
