<p>There are quite a few questions asking how to run backpropagation when dealing with skip connections, residual networks, "mixed" RNN-CNNs, attention mechanisms, etc. </p>

<p>I suspect the reason for the confusion is that backpropagation is usually taught as "manual automatic differentiation": the teacher or student manually writes out the symbolic form of the derivative at each layer of a simple feed-forward network using the chain rule. </p>

<p>However backprop / automatic differentiation in general works on arbitrary computation graphs (DAGs), so that <em>any</em> computation composed of differentiable primitives can be backpropagated, including all those different types of networks mentioned above.</p>

<p>(There are also questions about how to backprop through batch norm, which is itself composed of some simple mathematical primitives and is effortlessly handled by autodiff. Authors of highly optimized libraries might want to write out the "fused" gradient of batch norm to maximize performance, but I suspect that's not what people asking the question are trying to do.)</p>

<p>So my question is, how should I answer these types of questions? I could just post an outline of how backprop works on arbitrary computation graphs, and then add some good references for backprop/autodiff. But I'd have to repeat this on every question, and whenever someone wants to know how to perform backprop on a newly invented foo-network, they'll still start a new question because it's not obvious to them that BP in foo-networks is the same as BP in every other network.</p>
