Lesson from the free deep learning book by Simon Prince: why non-linearity is key.

A weighted sum of lines can only result in another line.

However, if you add a ReLU step (clipping negative values to 0) before combining the lines, then you can create any piecewise linear function you want.

Each additional input line allows you to have an extra piece.

(Book: github.com/udlbook/udlbook)