A neural net with a single, two-element hidden layer is capable of learning the identity map on the set of N standard unit vector in N-space. The exemplars map into points on a circle in two-space.
The usual training algorithms break down about N = 14, but the author's technique of using "training wheels" is successful for much larger values of N.
Bouyed with this success, we attempt to train a network to learn the identity function on a set of exemplars consisting of sums of two standard unit vectors (think of an image with two black pixels). These networks require four hidden nodes. The exemplars organize themselves into a Mobius strip in four-space. The training is, of course, accomplished using training wheels.
Such identity-function networks are generally valuable, and more so when training wheels are used in their education.
Colloquia Series page.