The OCHRE applet
This applet is named after what it does: Optical CHaracter REcognition. Documentation is included below.
Inputs arrive from other neurons, or
from the environment. By taking the
inner product, the neuron essentially
multiplies each input it receives by a weight which is specific
to that input. (This is loosely based on the fact that real neurons
are connected by synapses that vary in strength. Any given neuron in the
brain receives inputs from as many as thousands of other neurons, and it
pays more "attention" to some than to others.) The neuron adds all of these
weighted activations together, along with any bias that it might
have, to form the inner product. The inner product is then subjected to
a transfer function which is preferably nonlinear. The result is the neuron's
Backpropagation networks consist of either two or three layers of neurons, and all the neurons behave as shown above. Any given neuron receives input from all neurons in the preceding layer, and it sends its output to all neurons in the layer below. In the case of the input layer, the inputs are what we put into the network, and in the case of the output layer, the outputs are what we get from it.
One of the most typical problems to which a neural network is applied is that of optical character recognition. Recognizing characters is a problem that at first seems extremely simple- but it's extremely difficult in practice to program a computer to do it. And yet, automated character recognition is of vital importance in many industries such as banking and shipping. The U.S. post office uses an automatic scanning system to recognize the digits in ZIP codes. You may have used scanning software that can take an image of a printed page and generate an ASCII document from it. These devices work by simulating a type of neural network known as a backpropagation network.
How to work this thing
The ten icons with the digits in them, along the the top of the applet window, are the actual inputs that are used to test and train the neural network. To submit one to the network, press the "test" button that appears above it. The row of ten LEDs below the icons indicate the activity of the ten output neurons. When the network is fully trained, the only active output neuron should be the one below the icon which you just tested. It should glow bright red, while the others remain black. But on startup, the network is not trained yet- it doesn't "know" anything about what Arabic numerals look like- and so it won't know how to recognize one. You're likely to see all the neurons light up, at least a little bit.
To train the network, press the "Start training" button on the lower panel. As the applet begins to train the network, you will see the number of training epochs (individual training iterations) increase, and the sum squared error (the cumulative error in the network output) slowly decrease. You will also see the network output icon changing (at the lower right side of the applet). This icon summarizes all of the network's responses to all ten of the training inputs. Squares in the icon that are blue indicate appropriate responses by output neurons, and those that are red indicate inappropriate responses.
Training should take a few minutes, and is complete after about 150-250 training epochs, when the sum squared error reaches a low value (approximately 0.01). On my Pentium-133, it usually takes about two minutes of training for the network to reach an error that low. The network output icon should appear as a diagonal row of bright blue squares. At this point, you can press the "Stop training" button to stop training the network. If you press the "test" buttons above each digit icon, the appropriate LED should now light up.
You can test the network on your own hand-drawn symbol, by drawing one with your mouse in the large icon at the lower left of the applet. Your left mouse button will light a pixel up, and your right mouse button will darken it. (If your mouse only has one button, you can click while holding down your ALT key to darken a pixel.) You can blur or sharpen your drawing using the buttons along the left of the large icon. To see what the network thinks you just drew a picture of, press the "test" button at bottom, and see which of the LEDs light up. I have varying luck with it. Sometimes the network recognizes my digit immediately, and sometimes it refuses to unless the picture is smoothed with the blur button. One interesting thing to do is to test the network on a novel input, such as a letter "A". Neural networks behave unpredictably with novel inputs.
The network can be trained on any of your own novel inputs as well. You should be able to alter any of the digit icons at the top by drawing on it. To blur, sharpen, or clear them, use your "B", "S", and "C" keys respectively. To restore the original digits, press the "Reset inputs" button. To reset the network itself, press the "Reset network" button. You can change the number of input layer neurons and hidden layer neurons by changing the numbers in the corresponding fields before pressing "Reset network". Too few neurons, and the network will be unable to learn anything. Too many, and overlearning becomes likely- the network learns the specific training inputs so well that it won't tolerate any slight deviation from them.