A Simple Primer on
Back Propagation Neural Networks

Terry Bahill
Systems and Industrial Engineering
University of Arizona
Tucson, AZ 85721-0020, USA
terry@sie.arizona.edu
© 1998-2004 Bahill

The field of Artificial Neural Networks is arguably the fastest growing field in Artificial Intelligence. An artificial neural network is a massively-parallel, adaptive computer-system usually having multiple inputs and multiple outputs. During the past few years, neural networks have been used in a wide variety of applications, such as signature recognition in banks, loan underwriting in mortgage companies, planning and control of robot arm trajectories, chemical process control, analyzing infrared images of asteroids, and non-linear optimization

Neural network technology has several advantages over conventional methods. Neural networks can deal with noisy and imprecise data, learn automatically from training data, adapt to a changing environment, degrade gracefully in the face of component failure, generalize to new situations, and (once trained) execute quickly. However, neural networks also suffer several weaknesses. The first is a lack of semantic interpretability. The information is stored as values of the interconnecting weights, and it is impossible to understand the behavior of a network by looking at the weight values. Second, input training sets can be faulty because of undesired or unwanted information, inappropriate training parameters, or bad initialization of connection weights. Unfortunately, it is difficult to detect such problems. Third, testing and validation are difficult with neural networks. The cost of testing a large hardware network may exceed the cost of manufacture.

There are dozens of different types of neural networks. The main differences are in their method of training and weight adaptation. We will explain the type called Back Propagation, because it is the most common type used in control systems and it can be used to illustrate most of the techniques used in other types of networks.

In this lecture we will discuss the Delta Rule for weight adjustment, the Back Propagation weight changing formula, the learning factor, effects of initial weight values, activation functions, momentum terms, and bias terms.

References [42 and 54]. This lecture is suitable for engineers or the general public. This talk requires an overhead projector. This talk takes one hour.