**Neural Networks**

· __Problems Not Suited to a Neural Network Solution__- Programs that are easily written out as flowcharts consisting of well-defined steps, program logic that is unlikely to change, problems in which you must know exactly how the solution was derived.

· __Problems Suited to a Neural Network__ – pattern recognition__, __classification, series prediction, and data mining. Pattern recognition - network attempts to determine if the input data matches a pattern that it has been trained to recognize. Classification - take input samples and classify them into fuzzy groups.

· As far as machine learning approaches go, I thing SVMs are superior (see

http://en.wikipedia.org/wiki/Support_vector_machine ) - a neural network has certain disadvantages in comparison: an ANN can be overtrained, different training sets can produce non-deterministic weights and it is not possible to discern the underlying decision function of an ANN from its weight matrix – they are black box.

· In this post, I'm not going to go into internals (believe me I know them). An autoassociative network (e.g. a Hopfield network) will echo back a pattern if it is recognized.

· Under the hood, there is very little maths. In a nutshell - Some simple matrix operations occur during training: the input array is processed (normalized into bipolar values of 1, -1) - transposed from input column vector into a row vector, these are subject to matrix multiplication and then subtraction of the identity matrix to get a contribution matrix. The dot product is taken against the weight matrix to yield a boolean match result. For backpropogation training, a derivative function is required. In learning, hill climbing mechanisms such as Genetic Algorithms and Simulated Annealing are used to escape local minima. For unsupervised training, such as found in Self Organizing Maps used for OCR, Hebbs rule is applied.

· The purpose of this post is not to mire you in technical and conceptual details, but to show you how to leverage neural networks via an abstraction API - Encog

** **

**Encog**

· Encog is a neural network API

· Encog requires .Net 3.5 or higher – there is also a Silverlight version. Third-Party Libraries – log4net and nunit.

· Encog supports feedforward, recurrent, self-organizing maps, radial basis function and Hopfield neural networks.

· Encog neural networks, and related data, can be stored in **.EG** XML files.

· Encog Workbench allows you to edit, train and visualize neural networks. The Encog Workbench can generate code.

**Synapses and layers **

· __the primary building blocks__ - Almost every neural network will have, at a minimum, an input and output layer. In some cases, the same layer will function as both input and output layer.

· To adapt a problem to a neural network, you must determine how to feed the problem into the input layer of a neural network, and receive the solution through the output layer of a neural network.

· __The Input Layer__ - For each input neuron, one **double** value is stored. An array is passed as input to a layer. Encog uses the interface **INeuralData **to hold these arrays. The class **BasicNeuralData **implements the **INeuralData **interface. Once the neural network processes the input, an **INeuralData** based class will be returned from the neural network's output layer.

· convert a double array into an **INeuralData** object :

**INeuralData data = new BasicNeuralData(= new double[10]); **

· __the Output Layer__- The neural network outputs an array of doubles, wraped in a class based on the **INeuralData **interface.

· The real power of a neural network comes from its pattern recognition capabilities. The neural network should be able to produce the desired output even if the input has been slightly distorted.

· __Hidden Layers__– optional. between the input and output layers. very much a “black box”. If the structure of the hidden layer is too simple it may not learn the problem. If the structure is too complex, it will learn the problem but will be very slow to train and execute. Some neural networks have no hidden layers. The input layer may be directly connected to the output layer. Further, some neural networks have only a single layer. A single layer neural network has the single layer self-connected.

· connections, called synapses, contain individual weight matrixes. These values are changed as the neural network learns.

**Constructing a Neural Network **

· the XOR operator is a frequent “first example” -the “Hello World” application for neural networks.

· The XOR Operator- only returns true when both inputs differ.

**0 XOR 0 = 0 **

**1 XOR 0 = 1 **

**0 XOR 1 = 1 **

**1 XOR 1 = 0 **

· __Structuring a Neural Network for XOR__ - two inputs to the XOR operator and one output.

· input:

**0.0,0.0 **

**1.0,0.0 **

**0.0,1.0 **

**1.0,1.0 **

· Expected output:

· A Perceptron - a simple feedforward neural network to learn the XOR operator.

· Because the XOR operator has two inputs and one output, the neural network will follow suit. Additionally, the neural network will have a single hidden layer, with two neurons to help process the data. The choice for 2 neurons in the hidden layer is arbitrary, and often comes down to trial and error.

· Neuron Diagram for the XOR Network

·
· The Encog workbench displays neural networks on a layer-by-layer basis.

· Encog Layer Diagram for the XOR Network:

· __Create a __**BasicNetwork **- Three layers are added to this network. the **FinalizeStructure **method must be called to inform the network that no more layers are to be added. The call to **Reset** randomizes the weights in the connections between these layers.

**var network = new **__BasicNetwork__();

**network.**__AddLayer__(new __BasicLayer__(2));

**network.AddLayer(new BasicLayer(2)); **

**network.AddLayer(new BasicLayer(1)); **

**network.Structure.**__FinalizeStructure__();

**network.**__Reset__();

· Neural networks frequently start with a random weight matrix. This provides a starting point for the training methods. These random values will be tested and refined into an acceptable solution. However, sometimes the initial random values are too far off. Sometimes it may be necessary to reset the weights again, if training is ineffective. These weights make up the long-term memory of the neural network. Additionally, some layers have threshold values that also contribute to the long-term memory of the neural network. Some neural networks also contain context layers, which give the neural network a short-term memory as well. The neural network learns by modifying these weight and threshold values.

· Now that the neural network has been created, it must be trained.

**Training a Neural Network **

· __construct a __**INeuralDataSet** object - contains the input array and the expected output array (of corresponding range). Even though there is only one output value, we must still use a two-dimensional array to represent the output.

**public static double[][] XOR_INPUT ={ **

**new double[2] { 0.0, 0.0 }, **

**new double[2] { 1.0, 0.0 }, **

**new double[2] { 0.0, 1.0 }, **

**new double[2] { 1.0, 1.0 } }; **

**public static double[][] XOR_IDEAL = { **

**new double[1] { 0.0 }, **

**new double[1] { 1.0 }, **

**new double[1] { 1.0 }, **

**new double[1] { 0.0 } }; **

__INeuralDataSet__ trainingSet = new __BasicNeuralDataSet__(XOR_INPUT, XOR_IDEAL);

· Training is the process where the neural network's weights are adjusted to better produce the expected output. Training will continue for many iterations, until the error rate of the network is below an acceptable level. Encog supports many different types of training. Resilient Propagation (RPROP) - general-purpose training algorithm. All training classes implement the **ITrain **interface. The RPROP algorithm is implemented by the **ResilientPropagation** class. Training the neural network involves calling the **Iteration **method on the **ITrain** class until the error is below a specific value. The code loops through as many iterations, or epochs, as it takes to get the error rate for the neural network to be below 1%. Once the neural network has been trained, it is ready for use.

__ITrain__ train = new __ResilientPropagation__(network, trainingSet);

**for (int epoch=0; epoch < 10000; epoch++)**

**{ **

**train.**__Iteration__();

**Debug.Print("Epoch #" + epoch + " Error:" + train.Error); **

**if (train.Error < 0.01) break;**

**} **

**Executing a Neural Network **

· Call the **Compute **method on the **BasicNetwork** class.

**Console.WriteLine("Neural Network Results:"); **

**foreach (**__INeuralDataPair__ pair in __trainingSet__)

**{ **

**INeuralData output = network.**__Compute__(pair.__Input__);

**Console.WriteLine(pair.Input[0] + "," + pair.Input[1] + ", actual=" + output[0] + ",ideal=" + pair.Ideal[0]); **

**} **

· The **Compute **method accepts an **INeuralData **class and also returns a **INeuralData** object. Neural Network Results:

**0.0,0.0, actual=0.002782538818034049,ideal=0.0 **

**1.0,0.0, actual=0.9903741937121177,ideal=1.0 **

**0.0,1.0, actual=0.9836807956566187,ideal=1.0 **

**1.0,1.0, actual=0.0011646072586172778,ideal=0.0 **

· the network has not been trained to give the exact results. This is normal. Because the network was trained to 1% error, each of the results will also be within generally 1% of the expected value.