Neural Networks

      No Comments on Neural Networks

Artificial neural network is an inspirational technology of biological neural network. In general a biological neural network is composed of a group or groups of chemically connected or functionally associated neurons. A single neuron may be connected to many other neurons and the total number of neurons and connections in a network may be extensive. Connections, called synapses, are usually formed from axons to dendrites, though other kind of connections, also possible.

An Artificial Neural Network is also called as simulated neural network of many very simple processors (“units”), each possibly having a (small amount of) local memory. The units are connected by unidirectional communication channels (“connections”), which carry numeric (as opposed to symbolic) data. The units operate only on their local data and on the inputs they receive via the connections. Artificial neural networks are made up of interconnecting artificial neurons (programming constructs that mimic the properties of biological neurons). Artificial neural networks may either be used to gain an understanding of biological neural networks, or for solving artificial intelligence problems without necessarily creating a model of a real biological system. The real, biological nervous system is highly complex and includes some features that may seem superfluous based on an understanding of artificial networks.


The concept neural networks started in the late-1800s as an effort to describe how the human mind performed. These ideas started being applied to computational models with Turing’s B-type machines and the perceptron. In 1940  Donald Hebb made one of the first hypotheses for a mechanism of neural plasticity  (i.e. learning), Hebbian learning. Hebbian learning is considered to be a ‘typical’ unsupervised learning rule. In 1943 McCulloch and Pitts proposed the McCulloch-Pitts neuron model. In 1948 Alan Tuning, first proposed the concept of Neural Networks in his paper “Intelligent Machinery”.  In 1949 Hebb published his book The Organization of Behavior, in which the Hebbian  learning rule was proposed. In 1958 Rosenblatt introduced the simple single layer networks now called Perceptrons. In 1969 Minsky and Papert’s book Perceptrons demonstrated the limitation of single. In layer perceptrons, and almost the whole field went into hibernation. In 1982 Hopfield published a series of papers on Hopfield networks. In 1982 Kohonen developed the Self-Organizing Maps that now bear his name. In 1986 The Back-Propagation learning algorithm for Multi-Layer Perceptrons was rediscovered and the whole field took off again. In 1990s The sub-field of Radial Basis Function Networks was developed. In 2000s The power of Ensembles of Neural Networks and Support Vector Machines becomes apparent.


 Neural networks take a different approach to problem solving than that of conventional computers. Conventional computers use an algorithmic approach i.e. the computer follows a set of instructions in order to solve a problem. Unless the specific steps that the computer needs to follow are known the computer cannot solve the problem. That restricts the problem solving capability of conventional computers to problems that we already understand and know how to solve. But computers would be so much more useful if they could do things that we don’t exactly know how to do.

Neural networks process information in a similar way the human brain does. The network is composed of a large number of highly interconnected processing elements(neurons) working in parallel to solve a specific problem. Neural networks learn by example. They cannot be programmed to perform a specific task. The examples must be selected carefully otherwise useful time is wasted or even worse the network might be functioning incorrectly. The disadvantage is that because the network finds out how to solve the problem by itself, its operation can be unpredictable.

On the other hand, conventional computers use a cognitive approach to problem solving; the way the problem is to solved must be known and stated in small unambiguous instructions. These instructions are then converted to a high level language program and then into machine code that the computer can understand. These machines are totally predictable; if anything goes wrong is due to a software or hardware fault.

Neural networks and conventional algorithmic computers are not in competition but complement each other. There are tasks are more suited to an algorithmic approach like arithmetic operations and tasks that are more suited to neural networks. Even more, a large number of tasks, require systems that use a combination of the two approaches (normally a conventional computer is used to supervise the neural network) in order to perform at maximum efficiency.

Neural networks, with their remarkable ability to derive meaning from complicated or imprecise data, can be used to extract patterns and detect trends that are too complex to be noticed by either humans or other computer techniques. A trained neural network can be thought of as an “expert” in the category of information it has been given to analyze. This expert can then be used to provide projections given new situations of interest and answer “what if” questions.
Other advantages include:

  1. Adaptive learning: An ability to learn how to do tasks based on the data given for training or initial experience.
  2. Self-Organization: An ANN can create its own organization or representation of the information it receives during learning time.
  3. Real Time Operation: ANN computations may be carried out in parallel, and special hardware devices are being designed and manufactured which take advantage of this capability.
  4. Fault Tolerance via Redundant Information Coding: Partial destruction of a network leads to the corresponding degradation of performance. However, some network capabilities may be retained even with major network damage.


 An artificial neuron is a device with many inputs and one output. The neuron has two modes of operation; the training mode and the using mode. In the training mode, the neuron can be trained to fire (or not), for particular input patterns. In the using mode, when a taught input pattern is detected at the input, its associated output becomes the current output. If the input pattern does not belong in the taught list of input patterns, the firing rule is used to determine whether to fire or not.


Artificial neural networks are quite different from the brain in terms of structure. Like the brain, however, a neural net is a massively parallel collection of small and simple processing units where the interconnections form a large part of the network’s intelligence; however, in terms of scale, a brain is massively larger than a neural network, and the units used in a neural network are typically far simpler than neurons. Nevertheless, certain functions that seem exclusive to the brain such as learning, have been replicated on a simpler scale, with neural networks.


A typical feed-forward neural network is a set of nodes. In computing, feed-forward normally refers to a multi-layer perceptron network in which the outputs from all neurons go to following but not preceding layers, so there are no feedback loops. Some of these are designated input nodes, some output nodes, and in-between are hidden nodes. Each connection between neurons has a numerical weight. When the network is in operation, a value will be applied to each input node — the values being fed in by a human operator, from environmental sensors, or from some external program. Each node then passes its given value to the connections leading out from it, and on each connection the value is multiplied by the weight associated with that connection. Each node in the next layer then receives a value which is the sum of the values produced by the connections leading into it, and in each node a simple computation is performed on the value — a sigmoid function is typical. This process is then repeated, with the results being passed through subsequent layers of nodes until the output nodes are reached. Early models (circa 1970) had a fixed number of layers. More recently, genetic algorithms are used to evolve the neural structure


The sigmoid curve is often used as a transfer function because it introduces non-linearity into the network’s calculations by “squashing” the neuron’s activation level into the range [0,1]. The sigmoid function has the additional benefit of having an extremely simple derivative function, as required for back-propagating errors through a feed-forward neural network. Other functions with similar features can be used, most commonly tanh which squashes activations into the range of [-1,1] instead, or occasionally a piece-wise linear function that simply clips the activation rather than squashing it.

If no non-linearity is introduced by squashing or clipping, the network loses much of its computational power, becoming a simple matrix multiplication operation from linear algebra.

Alternative calculation models in neural networks include models with loops, where some kind of time delay process must be used, and “winner takes all” models, where the neuron with the highest value from the calculation fires and takes a value 1, and all other neurons take the value 0.

Typically the weights in a neural network are initially set to small random values. This represents the network knowing nothing; its output is essentially a random function of its input. As the training process proceeds, the connection weights are gradually modified according to computational rules specific to the learning algorithm being used. Ideally the weights eventually converge to values allowing them to perform a useful computation. Thus it can be said that the neural network commences knowing nothing and moves on to gain some real knowledge, though the knowledge is sub-symbolic.



 Feed-forward ANNs allow signals to travel one way only; from input to output. There is no feedback (loops) i.e. the output of any layer does not affect that same layer. Feed-forward ANNs tend to be straight forward networks that associate inputs with outputs. They are extensively used in pattern recognition. This type of organization is also referred to as bottom-up or top-down. ex.,  Fault Diagnosis systems.


Feedback networks can have signals traveling in both directions by introducing loops in the network. Feedback networks are very powerful and can get extremely complicated. Feedback networks are dynamic; their ‘state’ is changing continuously until they reach an equilibrium point. They remain at the equilibrium point until the input changes and a new equilibrium needs to be found. Feedback architectures are also referred to as interactive or recurrent, although the latter term is often used to denote feedback connections in single-layer organizations. Ex.,Radiosity for Virtual Reality Systems (ROVER)


A neural network has to be configured such that the application of a set of inputs produces (either ‘direct’ or via a relaxation process) the desired set of outputs. Various methods to set the strengths of the connections exist. One way is to set the weights explicitly, using a priori knowledge. Another way is to ‘train’ the neural network by feeding it teaching patterns and letting it change its weights according to some learning rule.

We can categorize the learning situations in three distinct sorts. They are:

  • Supervised learning or Associative learning in which the network is trained by providing it with input and matching output patterns. These input-output pairs can be provided by an external teacher, or by the system which contains the neural network (self-supervised).

This method has two sub varieties called auto-associative and hetero-associative. In auto-associative learning, the target values are the same as the inputs, whereas in hetero-associative learning, the targets are generally different from the inputs.

One of the most commonly used supervised NN model is back propagation network that uses back propagation learning algorithm. Back propagation (or back prop) algorithm is one of the well-known algorithms in neural networks.

  • Unsupervised learning method is not given any target value. Unsupervised learning or Self-organization in which an (output) unit is trained to respond to clusters of pattern within the input. In this paradigm the system is supposed to discover statistically salient features of the input population. Unlike the supervised learning paradigm, there is no a priori set of categories into which the patterns are to be classified; rather the system must develop its own representation of the input stimuli. A desired output of the network is unknown. During training the network performs some kind of data compression such as dimensionality reduction or clustering. The network learns the distribution of patterns and makes a classification of that pattern where, similar patterns are assigned to the same output cluster.

 Kohonen network is the best example of unsupervised learning network. According to Sarle (1997) Kohonen network refers to three types of networks that are Vector Quantization, Self-Organizing Map and Learning Vector Quantization. 

  • The third major category of learning that will be discussed is termed “reinforcement learning”. As mentioned earlier, reinforcement learning lies between supervised and unsupervised learning. It is often called learning with a critic rather than learning with a teacher as the feedback is evaluative (right or wrong) rather than instructive (desired output). Reinforcement learning in neural networks has its roots in behavioral psychology. Here the learning machine does some action on the environment and gets a feedback response from the environment. The learning system grades its action good (rewarding) or bad (punishable) based on the environmental response and accordingly adjusts its parameters. Generally, parameter adjustment is continued until an equilibrium state occurs, following which there will be no more changes in its parameters. The self organizing neural learning may be categorized under this type of learning.


  • Signal processing: suppress line noise, with adaptive echo canceling, blind source separation
  • Control: e.g. backing up a truck: cab position, rear position, and match with the dock get converted to steering instructions. Manufacturing plants for controlling automated machines.
  • Siemens successfully uses neural networks for process automation in basic industries, e.g., in rolling mill control more than 100 neural networks do their job, 24 hours a day
  • Robotics – navigation, vision recognition
  • Pattern recognition, i.e. recognizing handwritten characters, e.g. the current version of Apple’s Newton uses a neural net
  • Medicine, i.e. storing medical records based on case information
  • Speech production: reading text aloud (NETtalk)
  • Speech recognition
  • Vision: face recognition , edge detection, visual search engines
  • Business, eg.. rules for mortgage decisions are extracted from past decisions made by experienced evaluators, resulting in a network that has a high level of agreement with human experts.
  • Financial Applications: time series analysis, stock market prediction
  • Data Compression: speech signal, image, e.g. faces
  • Game Playing: backgammon, chess, ROBOCUP.

Robocup: Robot World Cup:-

The RoboCup Competition pits robots (real and virtual) against each other in a simulated soccer tournament. The aim of the RoboCup competition is to foster an interdisciplinary approach to robotics and agent-based AI by presenting a domain that requires large-scale cooperation and coordination in a dynamic, noisy, complex environment. RoboCup has three different leagues to-date. The Small and Middle-Size Leagues involved physical robots; the Simulation League is for virtual, synthetic teams. This work focus on building softbots for the Simulation League. Machine Learning for Robocup involves:

1.The training of player in the process of making the decision of whether (a) to dribble the ball; (b) to pass it on to another team-mate; (c) to shoot into the net.

 2. The training of the goalkeeper in process of intelligent guessing of how the ball is going to be kick by the opponents. Complexities arise when one opponent decides to pass the ball to another player instead of attempting a score.

3. Evolution of a co-operative and perhaps unpredictable team.


The computing world has a lot to gain from neural networks. Their ability to learn by example makes them very flexible and powerful. Furthermore there is no need to device an algorithm in order to perform a specific task; i.e. there is no need to understand the internal mechanisms of that task. They are also very well suited for real time systems because of their fast response and computational times which are due to their parallel architecture.

Neural networks also contribute to other areas of research such as neurology and psychology. They are regularly used to model parts of living organisms and to investigate the internal mechanisms of the brain.

Perhaps the most exciting aspect of neural networks is the possibility that some day ‘conscious’ networks might be produced. There is a number of scientists arguing that consciousness is a ‘mechanical’ property and that ‘conscious’ neural networks are a realistic possibility.

Finally, to state that even though neural networks have a huge potential we will only get the best of them when they are integrated with computing, AI, fuzzy logic and related subjects.

Leave a Reply