Page 1.CD – CSC-20023

Page 1.CD – CSC-20023

Computational Intelligence 1
Semester One Reassessment
CSC-20023
Neural Networks Reassessment – Spring 2015
Introduction
This assignment is based upon the use of the Multi-Layer Perceptron (MLP) neural network
architecture. In Part 1 you will be required to use MLP neural networks for a pattern classification task
using digitised character data. In Part 2 you will be required to develop and test auto-associative MLP
networks.
This assignment will require that you:
? learn how to use the Java Neural Network Simulator (JNNS);
? learn to create and operate simulations of MLPs;
? learn to train, test and evaluate the performance of MLP neural networks with JNNS;
? briefly sumarise your results in a report.
Part 1: Pattern Classification of Digitised Numerals (60%)
For this part of the assignment you will train and test several multi-layer perceptron (MLP) networks to
act as pattern classifiers. Each network will try to identify whether a particular input pattern is a
digitised numeral in the range 0-9. Several important files that you will need are stored in the
numerals.zip zip-file accompanying this assignment specification on the KLE. You should make a
copy of this zip-file and unzip it in your own area on the School’s teaching network. Amongst the files
that you have copied is one called numerals_trg.pat. You should inspect the contents of this JNNS
pattern file, but do not change anything. This file contains a set of ten training patterns for the digitised
numerals 0 to 9. Each input pattern for a numeral is represented by 63 binary values: where a zero
indicates a pixel is off and a one indicates that a pixel is on. For convenience each input pattern in the
pattern file is arranged as a 7×9 tiling of input-pixel values e.g.:
convenience each input pattern in the pattern file is arranged as a 7×9 tiling of input-pixel values eg:
. . .
but you should realise that, despite this cosmetic tiling appearance, each input pattern actually remains
as a 63 unit vector!
Page 2
CD – CSC-20023
Along with each input pattern there is a corresponding target pattern in the numerals_trg.pat file.
Each target pattern is a 10 unit vector (i.e. units 1 to 10) with a single unit set to one and all other units
set to zero – to indicate the position in the alphabet (i.e. the category) that an input pattern should be
classified into. The numerals_trg.pat file will be the training data you should use for this part of
the assignment.
Using this set of training data you are required to train a number of MLP neural networks to try and
perform the numeral classification task. The MLPs used for this task must all have 63 input units and
must also have 10 output units – these constraints are dictated by the training data we have just
reviewed in the numerals_trg.pat file. The distinguishing characteristic of each of your MLPs will
be the number of hidden units in their middle layer (see below).
Amongst the other files you have copied from the numerals directory is numerals_test.pat. This
file contains some corrupted, or noisy, input patterns which can be used to test the generalising power
of your trained MLPs. You should read the JNNS-comments which accompany the input patterns in
this file (but take care not to alter any of the data!) to see what the correct classification for each
corrupted/noisy input ought to be.
You will be restricted to using only three-layer MLPs for this part of the assignment. So the only
feature of the network topology that you can vary is the number of units in the hidden layer. As a
minimum you should attempt to train at least one MLP with each of the following network topologies:
63_2_10 i.e. 2 hidden units
63_4_10 i.e. 4 hidden units
63_6_10 i.e. 6 hidden units
63_8_10 i.e. 8 hidden units
63_10_10 i.e. 10 hidden units
[When you create each of the above MLP networks using JNNS you are strongly recommended
to specify the layout of the input-layer’s nodes as a 7 by 9 tiling. This is so that when you have
the network visible in the JNNS display window, the input-layer activations take on the
appearance of the digitised numeral as they are shown in the diagrams above. In order to do
this, when you enter the input-layer specification in the (Tools|Create|Layers) Create
Layers dialog you should specify that the 63 input units are arranged as follows:
Width is 7;
Height is 9;
Top left position is 1 1 1.
Your specifications for the hidden- and output-layers should be supplied in the usual way: but
take care to specify an appropriate Top left position for each of these succeeding layers.]
Page 3
CD – CSC-20023
The decision function that you should use to interpret the MLP’s output layer activations during
training is:
unit is off if 0.0 <= activation <= 0.2
unit is on if 0.8 <= activation <= 1.0
What to hand-in for Part 1
For all of the MLPs that you train you should save the trained network in a file and record the
following information in a table in your assignment report:
? Network topology and file name
? Number of training epochs to correctly classify the training data (or the number of epochs at
which you stopped training if not correctly classifying all training patterns: stating why you
chose to stop at that point)
? Number of training patterns correctly classified (if less than all ten)
? Number of testing patterns (from the numerals_test.pat file) correctly classified
Plot a graph to show the relationship between the number of training epochs (y-axis) and the MLP
hidden layer size (x-axis) using the data from your table.
Finally, in no more than 500 words: summarise the results of your testing and you must clearly state
what decision function you applied in order to interpret output-unit activations during your testing with
the numerals_test.pat file.
Part 2: Auto-Association of Digitised Numerals (40%)
The MLP neural network architecture can also be used to build auto-associative memory networks.
Such networks are trained to reproduce copies of their input patterns at their output layer. Once trained,
these networks constitute a form of neural network memory of their training dataset and can often be
used to clean-up/restore noisy/damaged copies of the data they were originally trained to remember.
The aim of this part of the assignment is to train the smallest auto-associator MLP that will correctly
clean-up nosiy/damaged versions of the digitised numerals.
To train neural networks to act as auto-associators for the digitised numerals we will require slightly
modified network topologies and training datasets.
Since the MLPs will be trained to regurgitate approximate copies of their input patterns at their output
layers it is important that both input and output layers have exactly the same number of units. For our
digitised numerals this means that the input and output layers will both need to consist of 63 units.
(Once again it would be best to specify that the input and output units should be displayed in JNNS as a
7×9 tiling.) The size of the hidden layer of each MLP network is a network-topology parameter that
we can choose freely: however, we usually seek the network with the smallest hidden layer that can
Page 4
CD – CSC-20023
successfully accomplish a given task. Clearly, the smallest possible hidden layer consists of 1 hiddenunit
(i.e. a 63_1_63 network), then 2 hidden-units (i.e. a 63_2_63 network) and so on. If the hidden
layer is too small the MLP will not be able to accomplish its memorisation task to a good level of
accuracy.
The pattern files that hold the auto-associative training and testing data are also different:
numerals_aatrg.pat
numerals_aatest.pat
In addition to the various JNNS pattern files that you have unzipped from the numerals.zip file
there should also be an executable Java program called DFuncAA.
Inside the auto-associative training dataset the target data has been edited so that it is exactly the same
as the input data. For the testing dataset the noisy/damaged versions of the digitised numerals are
presented to the input layer, but you also need perfect copies of the desired target-pattern for each
numeral (this is just to verify our off-line checking of the network’s outputs, see below).
When the test dataset is used to test a trained network it is possible to laboriously check the activation
of each output unit manually and apply a decision function to each one. Though with 63 output units
and several networks to train this is not recommended! Instead we can use another feature of JNNS
and a special Java program to produce output that is easier to interpret. This can be done as follows:
1. train an auto-associative MLP with the auto-associative training dataset;
2. load the auto-associative testing dataset and select it for use;
3. on the JNNS Save Data dialog (File | Save data) select the Result files *.res filetype
from the combo-box that appears in the dialog window. Then specify a file-name (with a
.res extension) of your own choosing, that the test-results should be stored in and generate the
file by clicking on Save;
a. in the Saving details dialog box that now appears make sure that you check the
boxes to ensure that both input patterns and output patterns are placed into the results
file, and that the Create (rather than Append) radio-button is selected;
b. then click OK
4. now open the Windows Command Prompt window and navigate to your JNNS working
directory, and ensure that your JNNS results file and the DFuncAA program are both present,
then you can execute the program DFuncAA as follows:
java DFuncAA <resfilename>
where <resfilename> is the name of the JNNS results file you have just created.
The DFuncAA program applies a decision function as follows:
unit is off if 0.0 <= activation < 0.5
unit is on if 0.5 <= activation <= 1.0
Page 5
CD – CSC-20023
and then displays the resulting patterns in form that is easier to read than manually scrutinising the data
on the JNNS display screen.
What to hand-in for Part 2
Summarise the results you have obtained with auto-associative networks that you have trained and
tested in a table. In the table clearly indicate which network you consider to be the best auto-associator
that you have trained. The filename of each trained auto-associative MLP should be clearly recorded in
your table and all of those network files should be part of your electronic submission for the
assignment (see below).
Marking criteria
Part 1 and Part 2 are marked on the comprehensiveness of your search for good neural network
implementations of the numeral classification and auto-association tasks. Your final assignment report
should be between 500 to 1000 words long and contain a short description of the work you have carried
out to solve both of the problems described above. You should ensure that any tables and graphs that
you produce in your report are clearly labeled and annotated. The file-names of all of the fully trained
neural networks should also be clearly recorded.
For Part 1 remember you must clearly state what decision function(s) you used to interpret the output
from your networks during your testing.
For Part 2 you should describe how well the best auto-associative network you obtained performed and
explain your rationale for deciding that it was the best.
Some objective reflection, on the common pitfalls and problems that arose with the properties of the
networks you developed during the assignment, should be present in a good report.
Submission Details:
The deadline for this assignment is 12noon on Thursday 11/6/2015. As part of your submission for this
assignment you must write a word-processed assignment-report and store it in a pdf file called
assignment.pdf.
Along with all of the JNNS network and pattern files needed to run your neural networks, this report
should be incorporated into a single zip-file using the compression software tools in the School’s Labs.
(Please do not use any other exotic/freeware compression tools that you might have access to, as this
could impair the examiners when they are assessing your work!) Also, take care to ensure that none of
the zipped files/folders are password protected – as this will prevent the marker from being able to
assess your submission. The compressed file should then be submitted via the CSC-20023 KLE pages:
go to the Assignments section of the module in KLE and attach the compressed file according to
the instructions found there for submitting a single zipped-file. It is important that you follow
these instructions precisely.
(Remember to use the correct file name for your assignment report!)