(Applied) Artificial Intelligence

Assignment 5: Neural Networks

Briefly:

Implement learning in neural networks based on backpropagation algorithm, in order to get acquainted with the amount of computations needed to train a network and the importance of proper handling of training parameters. For a better grade more experimenting and reasonable conclusions from those experiments are expected.

WARNING

Programming assignments, with possible exceptions of second and third, are expected to be solved in one, maximum two days of *hard* work, by an average student. Do more if you feel it is challenging, interesting or simply fun, but do not overdo it and do not read more from the description than is there. (BUT: do not read less either:-)

Long version, with all the legalese:

(OBLIGATORY) Implement the backpropagation algorithm. Test the basic working of your implementation on a simple training set such as the medical diagnosis example below. The medical diagnosis case is a warm-up: do not report on your results for this example.
(OBLIGATORY) Perform analysis of the impact of the number of hidden units on the classification quality and learning speed.
(OPTIONAL) Some other possible improvements are listed below in a separate section entitled just "Improvements".

The assignment should be carried out on an individual basis. You are welcome to discuss the problems with others, but you must do the implementation all by yourself and you must write the resulting report all by yourself.

1. Warm-up -- A medical diagnosis problem (an obligatory part):

An extract from a medical database contains eight attributes signifying characteristics such as 'high blood pressure', 'paleness', 'fever', etc, denoted by true (1) or false (0) in a set of examples. For each example the true diagnose, i.e. the classification, is recorded as A: (1 0 0), B: (0 1 0) and C: (0 0 1), respectively.

Implement the backpropagation algorithm. Use a net with eight input units (plus bias), three output units and one hidden layer with a number of units of your own choice.

a) Train your net to recognize the twelve examples below. The files containing the examples are available in /usr/local/cs/tai/neural-nets/examples/medical.txt.

b) Answer the question: What are the diagnoses for the following two patients whose clinical symptoms are described by:

patient 1's symptoms: (1 1 0 0 1 1 0 0) Right answer: A? B? or C?
patient 2's symptoms: (0 0 1 1 0 0 1 1) Right answer: A? B? or C?

     Symptoms         Diagnosis
1 1 1 0 0 0 0 0        1 0 0
0 1 0 0 1 0 0 0        0 1 0
1 0 0 0 0 1 0 1        0 0 1
1 0 1 1 0 0 0 0        1 0 0
0 1 0 1 1 0 1 0        0 1 0
0 0 0 0 0 1 1 1        0 0 1
0 0 0 1 0 1 1 1        1 0 0
0 1 0 0 1 1 1 0        0 1 0
0 0 1 0 0 0 1 1        0 0 1
0 1 1 0 0 0 0 1        1 0 0
0 0 0 0 1 1 0 0        0 1 0
1 0 0 1 0 1 0 1        0 0 1

The training is suggested to be run until the mean error E of an epoch is less than 0.0001. Advice: Don't expect 1.0 and 0.0 as output values. Instead, consider values sufficiently close, e.g. >0.9 and <0.1 as a valid indication of true and false.

The mean error for a completed epoch is computed as

E = Sum(p=1,N) (Sum(i=1,m) ((y_p_i - o_p_i)²)) / N ,

where p ranges over the number of examples (patterns), i.e. 1 to N, i ranges over the number of output nodes, i.e. 1 to m, , y_p_i is the specified target output of output node i in example p, and o_p_i is the actual output from that node.

Note: The results of the warm-up should not be presented in the report.

2. Impact of the number of hidden units (an obligatory part):

Take a look at the example sets in the directory /usr/local/cs/tai/neural-nets/databases/. They are retrieved (a long time ago) from a repository intended for testing and experimenting with machine learning algorithms including neural nets. You will find that the example sets are taken from a wide range of applications. They also vary in size, number of attributes and some of them contain unknown values. You will find a description of a datatbase in the file *.names, while the training set will be found in *.data. You are free to handle the input of the data in any way you want, i.e., do not waste time on any fancy input-handling program.

Your task: Select one of the less complicated databases of your liking and apply your backpropagation algorithm. Make some experiments with the number of hidden units to see if you can draw some conclusions of their impact.

If you think that the example sets are too simple, use the net to find more. In particular, on the Links page I have put a reference to a repository of pointers to interesting data sets.

3. Improvements (the optional part):

For the following select another data set, either from the local repository /usr/local/cs/tai/neural-nets/databases/, or from the net. Improve your solution according to two or more of the following suggestions:

Experiment with the number of hidden units. Try to find a 'reasonable' number of hidden units for one or two fairly complicated databases. What will happen if you use too few hidden units? Too many?
Investigate if there are databases that require two hidden layers.
Use multiple different partitions to train and test the net. A partition is formed when you decide to use some of the examples for training and some (or the rest) for testing. Keep records of the convergence speed during your experiments and present the results.
Experiment with the gain term and the momentum term (first of course find out what they are). Make a table of their combined effect on the convergence speed for one or more databases.
Evaluate the database with respect to the dependencies between the attributes. You can do this by varying the classifying (goal) attribute. E.g. in the flags database - is it possible to predict the religion of a country from its size and the colours in its flag? Try to find such interesting or unexpected dependencies.
Any other relevant experiment of your own choice.

One more data set

I have provided one more data set, this time coming from our scientific cooperation partner from University of Alicante, David Gil Mendez. The data describes several attributes related to a particular disease of the lower urinary tract. The rightmost column corresponds to the diagnosis. The data is divided into a training set and a test set, but feel free to rearrange them at will.

Final remarks

Deadline:

The report must be handed in for evaluation before 23.59 on Thursday, April 30th, 2009. The report should be put in the box labeled TAI (EDA132 or DATE15, respectively) in the stairwells outside the secretary offices (LTH or NF).

Problems:

Send e-mail to aikurs AT cs.lth.se .

The report:

The assignment must be documented in a report, which should contain the following:

A presentation of the assignment and the tasks you have chosen.
A presentation of your implementation and how to run the executable (so that your results may be verified).
Description of the results (both tabular and graphical forms are encouraged).
In case you used code from the repository (or anywhere else, for that matter), state which part is yours and which is not. You do not need to provide all or any code printout in the report - the (source!) code is available in your solution directory - but only comments on its interesting parts, including the authorship.
Comment on the results you have achieved.
The report should be nicely typeset and formatted.
The front page should state the name of the author, that the report concerns Assignment 4b, and any relevant information you think could be useful.
The report must be filed in on paper. Neither e-mail nor web pages will be accepted. You may receive feedback, but do not count on the report to be returned, so do not hand in originals.

Programming language and environment:

You may use any suitable implementation language and you may use your own computer to develop your program. However, the documented results of your investigations must be possible to be validated by executing your code on the *.student.lth.se Linux computers. This version should of course be the same as the one handed in (or quoted) as program source.

Your final program must be available and runnable at the E-house LINUX computers at the *.student.lth.se net (e.g. login-4.student.lth.se). It is NOT possible to hand in a medium containing the program. Remember to make your programs and all the directories in their path read and execute accessible to 'world' (chmod -R 705 ). Remember also to quote where does your solution reside and how should it be run (kind of "User's Guide").

The resulting programs should remain in your directory until you have been notified of the result, e.g. on the notice board and/or web or by e-mail. You may expect that your report and implementation will be examined within two weeks. If your report or implementation is unsatisfactory you will be given one chance to make the corrections and then to hand it in within a week after you have been notified (on the notice board and/or web or by e-mail).

On Assignment 4

Please note that there are three assignments bearing number 4: one about decision trees, one about neural nets (this one) and one about genetic algorithms. You are expected to file in only one of them in order to get Assignment 4 passed. Filing two or three will not count towards the total number of assignments filed in, although you may be interested in getting feedback on them anyway.

Have fun!

Last update:
Wednesday, 12-Jan-2011 16:23:27 CET

[an error occurred while processing this directive]