4005-759-01: Pattern Recognition (RIT CS, 20092)

Department of Computer Science
Pattern Recognition (Topics in Artificial Intelligence, Winter 2009-2010)
4005-759-01 (Calendar Description)
Home --- MyCourses --- Syllabus --- Slides and Notes --- Assignments --- PRTools Examples --- Resources

Instructor: Richard Zanibbi, Office: 70-3551 (Golisano College)
Office hours: 2-3pm Tues. and Thurs., 10am-12pm on Wednesdays
Lectures: 4-5:50pm Tuesdays and Thursdays, Room 70-3445 (Golisano College)

Project: Handwritten Digit Recognizer

Weight: 20% of final grade

For the course project, students will work in teams to construct a pattern recognizer for handwritten digits. Students will use the MNIST dataset for experiments. This data set contains 60,000 training images, and 10,000 testing images. Yann LeCun, one of the individuals that developed the MNIST dataset was also involved in the creation of algorithms for recognizing the set, including the LeNet5 Convolutional Neural Network.

Students may use any language of their choosing for the project, though at least starting with the PRTools MATLAB toolkit is recommended (see below). MATLAB is recommended, because it provides a simple, complete environment for implementing algorithms, running experiments, and visualizing results. PRTools makes it very easy to generate data, simple classifiers, etc. within the MATLAB environment. The course web pages provide links to available libraries for pattern recognition under the "Resources" link; there are others available elsewhere.

Please consult the "Resource" page in the course web pages for materials on carrying out research and writing research papers.

PRTools Data and Utility Functions for MNIST

To obtain the MNIST data set, download the files stored here, and place them in a directory within your MATLAB search path. The training and testing sets are raw image data: each feature vector is a 28 x 28 image, stored as a single vector with 784 elements.

Loading: You can load the 60,000 training images using the command load mnist_train, and the 10,000 testing images using the command load mnist_test. These will define PRTools datasets in your workspace named Train and Test. To see a description of their contents, use struct, e.g. struct(Train).
Viewing Images: use the PRTools 'show' command to view images (issue 'help dataset/show' in MATLAB for details). Once you have loaded the training set, you can issue the command: show( selx( Train, 1, 200 ) ) to view the first 200 images.
Selecting Instances (Inputs): I have defined two functions to make it easy to select images/instances from a PRTools dataset. selx( Data, start, finish ) returns a new PRTools data set containing elements [start: finish] in Data.
selx makes use of a more general selection function called selectx. To select all the testing data images containing the digit '3', and then print the first 200 of them, issue the following:
Threes = selectx( Test, findlabels(Test, [3]) );
show( selx(Threes, 1, 200) );
The semi-colon (;) prevents MATLAB from printing out the results of an operation (this can have a significant impact on the speed of a MATLAB program). Here findlabels is a PRTools method used to obtain a vector of index values for instances in a dataset with given label(s) (here just threes: [3]).

Once you have downloaded the data, you can use the built-in PRTools functions to generate simple classifiers (e.g. linear discriminant classifiers using ldc), apply feature transformations, etc. (see the PRTools documentation).

Part I. Proposal

Due: December 17, 2009 (start of class Thurs. Week 3) Weight: 5% of final grade

Each team will provide a proposal for their digit recognizer (maximum 5 pages single-spaced, including references). It must include:

A brief description of the problem and data set
A brief description of three to five references which present various state-of-the-art techniques for recognizing MNIST. the
Problem description, including challenges in recognizing handwritten (and particularly) MNIST digits
Features to be used in your system for classification
Pattern classification and/or machine learning algorithms to be used in your system
A (rough) design for your experiment: what error metrics will you use, what variables will be manipulated, and how will classifiers be compared?

The instructor will use this to check that experiments are well-defined, appropriate, and executable within the quarter. Reports will be graded for clarity, completeness, and correctness. The proposal does not need to describe an implemented system.

Part II. Final Report

Due: February 18, 2010 (start of class Thurs. Week 10) Weight: 15% of final grade

Each team will provide a technical report (maximum 10 pages, including references) summarizing the outcome of their experiment, and comparing their results to published results. The report will include:

Problem description, including challenges in recognizing handwritten digits
Features, algorithms, error metrics, etc. used for experiments
Experimental design ('methods')
Results: comparison of error rates for different algorithms, etc., and comparison with other published results
Discussion, where you are encouraged to conjecture about possible influences on the results (i.e. you can make claims that you do not have to provide evidence for in this section), and identify further experiments of interest. You do not need a formal conclusion; the discussion will serve this purpose as well.
References

The performance of your algorithms is important, but do not worry if your algorithm is not performing as well as or better than the state-of-the-art: you primarily need to make a serious attempt at creating an effective algorithm, and then be able to intelligently discuss the outcome of your experiment. Reports will be graded for clarity, completeness, correctness, and reproducibility: the report should provide enough detail to allow someone working in pattern recognition to repeat the experiment.

Make sure to use use figures and tables to summarize your results and clarify your presentation.