As shown, the pre-processing, localization, and classification components have "Basic Functionality" implemented, though they have a long way to go until they have "Decent Functionality".
A current goal is the try to get all the components "Basic Functionality" implemented (even if they don't exactly "work") so we can do ceiling analysis. Ceiling analysis is where you see how much a part of the pipeline working 100% would increase overall performance of the system. To do this you iteratively hand feed correct results from each part to the rest of the parts and see how much the entire systems accuracy increases.
This is a method to help prioritize which parts of the pipeline to work on. More information can be found on the Stanford ml-class preview videos site in unit XVIII video "Ceiling Analysis: What Part of the Pipeline to Work on Next "
________________________________________________________________________
Classifier Improvement
Last post, we implemented cross validation and discovered that our classifier didn't perform better than chance. We were just using the raw pixels so this wasn't too surprising.
I found a file on the matlab fileexchange for HOG(Histogram of Oriented Gradients) features. Putting our toy dataset through it transformed a 1000x24963 matrix into a 1000x83 matrix.
This significantly increased the speed it takes to run from 8 minutes for one fold to 5 folds in 32 seconds.
The accuracy also went up, from 5.7 to 11.4! Which is *better* than chance. Confusion matrices have been generated for each fold and put in mistakes.mat in the repo, but we haven't interpreted them yet to figure out where to go from here.
The speed increase made it easier to explore the effect of different values of the lambda parameter for regularized logistic regression .
 |
| High values of lambda didn't help |
 |
| Exploring closer to .1 |
The values
[0 0.0900 0.1000 0.1100 0.1250 0.1500 0.2000 0.5000 0.7500 1.0000
10.0000 20.0000] were tested.
The corresponding mean cross-validation accuracies were:
[10.5000 11.2000 11.4000 10.6000 11.2000 10.6000 10.3000 9.9000 10.3000 10.7000 9.6000 9.7000]
In the end .1, the value that lambda had originally been set to, worked best.
Accuracies for different values of lambda at each fold can be found here.
___________________________________________________________________________
Localization Update
We still haven't gotten around to figuring out how use some of the infty dataset. We need to parse some comma-separated values in .txt files to do this. Some of the dataset, however, was given in raw images. I passed these through the localizer to see how it performed. It does well (which is to be expected, as these aren't noisy handwritten images), but still has the same problems with non-connected parts.
Misc
A few things accomplished since monday:
- Cleaned up some code (vectorizing stuff, replacing own implementation of a few things with matlab commands, added/deleted/fixed comments)
- Started research on making synthetic data to expand toy dataset
- Made folder of each toy dataset sample as an individual image, to help with feature extraction/testing
No comments:
Post a Comment