Recent Changes - Search:

PmWiki

pmwiki.org

edit SideBar

ChatfieldObjects

The Devil is in the Details: An Evaluation of Recent Feature Encoding Methods.

K. Chatfield et.al. BMVC 2011, Dundee.

Inventing genuinely novel ideas in computer vision is a difficult and time consuming process. The recent trend for the rapid generation of work for conference publication involves making (relatively minor) modifications to code downloaded from the internet and running it on standard data sets in order to obtain a performance figure. We might expect that obtaining high scores can only be achieved by making use of new and meaningful insights, and not simply by adjusting exisitng control parameters. High performance scores are likely to get accepted at conferences and may become regarded as `state-of-the-art'.

This paper takes several recent modification to algorithms and performs an independent evaluation of performance, on an equal footing. What is found is that representative performance figures are often very different to those in the original publication, wiping out any claimed advantage. Therefore, the factors which generated good performance in the original paper may well have been due to parameter `tuning' or other unspecified methods (the details) which have no real bearing on the published ideas.

There are several reasons why it is difficult to run fair competitive evaluations, well known experimenter effects being only one example. Another possibility is the use of test data during algorithm development (strictly training and test data should be kept separate in order to avoid over-training). In the absence of independent testing it may well be the case that the first time an algorithmic evaluation can really be trusted is the first time it is done by someone other than the original author and on a different test set. Even then, the test dataset is too small to quantify meaningful differences in performance. If this is true (and it seems it is), it will require us to seriously reconsider our concept of `state-of-the-art'.

NAT 12/7/2013

Edit - History - Print - Recent Changes - Search
Page last modified on July 12, 2013, at 12:00 PM