Two days ago, at the Poster session of the final Machine Learning projects, a classmate of us present nothing but a work repeat a state-of-the-art paper, which claims that they can achieve 90% accuracy in gender prediction on blog data. The classmate reported that he tried almost his best using the method proposed by this paper and the best he can get in a well-known dataset is a little bit more than 80%. Considering the difficulty of gender prediction, he did not doubt but was very curious about how the published paper can get more than 90% accuracy.
I talked with him at his poster later. I think most of us have the same concern. He said, if a new researcher come into a specific area and find that previously someone have already get such a high accuracy, he probably won't dig into this problem later.
A lot of interesting work are presented in the final poster session. Wiki documents recommendation system, predicting location based on your friends location on social networks and so on. And our group got an frog which can dance and sing as the best poster prize ^^
那个维基百科推荐的评测方式很有问题，他实际是在预测用户行为，但用户自己还不知道自己的行为么？这么做推荐出来的东西只能是没有推荐用户也会去写的东西。
ReplyDeletecongrats, good job!
ReplyDeleteyes, you're putting up an interesting question. when i implemented a CVPR paper w/ "Best Demo Award" for my bachelor thesis project, the performance was not even comparable to what they claimed. it sure reflected my poor programming, but i think there's more to it. people do tend to report good news. for a researcher, he should be able to raise questions on suspicious accuracies, but that's only when he has done something and have got experienced and insightful enough.
also, i think very often good accuracy is only a result of overfitting.