Creating a Dataset Sucks. Here's What I've Learned to Make it a Little Bit Easier

Creating datasets is hard, even for relatively simple tasks like document classification/sentiment analysis. Even a relatively high agreement score can allow for 10% of your data to be wrong. Hiring a set of annotators is hard too, but once you figure out the hiring, here's some things that I've learned managing annotators. Getting a minimal set of data annotated, and trying to build models on it will help target the specific kinds of annotated data that you need, or even bad classifiers can help filter through a lot of data.