We can do this by ensuring that both the “normal” and “anomalous” groups have a diversity of classes assigned to them. But we want to make our benchmarks datasets hard on our anomaly detectors. We could arbitrarily group classes into two sets and make those “normal” and “anomalous”. Things are a bit more complicated if our objective has multiple classes. One objective field value is chosen as “anomalous”, the other as “normal”. For binary classification tasks it’s even easier. But how should we split the dataset? If a dataset has a numeric objective field, we can simply test whether the objective value is above or below the median labelling one side as “anomalous” and the other “normal”. This is the fundamental operation in transforming a supervised learning dataset into a anomaly detection benchmark. With the click of a button, a single supervised “mother” dataset can generate hundreds of children datasets defined by these dimensions and ready for benchmarking.īefore we generate these children datasets, however, we need to label some of the rows as “anomalous” and some as “normal”. Using the flexibility of WhizzML (our LISP inspired DSL for Machine Learning workflows), BigML has replicated their process. The benchmark datasets will have points labeled as “anomalous” and “normal”, so the detectors can be scored against them (specifically AUC ). But it lets us push the anomaly detectors to their limits in a variety of ways, giving us a robust set of tests for comparison.
This blog post won’t dive into the details of those dimensions. They devised a way to sample real-world supervised learning datasets so that they produce benchmarks that vary along three dimensions: point difficulty, relative frequency, and semantic variation. This is what our Chief Scientist, Professor Tom Dietterich, and his research group at Oregon State University set to remedy with their paper ”Systematic Construction of Anomaly Detection Benchmarks from Real Data”. While there are many algorithms for detecting anomalies, there is a lack of publicly available anomaly detection benchmark datasets for comparing these techniques. Anomaly detectors are a useful tool for any machine learning practitioner, whether for data cleaning, fraud detection, or as early-warning for concept drift.
0 kommentar(er)
