After opening the beta to ~100 athletes we started to see some issues with the bike classifier failing to handle certain situations. For example, many of the higher end athletes have several bikes equipped with power with surprisingly similar riding characteristics.
We analyzed the situation and the worst performing athlete (assuming Strava activities are always coded properly WRT gear) was ~65% accuracy which we felt was unacceptable and worth a push to improve! It’s too bad Strava doesn’t make the raw device file available through their API as we’d be 100% correct in every case if we could train off the sensor identifiers!
We decided to give another type of classifier a go in a poor man’s ensemble learning model where we execute multiple classifiers and only message the user if they all fail to match and the user hasn’t changed the bike on the activity.
We tested it on the 5 worst performing bike prediction athletes and it made a noticeable difference. Time will tell if it improves future predictions. We still have a few other tricks up our sleeves should this be proved to be insufficient.