At WHOOP, some of our most demanding infrastructure challenges arise when we need to run a computation at population scale. Earlier this year, we needed to simulate one of our ML models across a large internal dataset to support a time-sensitive research workload. Each simulation used several weeks of historical inputs, and the total came out to roughly 15.8 million inference tasks.
The tool we had on hand was built for a much smaller job a couple of years ago. At its current per-task latency, completing the new run would have taken more than two months, far too slow to get researchers the data they were waiting on. We brought that down to six days. This post covers the changes that made this possible.