If you think you can chow down guilt-free on a 540-calorie Big Mac because the smart watch on your wrist says you just burned that many calories at the gym, guess again.
A new study found that while devices such as the Fitbit Surge and the Apple Watch provided accurate heart rate measurements, they were way off as far as how many calories people burned when exercising. Sometimes the results were too high, sometimes they were too low, but they were never just right, or even close to it.
TheFitbit Surge had the lowest error margin for energy expenditure of the seven devices tested, although it was still off by an average of 27%. The PulseOn had the highest error margin for energy expenditure, off by an average of 93%.
On the other hand, coauthor Dr. Euan Ashley told me, “I was surprised at how accurate they were for heart rate. It made me feel all warm inside. As a physician, I would really love it if these devices were100% correct.”
Ashley, a professor of cardiovascular medicine, of genetics and of biomedical data science at the Stanford University School of Medicine, said he and his collaborators were curious about the accuracy of popular wrist-worn fitness trackers, used by millions of people, because consumers don’t have access to the results of tests conducted by the manufacturers.
Stanford University School of Medicine researcher Euan Ashley and his team in a testing lab in Palo... [+] Alto, Calif., on Friday, May 19. (Photo by Paul Sakuma/Stanford University School of Medicine)
(Photo by Paul Sakuma Photography)The researchers enlisted the help of 60 volunteers, 31 women and 29 men, who were at least 18 years old. The participants were a diverse group, a deliberate move on the researchers’ part so they could examine whether factors such as age, fitness level, weight or skin color might affect the devices’ accuracy. Besides the Fitbit Surge and the PulseOn, the volunteers tested the Apple Watch, the Samsung Gear S2, the Basis Peak (all of which were recalled last August, after the study was conducted, because they overheated), Microsoft Band and Mio Alpha 2 while walking or running on treadmills or using stationary bicycles. The researchers had planned to test an eighth fitness tracker, but multiple ePulse 2 devices had technical problems and were excluded.
While the study subjects exercised, the researchers measured their heart rate with an electrocardiograph and estimated their metabolic rate and energy expenditure based on the oxygen and carbon dioxide in their breath. They then compared those findings with the numbers generated by their fitness trackers.
“Is it possible that some people are just very hard to measure?” wondered Ashley and his collaborators, whose research was published online Wednesday in the Journal of Personalized Medicine. But no study participants ended up with consistently inaccurate measurements across all activities, he said, although device error on average was higher for men, greater body mass index (BMI), darker skin tone and walking (cycling had the lowest error rate of all the types of exercise studied).
So why was the calorie-counting so inaccurate? No one besides the manufacturers can know for sure, because the algorithms they developed to measures things like energy expenditure are proprietary. “We think each of the devices would use a different equation,” Ashley said. Considering that the devices measured heart rate accurately, he speculated that the companies’ algorithms for energy expenditure might not take heart rate into account.
I sought a comment from Fitbit, which held the largest share of the global wearables market until Apple overtook it in the first quarter of this year, according toresearch by Strategy Analytics.
"Fitbit trackers show an estimated total number of calories burned based on users' BMR (basal metabolic rate) and activity energy expenditure (AEE)," the company told me in a prepared statement. "Fitbit uses a scientifically validated estimate of BMR based on height, weight, age and gender information that users provide when setting up their Fitbit account."
Fitbit wrist-worn devices with continuous heart rate tracking do use calculated heart rate to help estimate AEE, according to the statement. "This is particularly helpful for non-step-based workouts like cycling, elliptical or group exercise classes, as well as more everyday activities," the company said, adding that information about calorie burn estimates can be found on its website.
I always thought that you could figure out how many calories you burn running or swimming or whatever based on how much you weigh and, maybe, your sex. Indeed, "your personal information, such as your height, weight, gender and age, is one of the things your Apple Watch uses to calculate how many calories you burn and more," according to that device's website.
But Ashley set me straight. Sure, you’re going to expend more energy if you’re 50 pounds overweight than if you’re 5 pounds overweight, but it’s more complicated than that, which might help explain why the devices had margins of error that were greater than 20% when measuring energy expenditure.
“Some people are extremely efficient in movement, some people are not,” Ashley noted. “Some are smooth and flowing, some are all over the place.” Factors such as coordination, flexibility and genetics play a role, but people can train to become more efficient in how they move.
Ashley and his coauthors are now testing the accuracy of the fitness trackers outside their research lab. Volunteers are using portable machines to measure the oxygen and carbon dioxide in their breath while they go about their normal activities, such as sitting at a desk, taking a stroll or riding a mountain bike. The researchers have also set up a website that shows their data and invites others to upload data related to performance of the devices.
For now, “it’s worth taking these estimates with more than a pinch of salt,” Ashley said of the fitness trackers’ measurements of energy expenditure.
But maybe don’t take them with a side of fries.