7 ms·

Semi-related: I was thinking about getting one of the newer iWatches, the one with the heart-rate monitoring and all that jazz. To me, those features are really

Semi-related: I was thinking about getting one of the newer iWatches, the one with the heart-rate monitoring and all that jazz. To me, those features are really cool and future-y. So, I waltz on down to the store and try one on. It reads my heart-rate and it seems to be fairly accurate. Ok, well, let's test it a bit. I start running in place. The heart rate goes up. I then just stand there, after a bit, the heart-rate goes down. Nice, the thing actually works! Well, they had a few of the iWatches sitting there, without anyone else taking a poke at them. So, I strap all the free ones on the arm. Well, they all gave different readings. I repeated the running in place test. Some went up and down much more than the others; maybe 10 bpm more. I tried re-arranging them on my arm and doing the running in place test again, you know, maybe it only works on the wrist since the blood is closer to the surface there? Nope, the readings were all over the place still. It was not super scientific of me. I mean, I was some strange person running in place in a store with a bunch of iWatches strapped to my arm. Maybe the watches have some sort of learning algo in them that was getting messed up with all the folks trying them on. Maybe I was wearing them 'wrong'. Maybe I should have waited 10 minutes between jogging sessions. I don't know. But, to me, these things have a long way to go. A lot of the bio-tech and bioengineering out there seems to be a bit of snake-oil right now. I really do want it all to work, I think that would be a great boon to us all [0]. If something like the iWatch can't give repeatable readings in nearly any way, then it's all just a gimmick and not useful. [0] I mean, can you imagine if Theranos' tech actually did work?! That would change medicine for the better in incalculable ways. I can see why that company's plan was so intoxicating.

falcrist 8y ago
That's surprising, since the heart rate monitors on many phones are quite accurate. Many will actually give an ECG-like trace indicating brief changes in blood pressure. I don't see any good reason they shouldn't be extremely accurate.
- pc86 8y ago
  Well unless you're wearing a cuff you're not getting any blood pressure readings on your phone, and any "ECG-like trace" is a doodle at best as there's no electrical monitoring. Even chest straps that measure electrical output don't give you an ECG display.
  - falcrist 8y ago
    I'm not suggesting that absolute blood pressure is being measured by a phone PPG, only relative changes. Far from being just a doodle, the PPG readout ("ECG-like trace") comes from changes in skin reflectivity that indicate relative changes in blood pressure. [1] Obviously this isn't as good as an actual ECG (and can be disrupted by movement), but it's far more information than is needed to accurately track heart rate. These devices can be as accurate as a pulse oximeter [2], though they don't account for the movement of the user. [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4822420/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4822420/ [2] https://www.ncbi.nlm.nih.gov/pubmed/28464700 https://www.ncbi.nlm.nih.gov/pubmed/28464700
- deleted 8y ago
  [deleted]
sjroot 8y ago
This is a very interesting story and I imagine there is some amount of ML used to gradually improve the accuracy of results for a specific user. Unrelated: It is called the Apple Watch. :)
- rando444 8y ago
  It's not an interesting story, and his method of testing was severely flawed. There are tons of people that test these devices against chest straps online, and almost all of these bands give results that are surprisingly accurate.. or at very least about what you'd expect out of a wrist based monitor.
- nradov 8y ago
  There's no ML. It's a direct measurement. ML only works for these types of measurements when you have some kind of reliable source of truth or fitness function to compare against.
reaperducer 8y ago
So, I strap all the free ones on the arm I can't say that I know exactly how these things work, but I wouldn't be surprised if they were built to work with the blood vessels and skin thickness of a wrist, where watches are worn, rather than the skin thickness+hair+different vessel locations of a forearm. Different parts of the body are used for different things. It's why blood oxygen monitors that work on fingers don't work on toes.
- Balgair 8y ago
  Yeah, I tried to move them about over my arm, thinking exactly the same thing as you just did. It gave me all kinds of results. Another commentor points out that the data is 'demo only' data on the in-store watches. The reason for demo only data is that the data might be considered 'medical' information and, I assume, therefore covered under HIPAA. Exposing that to just about anyone that can come in the store and see the record of the bpm is, rightly, very illegal and unethical.
ip26 8y ago
Some of those things stop working correctly when your skin gets damp. Think about that for a second. An iWatch is not the pinnacle of wrist biometrics, however. I'd be a lot more concerned if a high end Garmin got it wrong, that's their whole business.
- nradov 8y ago
  Garmin and Apple wearable devices use basically the same optical heart rate sensors. They don't make their own. Every company in the industry buys from the same couple of suppliers which have developed the technology.
- pc86 8y ago
  Part of how you're supposed to use the dedicated higher-end models (I have a Garmin Forerunner 645) is that it's supposed to be pretty tight against your wrist. Even if you "wear your watch tight" it's probably not as tight as you're supposed to wear it for fitness purposes. If at any point you can't feel it or you forget you're wearing it, it's definitely not tight enough. This helps limit optical issues with movement but also largely eliminates the impact of sweat on the reading. Still not nearly as accurate as the chest-based models as mentioned elsewhere, but for 95% of people it's plenty.
nradov 8y ago
Wrist worn optical heart rate monitors are actually very accurate, under normal circumstances when worn properly. It's easy to verify accuracy yourself by just counting your own pulse rate with your finger. Try it. Wrist devices will give inaccurate pulse counts if the band is too loose, skin is dirty/wet, or your arm is moving around quickly. If you need accurate numbers in all circumstances then a chest strap which detects electrical signals is a much better option. I usually wear a Garmin HRM-Tri strap while exercising and it transmits a wireless signal to my fitness tracker.
- the_duke 8y ago
  Manually (as in manus, the hand) measuring your pulse while not at rest (sitting down, in a calm environment) is so error prone, it's almost useless. The only proper comparison would be with a chest strep, which are much more accurate. Also, it really differs per person and how you wear the watch. Just wearing it a bit differently can change the results significantly. Sweat also gets them messed up easily. All reasons why, if you want accurate data for workouts, you should always use a chest strep. Even if they are much more awkward.
  - sfilargi 8y ago
    Anecdotal, but I got an Apple Watch because I never liked the squeezing feeling of the chest strap of my Suunto. I put them to test for the first couple weeks and both at rest and while running the difference in the HR reported by both was less than 1bps.
deaps 8y ago
So I ride bikes and always monitor my heart rate while working out. I've always used a chest-based heart rate sensor until I started trying wrist-based about 4-5 years ago. I've used countless brands and models (ranging in price from about $49.99 up to about $399) on my wrist - and literally none are as accurate as the chest-based strap. The chest strap responds immediately to increases and decreases in heart rate. Whereas the wrist sensors always seem to lag, sometimes by up to a minute or so. The wrist-based sensors always seem to differ depending on where on the wrist they are worn - high, low, inside of the wrist, outside of the wrist, etc - by maybe 5 bpm or so. In either event, if you really want to test the accuracy of the wrist-based sensor, you should really compare it to the output of a chest-based strap. In fact, the results will largely be personal - because the electric signals may differ per person greatly in different areas of the arm. Needless to say, my apple watch is worn daily, but when riding, I have gone back to a chest-based strap. That all being said, there is a guy that reviews these gadgets and will often wear 2-4 sensors while working out, and post an overlay of the results from his workouts. The baseline should generally always assume that the chest-based straps are accurate. His site is called DCRainmaker, if anyone wanted to look up his comparisons/reviews.
- jnwatson 8y ago
  The wrist lag is the biggest problem I have with my Fitbit wrist band. It is OK for looking backwards of how my overall work out went, but it is useless for trying to hit a target heart rate.
- DoctorOetker 8y ago
  I wonder if the difference in results is partially due to a difference in measurement strategy: a naive way is to wait one minute while counting the pulses which will necessarily be slower in response (it could still update the screen at a faster rate of once per second by having 60 different integers but each integer gives the count displayed on the screen when it's last second has passed, but that would still have a slow response in the sense that given a hypothetical heart rate change from a fixed constant initial rate to a new fixed higher rate, like when you got scared by say unexpected fireworks, it would still take a minute for the final rate to show up). A faster response could be got by measuring the time between heartbeats (probably a low but even number of beats).
odyssey7 8y ago
When I tried on an Apple Watch at an Apple Store, I noticed that the heart rate it displayed was definitely different from mine. I mentioned it to the employee who was manning that table, and she told me the demo watches had some pre-programmed data for demo purposes. I was puzzled by this since heart rate monitoring is a key feature users care about. It occurred to me that heart rate data of everyone who has tried on the watch might count as the kind of medical information you shouldn’t publish on demo watches. Imagine ridiculous stories like “CEO of X’s heart rate was 130bpm when he tried on the new Apple Watch, says journalist who tried it on right after.”
- Balgair 8y ago
  Well, I'll be damned! Thanks for the explanation, I really appreciate it. I'm still wanting to buy one, but now I don't know what to think. I wonder how I am supposed to try one out before I buy one and to make sure it works.
  - saganus 8y ago
    I guess you would have to bring a proper heart rate monitor and compare the two readings. Although people might give you strange looks :)
  - ntlk 8y ago
    You can buy any Apple device, use if for two weeks and return if for a full refund if you don’t like it.
  - ralfd 8y ago
    In a study two years ago Apple Watch was the most accurate tested device with a median error of 2%. https://www.macrumors.com/2017/05/24/apple-watch-heart-rate-most-accurate-fitness-study/ https://www.macrumors.com/2017/05/24/apple-watch-heart-rate-...
- darpa_escapee 8y ago
  > It occurred to me that heart rate data of everyone who has tried on the watch might count as the kind of medical information you shouldn’t publish on demo watches. That's an overly generous explanation. Fitbit-like devices are notoriously inaccurate[1], and the simplest explanation seems to be that their inaccuracy is unintentional. [1] https://www.independent.co.uk/life-style/health-and-families/health-news/heart-rate-monitor-unreliable-fitbit-garmin-health-science-technology-a8413196.html https://www.independent.co.uk/life-style/health-and-families...
com2kid 8y ago
Disclaimed: I used to work in this field, but not on the Apple Watch. The wearing a bunch of them at the same time is going to be wrong, yeah. Wrist mounted fitness devices are calibrated to read on the top side of the wrist. Even reading on the bottom side of the wrist would be a bit different, way up the arm? Very different. It is possible to make accurate devices that read there, they just need to be constructed to do such! Wearables are also meant to give an accurate rating over the long term. If you go running for 30 minutes, are the overall calorie burn, average HR, max HR, and min HR, all correct? Tracking in perfect sync with the heart is nice, and of course everyone tries to do that, but if the number is off for a second here or there, well so long as the overall result is the same the usefulness hasn't really changed. Finally, due to the way optical HR works, random unexpected, movements are really hard to deal with. It can take some time for an HR sensor to lock on to the heart rate. This isn't too long (measured in seconds, and each generation of devices is getting better and better), but it relies on regular patterned movement, such as arms pumping while running or walking, the regular motion of a bench press, etc. Stopping and checking the HR value every few seconds to see if it is correct is literally the worst case scenario for a lot of wrist mounted sensors! Ground truth is strapping a heart rate strap to your chest, putting on the wearable, going for a run as you normally would, and comparing the two at the end. Multiple sources have done this, and found the Apple Watch to be best in class when compared to directly measuring HR. tl;dr: Stopping and checking wrist mounted devices can confuse them, Apple Watch, and everyone else in this field who is serious, are tested against HR straps and they come within spitting distance of accuracy.
ALittleLight 8y ago
The research I've read [1] says that heart rate monitor in general and Apple Watch in particular should be pretty accurate. If you have darker skin or wider wrists that accuracy may be diminished. [1] https://www.mdpi.com/2075-4426/7/2/3 https://www.mdpi.com/2075-4426/7/2/3
- Balgair 8y ago
  Thanks for the link!
deleted 8y ago
[deleted]
gist 8y ago
I have the first apple watch. (I bought the newest but haven't used it yet still boxed). What I found was that the heart rate correlated well with taking it manually and/or a finger heart monitor (pulse ox cheap from Amazon). But with both devices you could actually tell when it was clearly wrong independent of comparison. The point is you more or less (at least I found) could figure out when it is right and wrong just based on past use and what you would expect (feeling your pulse). So if it was off it was grossly off and obvious. That is my experience with the watch that I have had for years. Either 'correct' or 'vastly off and easy to tell that is the case'.

falcrist 8y ago

That's surprising, since the heart rate monitors on many phones are quite accurate. Many will actually give an ECG-like trace indicating brief changes in blood pressure. I don't see any good reason they shouldn't be extremely accurate.

pc86 8y ago
Well unless you're wearing a cuff you're not getting any blood pressure readings on your phone, and any "ECG-like trace" is a doodle at best as there's no electrical monitoring. Even chest straps that measure electrical output don't give you an ECG display.
- falcrist 8y ago
  I'm not suggesting that absolute blood pressure is being measured by a phone PPG, only relative changes. Far from being just a doodle, the PPG readout ("ECG-like trace") comes from changes in skin reflectivity that indicate relative changes in blood pressure. [1] Obviously this isn't as good as an actual ECG (and can be disrupted by movement), but it's far more information than is needed to accurately track heart rate. These devices can be as accurate as a pulse oximeter [2], though they don't account for the movement of the user. [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4822420/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4822420/ [2] https://www.ncbi.nlm.nih.gov/pubmed/28464700 https://www.ncbi.nlm.nih.gov/pubmed/28464700
deleted 8y ago
[deleted]

sjroot 8y ago

This is a very interesting story and I imagine there is some amount of ML used to gradually improve the accuracy of results for a specific user. Unrelated: It is called the Apple Watch. :)

rando444 8y ago
It's not an interesting story, and his method of testing was severely flawed. There are tons of people that test these devices against chest straps online, and almost all of these bands give results that are surprisingly accurate.. or at very least about what you'd expect out of a wrist based monitor.
nradov 8y ago
There's no ML. It's a direct measurement. ML only works for these types of measurements when you have some kind of reliable source of truth or fitness function to compare against.

reaperducer 8y ago

So, I strap all the free ones on the arm I can't say that I know exactly how these things work, but I wouldn't be surprised if they were built to work with the blood vessels and skin thickness of a wrist, where watches are worn, rather than the skin thickness+hair+different vessel locations of a forearm. Different parts of the body are used for different things. It's why blood oxygen monitors that work on fingers don't work on toes.

Balgair 8y ago
Yeah, I tried to move them about over my arm, thinking exactly the same thing as you just did. It gave me all kinds of results. Another commentor points out that the data is 'demo only' data on the in-store watches. The reason for demo only data is that the data might be considered 'medical' information and, I assume, therefore covered under HIPAA. Exposing that to just about anyone that can come in the store and see the record of the bpm is, rightly, very illegal and unethical.

ip26 8y ago

Some of those things stop working correctly when your skin gets damp. Think about that for a second. An iWatch is not the pinnacle of wrist biometrics, however. I'd be a lot more concerned if a high end Garmin got it wrong, that's their whole business.

nradov 8y ago
Garmin and Apple wearable devices use basically the same optical heart rate sensors. They don't make their own. Every company in the industry buys from the same couple of suppliers which have developed the technology.
pc86 8y ago
Part of how you're supposed to use the dedicated higher-end models (I have a Garmin Forerunner 645) is that it's supposed to be pretty tight against your wrist. Even if you "wear your watch tight" it's probably not as tight as you're supposed to wear it for fitness purposes. If at any point you can't feel it or you forget you're wearing it, it's definitely not tight enough. This helps limit optical issues with movement but also largely eliminates the impact of sweat on the reading. Still not nearly as accurate as the chest-based models as mentioned elsewhere, but for 95% of people it's plenty.

nradov 8y ago

Wrist worn optical heart rate monitors are actually very accurate, under normal circumstances when worn properly. It's easy to verify accuracy yourself by just counting your own pulse rate with your finger. Try it. Wrist devices will give inaccurate pulse counts if the band is too loose, skin is dirty/wet, or your arm is moving around quickly. If you need accurate numbers in all circumstances then a chest strap which detects electrical signals is a much better option. I usually wear a Garmin HRM-Tri strap while exercising and it transmits a wireless signal to my fitness tracker.

the_duke 8y ago
Manually (as in manus, the hand) measuring your pulse while not at rest (sitting down, in a calm environment) is so error prone, it's almost useless. The only proper comparison would be with a chest strep, which are much more accurate. Also, it really differs per person and how you wear the watch. Just wearing it a bit differently can change the results significantly. Sweat also gets them messed up easily. All reasons why, if you want accurate data for workouts, you should always use a chest strep. Even if they are much more awkward.
- sfilargi 8y ago
  Anecdotal, but I got an Apple Watch because I never liked the squeezing feeling of the chest strap of my Suunto. I put them to test for the first couple weeks and both at rest and while running the difference in the HR reported by both was less than 1bps.

deaps 8y ago

So I ride bikes and always monitor my heart rate while working out. I've always used a chest-based heart rate sensor until I started trying wrist-based about 4-5 years ago. I've used countless brands and models (ranging in price from about $49.99 up to about $399) on my wrist - and literally none are as accurate as the chest-based strap. The chest strap responds immediately to increases and decreases in heart rate. Whereas the wrist sensors always seem to lag, sometimes by up to a minute or so. The wrist-based sensors always seem to differ depending on where on the wrist they are worn - high, low, inside of the wrist, outside of the wrist, etc - by maybe 5 bpm or so. In either event, if you really want to test the accuracy of the wrist-based sensor, you should really compare it to the output of a chest-based strap. In fact, the results will largely be personal - because the electric signals may differ per person greatly in different areas of the arm. Needless to say, my apple watch is worn daily, but when riding, I have gone back to a chest-based strap. That all being said, there is a guy that reviews these gadgets and will often wear 2-4 sensors while working out, and post an overlay of the results from his workouts. The baseline should generally always assume that the chest-based straps are accurate. His site is called DCRainmaker, if anyone wanted to look up his comparisons/reviews.

jnwatson 8y ago
The wrist lag is the biggest problem I have with my Fitbit wrist band. It is OK for looking backwards of how my overall work out went, but it is useless for trying to hit a target heart rate.
DoctorOetker 8y ago
I wonder if the difference in results is partially due to a difference in measurement strategy: a naive way is to wait one minute while counting the pulses which will necessarily be slower in response (it could still update the screen at a faster rate of once per second by having 60 different integers but each integer gives the count displayed on the screen when it's last second has passed, but that would still have a slow response in the sense that given a hypothetical heart rate change from a fixed constant initial rate to a new fixed higher rate, like when you got scared by say unexpected fireworks, it would still take a minute for the final rate to show up). A faster response could be got by measuring the time between heartbeats (probably a low but even number of beats).

odyssey7 8y ago

When I tried on an Apple Watch at an Apple Store, I noticed that the heart rate it displayed was definitely different from mine. I mentioned it to the employee who was manning that table, and she told me the demo watches had some pre-programmed data for demo purposes. I was puzzled by this since heart rate monitoring is a key feature users care about. It occurred to me that heart rate data of everyone who has tried on the watch might count as the kind of medical information you shouldn’t publish on demo watches. Imagine ridiculous stories like “CEO of X’s heart rate was 130bpm when he tried on the new Apple Watch, says journalist who tried it on right after.”

Balgair 8y ago
Well, I'll be damned! Thanks for the explanation, I really appreciate it. I'm still wanting to buy one, but now I don't know what to think. I wonder how I am supposed to try one out before I buy one and to make sure it works.
- saganus 8y ago
  I guess you would have to bring a proper heart rate monitor and compare the two readings. Although people might give you strange looks :)
- ntlk 8y ago
  You can buy any Apple device, use if for two weeks and return if for a full refund if you don’t like it.
- ralfd 8y ago
  In a study two years ago Apple Watch was the most accurate tested device with a median error of 2%. https://www.macrumors.com/2017/05/24/apple-watch-heart-rate-most-accurate-fitness-study/ https://www.macrumors.com/2017/05/24/apple-watch-heart-rate-...
darpa_escapee 8y ago
> It occurred to me that heart rate data of everyone who has tried on the watch might count as the kind of medical information you shouldn’t publish on demo watches. That's an overly generous explanation. Fitbit-like devices are notoriously inaccurate[1], and the simplest explanation seems to be that their inaccuracy is unintentional. [1] https://www.independent.co.uk/life-style/health-and-families/health-news/heart-rate-monitor-unreliable-fitbit-garmin-health-science-technology-a8413196.html https://www.independent.co.uk/life-style/health-and-families...

com2kid 8y ago

Disclaimed: I used to work in this field, but not on the Apple Watch. The wearing a bunch of them at the same time is going to be wrong, yeah. Wrist mounted fitness devices are calibrated to read on the top side of the wrist. Even reading on the bottom side of the wrist would be a bit different, way up the arm? Very different. It is possible to make accurate devices that read there, they just need to be constructed to do such! Wearables are also meant to give an accurate rating over the long term. If you go running for 30 minutes, are the overall calorie burn, average HR, max HR, and min HR, all correct? Tracking in perfect sync with the heart is nice, and of course everyone tries to do that, but if the number is off for a second here or there, well so long as the overall result is the same the usefulness hasn't really changed. Finally, due to the way optical HR works, random unexpected, movements are really hard to deal with. It can take some time for an HR sensor to lock on to the heart rate. This isn't too long (measured in seconds, and each generation of devices is getting better and better), but it relies on regular patterned movement, such as arms pumping while running or walking, the regular motion of a bench press, etc. Stopping and checking the HR value every few seconds to see if it is correct is literally the worst case scenario for a lot of wrist mounted sensors! Ground truth is strapping a heart rate strap to your chest, putting on the wearable, going for a run as you normally would, and comparing the two at the end. Multiple sources have done this, and found the Apple Watch to be best in class when compared to directly measuring HR. tl;dr: Stopping and checking wrist mounted devices can confuse them, Apple Watch, and everyone else in this field who is serious, are tested against HR straps and they come within spitting distance of accuracy.

ALittleLight 8y ago

The research I've read [1] says that heart rate monitor in general and Apple Watch in particular should be pretty accurate. If you have darker skin or wider wrists that accuracy may be diminished. [1] https://www.mdpi.com/2075-4426/7/2/3 https://www.mdpi.com/2075-4426/7/2/3

Balgair 8y ago
Thanks for the link!

deleted 8y ago

[deleted]

gist 8y ago

I have the first apple watch. (I bought the newest but haven't used it yet still boxed). What I found was that the heart rate correlated well with taking it manually and/or a finger heart monitor (pulse ox cheap from Amazon). But with both devices you could actually tell when it was clearly wrong independent of comparison. The point is you more or less (at least I found) could figure out when it is right and wrong just based on past use and what you would expect (feeling your pulse). So if it was off it was grossly off and obvious. That is my experience with the watch that I have had for years. Either 'correct' or 'vastly off and easy to tell that is the case'.