Venk Murthy Profile picture
May 11, 2018 51 tweets 14 min read Twitter logo Read on Twitter
More hype around AI, mHealth, wearables. @AliveCor devices can use #AI to find LQTS:

prnewswire.com/news-releases/…
From the press release:

AUC 0.83, specificity 81%, sensitivity 73%, accuracy 79%

Prevalence of LQTS 160,000 out of US population of 325.7 million (0.049%)
Do we need to do a tweetatorial about how this is grossly insufficient accuracy and will almost certainly lead to more harm than good?
Let's say @GundotraVic is wildly successful and sells these things to 10% of the US population... how many is that?
If this was a random sampling of 10% of the US population, how many of the 160,000 LQTS patients would have these devices:
Now lets say its not a random sampling but enriched for LQTS at 2X the rate of the population. How many people have LQTS and one of these devices?
Okay. About ten votes so far. Hopefully more soon.

Let's take the most popular answers thus far.

~32 million people with devices
~32 thousand of them with LQTS
So what fraction of @AliveCor users have LQTS?
Given ~0.05% of the US population has LQTS and we are guessing 2X enrichment in the @AliveCor user group, thr above question should be easy!
Next questions are hard. A few abbreviations first.

TP = True Positive, meaning pt has LQTS and test is abnl

TN = True Negative, meaning pt does not have LQTS and test is normal
FP = False Positive, meaning pt does not have LQTS but test is abnl

FN = False Negative, meaning pt has LQTS but test is normal
please forgive me for the sins I am about to commit.
Sensitivity is the True Positive Rate or TP / All Disease Positives. All Disease Positives are TP + FN. So Sensitivity = TP / (TP + FN)

Specificity is the True Negative Rate or TN / All Disease Negatives. All Disease Negatives are TN + FP. So Specificity = TN / (TN + FP)
So which measure is more useful to calculate your false positive rate?
Good! Now which measure is most useful for calculating the false negative rate?
Once I get a few answers to these questions, we'll move on to the next part...
Okay, for the next part, let's round off a few of the numbers thus far:

Total with device: 30 million
Total with device and with LQTS: 30 thousand
Sensitivity: 80%
Specificity: 75%

I'm only doing this to make brain math easier.
What is the number with a device but WITHOUT LQTS?
While we wait for answers to the prior question to roll in, let's turn to sensitivity.

I do some algebra in the picture to show that 1-sensitivity = FN/(TP+FN) = FN/(all pts with disease)
Feel free to ignore the picture in the tweet above if algebra gives you the heebie jeebies, but I think math is fun. This is pretty simple algebra anyway so...
So we miss (1-0.75)=25% of all patients who really have LQTS.

That is 30,000 * 25% = what?
Now lets turn to the other side of the problem. The false positives. I again do some algebra in the attached picture to show that:

1 - Specificity = FP / (TN + FP) = FP / # people without disease
So if we want to know what fraction of people without LQTS will have a false positive result, we just take 1- Specificity = 1- 0.80 = 20%

Doesn't sound so bad... except when you think that this is out of 30 MILLION people without LQTS. so the number of false positives is
Don't read the next tweet until you answer all of the above questions.
By using this device on 30 million people, @GundotraVic is very happy that he is a business success. We find a total of 22,500 cases of LQTS (and miss 7,500). We also falsely label 6 MILLION people with LQTS.
So what fraction of people who get diagnosed with LQTS really have it? This is known as the Positive Predictive Value (PPV).

The formula for PPV = TP / (TP + FP).

In this case PPV = 22,500 / (22,500 + 6 million)

What is that?
Again, don't continue until you have answered all the questions above. No cheating!
The PPV is important because it helps you understand how many people could be harmed by this technology.

For LQTS, the PPV is <1%. So as it turns out 99% of people labeled by this SUPER INNOVATIVE technology will have nothing wrong with them.
They will have doctor bills, genetic testing, possibly changes to their medications with zero potential for benefit.
Is that a good deal?
Here is a very polite tweet from the CEO of @AliveCor
Turns out they aren't anywhere near shipping this as a product. Which is good. But they are using this to drum up interest and maybe to keep investors interested.
Problem is that for rare diseases, you need specificity of 99% or greater to not suffer from a flood of false positives.
It's really nice to say this is just promising, preliminary work but is there really hope to go from 0.80 specificity to 0.999 specificity just by tweaking the algorithm.

I suspect not.

Algorithms and sensors passively identifying rare health conditions is super hard.
An honest evaluation of this might have been "We tried but it turns out its nowhere near fit for widespread screening. We'll think further about other things we could do".

Instead we hear the word "Stunning" to describe the tech because of all the #AI & #MachineLearning hype.
Here is an article in the non-medical press which is takes part in the hype in @engadget:

engadget.com/2018/05/11/ai-…
Don't read @engadget ? How about @CNBC?

cnbc.com/2018/05/10/ali…
“There can be no better illustration of the importance of our AI to medical science than using it to detect that which is otherwise invisible,” said AliveCor CEO Vic Gundotra.

fiercebiotech.com/medtech/mayo-c…
Does this AI have "importance to medical science"?
From the @CNBC article above...
Is this true that LQTS is "often" the cause of sudden death during a game? I always thought about HCM, ARVC, other cardiomyopathy, coronary anomalies, etc were more common.
Most recent data from NCAA athletes says that LongQT causes about 1% of sudden cardiac death.

circ.ahajournals.org/content/132/1/…
LongQT is that tiny slice just before the 12 o'clock position...
Turns out the biggest cause of death of NCAA athletes over all is accidents.
So 99% wrong on positive calls, but let's still screen all high school athletes for something that is maybe responsible for 1% of a rare event (sudden cardiac death) that accounts for a small portion of all deaths of student athletes.

**But think of the children**
I'd rather not harm children by telling them they can't play sports when nothing is wrong with them...
The infamous @doctorwes (of MOC fame) has written a terrific and funny blog post about the harms of excess EKG screening in athletes:

drwes.blogspot.com/2012/05/dark-s…

H/t @EJSMD

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Venk Murthy

Venk Murthy Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @venkmurthy

Sep 18, 2018
Nice review of the media coverage and its problems related to @Apple watch with added clarity about data @US_FDA used to approve from @thejoyvictory at @HealthNewsRevu.

Must read.

Updated tweetatorial showing just how bad the PPV might below.

healthnewsreview.org/2018/09/what-d…
Are you interested in understanding just how bad @Apple watch's false positive rate might be?

1. Yes
2. No, it's AHA endorsed and cool
3. Other
Read 72 tweets
Aug 25, 2018
Lots of headlines @escardio re CT-FFR. 1 of most prominent is ADVANCE multicenter registry from @manesh_patelMD & colleagues using @HeartFlow. I'd like to dissect this a bit to see how excitement in a field combined with lots of industry funding can blind us to basic problems.
Team worked across 38 international sites over >2 years to collect data from N=5083 patients w/ “symptoms concerning for CAD”. They had primary endpoint of change in management plan from coronary CTA. That happened in 67% of patients. Most impressive.
Is this a valid endpoint? I contend that this is a measurement of physician psychology rather than of anything objective.
Read 50 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!

:(