Thread by @ProfDFrancis

Intention To Tweet
============

1. Why is intention-to-treat the ONLY right way to interpret trial results?
2. Why are these fine gentlemen wasting their prayers?

Tweetorial for cardiology fellows from ORBITA-HQ
#cardiology #meded #foamed

Thank you to Ali Jazayeri for suggesting this subject on the highly secret DM contact address (@ProfDFrancis).

At first I hesitated over whether this was truly needed amongst such an elite body of scienntific minds as we have reading this thread.

But Pascal Meier insisted that it was.
He said he was watching a whole session dedicated to not understanding this at a conference. Sent a pic!

Francis Institute for Developing Duds into Luscious Earnings (FIDDLE) has of course moved into this market, once it learned that the gullible are all around us.

We have noted the immortal words of Prof T Luscher, and have followed his suggestion for focus.

FIDDLE-1, our paradigm-shifting new RCT, will demonstrate the efficacy of bone-marrow-derived stem cells in cardiogenic shock due to STEMI.

Conducted in collaboration with Zygotron Metadynamics [the only stem cell company you can trust] FIDDLE have set up their state-of-the-art stem cell elucidation lab on the 32nd floor of the Auxilliary building of our hospital:

Unfortunately the elevator is not working but the patient can easily walk up the stairs. Sicker patients may want to take the journey in a few stages, with refreshments (and CPR) along the way.

Reaching the stem cell lab, it's really worth it - look at all this sciency stuff!

And here is the primary result which will be presented at ACC 2018.

Highly statistically significant, p<0.0001

2000 patients randomized, very clear result.

The all cause mortality rates are 61% in the control arm and 5% in the active arm.

What is the ABSOLUTE risk reduction?

(In absolute percentage points)

Remember the Number Needed to Treat? So much vaunted by people who like to confuse you with reciprocals?

What is the NNT? [Can do without a calculator]

Hint: it is 1/ARR. If expressing ARR in absolute percentage points, it is:
100 / (ARR in absolute percentage points)

People are doing REALLY well on this, I am so pleased.

How about as a RELATIVE risk reduction?

Again do this without a calculator. Get a very rough idea and pick the one that is closest to it. I am telling you in advance that ONE of these is correct.

So, in conclusion, what is this therapy?

Anyone who answered "All Of The Above" is immediately awarded "Speaker's Bureau" status by FIDDLE, LLC.

Please collect your checks in the foyer.

Our CEO uses your support to get it through the FDA.

But the FDA are a bunch of hard-hearted bastards. All they do is ask questions. They are not real pioneers, like us at FIDDLE.

They asked, "Did you really make your patients with cardiogenic shock walk up all those stairs?"

"No, of course not!" said our CEO soothingly, and edged out of the room before more questions could follow.

But his ploy didn't work. In due course an email arrived from FDA:
"What happened to the patients who didn't make it up the stairs to get the stem cells?"

Our lawyers replied,
"They were protocol violators and were excluded from the study."

How did the FDA reply?

Ha ha, very funny the 5 people that are trolling us by saying the FDA said it is OK to delete people at will. This would never get past them.

No, the FDA would want everyone included.

So our weary statistician spends another all-nighter re-cooking the books.

Dawn breaks over the FIDDLE HQ office and a tsunami of text messages and emails spreads around the Board of Directors.

Each wants to be the one to give out the good news!

Here it is, p still less than 0.001! Nobody had to be deleted!

In the new "Everyone included" analysis, the event rates are:

71% in the control arm
5% in the active arm

What is the absolute risk reduction? (in absolute percentage points)

What is the Number Needed to Treat?

One of these numbers is correct. In your head, you should be able to see that ONLY ONE is plausible: pick that one.

And the relative risk reduction, from 71% to 5%?

If anyone has difficulty doing these by mental arithmetic without a calculator, please DM me and I will explain on the main thread.

There are very simple tricks to do it but I am assuming people know them if they aren't asking.

FDA are not fooled by this stuff, of course.

These people are very, very good at math and logic.

All except the boss, who is a bit zany:

External Tweet loading...
If nothing shows, it may have been deleted
by @ProfDFrancis view original on Twitter

[Oddly in the thread here, the correct answer, B, was only picked by one out of 7.]

The FDA boffins have both of your result graphs.

What do you think they immediately recognised was strange?

Which arm did they expect to have a change in mortality when you NO LONGER DELETED the "protocol violators" in the active arm?

How could this have happened?

What has the FIDDLE-1 steering committee NOW done with the protocol violators?

Randomized to active Rx, but collapsed on the way up the stairs. Some had died and tumbled down. Others begged to be taken back down, and died later. A few survived.

Yes, the FIDDLE-1 investigators have argued the following:

"Some ungrateful wretches didn't even have the courtesy to show up for their very expensive treatment, given FREE, gratis and for nothing, by our illustrious corporation.

At first we excluded these parole violators..."

"... er, protocol violators, and thereby presented a PER PROTOCOL analysis.

This is CLEARLY the best way to examine device trial data. How can you benefit from a device if you don't get the device? Ridiculous nonsense spread by the likes of F2Harrell: ivory tower do-gooders."

"But FDA were satisfied by our 92% relative risk reduction, NNT 1.8, and asked for us to keep EVERYONE in the trial.

Even the parole violators! Hmph.

So we did. Obviously they didn't get the stem cells, so they are controls, yes?

This is the well known 'AS TREATED' analysis."

"So the results were:

PER PROTOCOL, 92% RRR, p<0.001
AS TREATED, 93% RRR, p<0.001

We pay taxes that provide your salaries, you FDA nobodies.

Shut up and give us our licence or we will call our friend Scott Gottlieb."

I see Donald Trump has voted.

Fortunately the banning of excessively sciency terms by US government staff...

has not yet reached the term "Intention To Treat", so the FDA officials raced to ask for the ITT analysis results before they were no longer allowed to.

Ah, the day before the ITT results are made public, this is the last year of share prices of FIDDLE Inc.

Well, the ITT results are not out, but clever folk like you and me have been doing some calculations.

Using the width of the confidence intervals, we back-calculated the numbers of patients ANALYSED in each arm [Another weekend we will study how to do that for fun and profit]

And since we know the PROPORTIONS, we can calculate the number of events in each arm, to be as follows.

First the first presented analysis, DELETING THE PROTOCOL VIOLATORS.

The PER PROTOCOL analysis.

While eating dessert, the ORBITA-HQ "calculation is fun" club also extracted the values for the second presented analysis, MOVING THE PATIENTS WHO COLLAPSED ON THE STAIRCASE INTO THE CONTROL ARM.

This is the "AS TREATED" analysis.

Can you immediately recognise what the ITT data to be published tomorrow will be?

Or shall I take you through it?

Let me make clear that unless you are a con-man of science there is no need to know the difference between as-treated and per-protocol.

Seeing either, is an immediate warning sign: "COME HERE TO BE CONNED".

However, let me take you through it, so you too could have sold your shares before the ITT results come out.

The difference between PER-PROTOCOL and AS-TREATED is that in one you DELETE people who collapsed on the stairs, and in the other other you PUT THEM INTO THE CONTROL arm.

Look at the two circled numbers. How many patients collapsed on the stairs on the way to active treatment, and therefore got DELETED in one analysis and PUT IN THE CONTROL ARM in the other.

Hint: it is the difference between these two numbers

Where should they actually be?

(For an ITT analysis)

es, the majority is right on both the number of patients and where they should go.

So how many ACTIVE arm patients should there be in the ITT analysis?

Hmmm. 6 votes, only 1/2 right.

HINT.
Both naughty analyses (per protocol and as treated) listed 300 pts in the active arm. Those made it all the way up the stairs.

There were another 700 active arm patients who didn't.

How many active arm altogether?

MEGA HINT.
300+700=what?

Ah, good, by 10 votes the right answer is in the clear lead.
So we can fill in the NUMBERS OF PATIENTS for the ITT analysis, as shown below.

How about the numbers of DEATHS?
How many DEATHS were shunted into the control arm when the FDA said not to delete them?

Number of deaths added to the control arm in the second analysis =

HINT.
Round the numbers to the nearest 100...

Instead of deleting them in the first analysis, or adding them to the control arm in the second, those deaths belong in the ACTIVE arm.

How many active arm deaths are there in total, in the ITT analysis?

The number you just said, plus the 15 we admitted to earlier.

Several people have pointed out to me that the banning of sciency words in US official documents was in fact fake news

I use the authority given to me by the statin denialists to ignore this fact and carry on quoting it, regardless.

It has a certain "truthiness", even if untrue

Anyway we have now recreated the ITT analysis, before they actually announce it!

Here it is!

Or, for those that don't like numbers:

ITT is the rubber of which the RCT balloon is made.

Per-protocol and As-treated are two different types of pin: don't let them near your reading of a trial, however persuasive the sales patter.

We don't have to be as blatant as the FIDDLE-1 investigators, to destroy randomization by bursting the ITT balloon.

There are more subtle ways.

The simplest is Cheetonics, also called landmark analysis. You require a patient to eat a course of Cheetos for 7 days.

If they don't do it, due to unfortunately dying in the first 7 days, they are parole violators. Delete them (or, for maximum fun, put into the "enemy" arm).

Cheetonics (see full explanation in this thread from Bob Yeh

External Tweet loading...
If nothing shows, it may have been deleted
by @venkmurthy view original on Twitter

) is particularly good in acute, serious conditions.

A more subtle way to ravage a trial once you burst the ITT balloon is to allow patients to be removed from ANALYSIS because the protocol is too arduous. It is easy to MAKE a protocol be (or feel, or sound) arduous if we think the patient is very sick and will have a poor outcome.

Therefore, unblinded to the allocation arm, and strongly believing in the therapy, it is easy for us to weed out the patients who are at highest risk of bad outcome, and blame patient choice.

Bonus marks if we can shove them into the control arm - "as treated" analysis.

Ultimately, allowing people to switch ANALYSIS arms prevents randomization from working at all.

What is the point of randomization, if the investigator (sometimes directly, sometimes by biasing the patient's choices) can move a patient into any arm at will?

If a treatment has a genuine BENEFICIAL effect, and some patients allocated to treatment refuse to have it when the time comes, what happens to the effect as reported by the trial in ITT analysis?

However that is life. The RCT is comparing the decisions you make AT THE TIME YOU ARE DECIDING.

You are advising future patients at this juncture that DECIDING to have (say) an angio is a good idea, or not.

If you later find there is no PCI to do, you don't remove the patient from analysis. They are in the invasive arm, even though you didn't PCI.

Or still in the CABG arm, even if the patient or surgeon decided in the end not to do it.

Or still in the diet arm, even if they have given up dieting and are back on the cream cakes.

Or still in the ezetimibe arm, even if they stopped taking their blinded medication.

Ali - I hope that answers your question.

Yes, ITT deflates the apparent benefit of the actual intervention but
(a) It is VITAL to prevent massive manipulation of RCT results
(b) In its defence, it reflects the effect of the DECISION to intervene.

Open to questions?

[Sorry I didn't cover the relevance to the panel recommending complete revasc. I realise it is not obvious so will cover another weekend]

Important point raised by Prof Rod Hayward of U Mich, an actual statistical expert. I was a bit anxious until he made clear that Per-protocol and As-treated are indeed a load of codswallop.

Prof Hayward is making a far more sophisticated point. If you could work out what was going on in the staircase, you might be able to find out whether the stem cells really helped or not.

However, this requires a lot of brainwork, and trust in the investigators.

If they've tried to foist an as-treated or per-protocol analysis on you, I suggest you don't trust them to do an Instrumental Variable analysis fairly.

The cited paper (bmj.com/content/340/bm…) makes an excellent case for instrumental variable analyses where MANY patients did not get their assigned treatment.

For example, about half of patients assigned to screening don't show up for it.

I would argue that the RCT shows the true effectiveness of (that particular) screening programme: lots of people don't show up, tough luck, the efficacy of deciding to do screening is _genuinely_ smaller.

However the COST EFFECTIVENESS calculation may have been biased by the non-attenders: we paid for the sessions even though they didn't show up.

In real life, we could double-book people and hope only half show up (you can tell I work in the UK NHS, can't you!)

So we MIGHT indeed want to extract the effect in the half that show up.

But there are subtleties (read paper, explains well). NOT just "shove them into Control arm if they didn't bother to show up."
e.g. They may be less interested in their health and therefore do worse.

Interesting question from Larry Husten at @cardiobrief @medpagetoday

Ah, remember Larry, Per Protocol is when we DELETE patients who didn't play by the rules.

E.g. Collapsed on the way to getting Stairway to Stem Cells.
Or Died waiting for LaserElectroDenervatisation.

"As Treated" is where we stick these troublemakers into the "enemy" arm.

Like James Bond finding Dr Dastardly has planted a bomb in his car. Instead of just throwing it out (Per-protocol) he actually throws it into Dr Dastardly's car (As-treated).

Per-protocol hides complications nicely, which is good if you want to hide them.
As-treated lets them show up, but in your enemy arm's data.

Either way, the very fact that either of these shady analyses are in a manuscript is a red flag.

Ali has a further question.

Yes, 42% is a large fraction not actually receiving a treatment.

But the IMPROVE-IT dudes _knew_ they would have to analyse ITT, so they encouraged persistence but couldn't do it in a biased way.

If Francis Industries took control of IMPROVE-IT, and knew that a big softie like you Ali was the regulator, you know what we would do?

(A) Find the highest risk people in the active arm, and make them stop the medication. Lots of ways to do it. "Oh aren't your muscles aching a bit? Surely, they must be! Look at this article by A Malhotra and the Cheeky Boys. Go on this cruise to listen to statin denialism 24/7"

(B) Find the LOWEST risk people in the control arm, and make them stop the medication. Can use the same script. (Under blinded conditions, rates of symptomatic side effects are same between statin and placebo ndph.ox.ac.uk/files/forms/eu… ).

So we would totally manipulate the trial, if we were allowed to delete the people we want to delete (or, even better, move them to the other arm).

To stop the temptation to do that, we insist on ITT.

Yes, it deflates the effect size. Tough.

Do you lock the front door of your home when you leave?

How many people would actually physically TRY to open your front door tomorrow, if you left it closed but secretly unlocked?

Give a realistic estimate.

We lock the door, despite the effort of keeping track of keys, the ease of lockpicking, and the annoyance of losing your keys. Because OCCASIONALLY someone might walk in and steal our stuff.

Likewise we insist on ITT because otherwise OCCASIONALLY someone will totally con us.

Excellent followup from Ali

"RCTs exist to prevent exaggeration."
Albert Einstein

It is prevention of OVERstatement of effects that RCTs exist for. If we weren't disproportionately fearful of overstatement, we would simply do a one-armed trial and compare with historical controls, guaranteeing a ~3-fold reduction in events (ncbi.nlm.nih.gov/pubmed/19875407)

So yes, all RCTs underestimate the true benefit of actually receiving the therapy.

Some RCT active-arm patients never get their pacemaker, CABG or simply don't take their meds (either secretly or by withdrawing from active participation).

Tough! That happens in real life too.

Anish, you are doing what Rx-advocates do. I am surprised!

Two reasons to stand by the ITT.

1.

An RCT informs us the benefit of deciding "OK let's take path X".

Here Path X =

Wait for slot
Do the op
Followup Rx etc.

If that is still the path in real life, then the RCT is CORRECTLY advising on the benefit of the Path X. Yes, you might get a stroke before the device.

2.

If you allow Francis Industries to remove that patient from analysis, then in other trials, of more acute conditions (e.g. STEMI, new onset AF) it will have a prolonged pre-procedure preparation period to flush out and delete as many events as possible.

An example of multiple cheeky things happening simultaneously here.

Per Protocol
Adjustment for selected baseline variables (but not others)
etc

And then a rather dramatic claim in the abstract conclusion that this is the new standard of care.

External Tweet loading...
If nothing shows, it may have been deleted
by @HenryJOncology view original on Twitter

A couple of other scenarios where a per protocol or as treated analysis will brilliantly enhance the results of the active arm:

ruptured aortic aneurysm

Or (Sergio Pinski):

Two excellent points from Venk Murthy:

The difference between ITT and As-Treated is where you put the parole violators (active people who fell down the stairwell, and control people who leapt out of their bed and ran up to the 32nd floor to demand stem cells).

If the stem cells cause subtle (say) dandruff it will be easier to pick up in the As-Treated. But beware not to let FIDDLE Inc. present "efficacy" data by As-Treated. they will usually try, once you allow the As-Treated in. Watch like a hawk.

Venk makes a second excellent point.

In a noninferiorty trial (medpagetoday.com/publichealthpo…) the sponsor's aim is to show NO difference, so they will love the ITT if very many people are parole violators, so both arms are getting almost the same treatment.

As-Treated can in that case be used to look out for the possibility that the amount of protocol violation is ENOUGH to falsely make the trial negative.

But there is no good way to confidently calculate the true treatment effect then.

There is Instrumental Variable adjustment where you try to adjust for the patient characteristics of those who cross over (protocol violators) but as we stated before you have to trust the people doing the adjustment to have not deliberately picked the variables to adjust for.😞

SUMMARY of the 3 analytical policies

1

INTENTION TO TREAT =

Randomize once, no cheating afterwards, whatever you or the patient choose to do, whatever happens to the patient, global warming, whatever ...

2

DELETE some people who didn't make it through the protocol (through choice or through unhappy events such as death)

This is "Per Protocol"

3

Don't just delete them from their randomized arm, actually STICK THEM INTO THE OTHER ARM.

So their terrible outcomes poison your enemy's arm.

This is "As treated"

CME quiz

I have a trial DUFF-1 of the efficacy of a new treatment.

It shows an absolute risk reduction of 12 percentage points on Intention to Treat.

p=0.07 unfortunately.

I use the "Per protocol" ploy. Things improve to an absolute risk reduction of 15 percentage points.

p=0.0542

Hmmm...

Lady from Francis Industries advises "let's just call it p=0.05, see if the reviewers notice?"

What about doing "As Treated" instead?

Will that make the absolute risk reduction...