NICE and the cost-effectiveness thresholds: Can good intentions compensate for bad practice?

That’s the title of the workshop we’ve held at UCL yesterday (I’d mentioned it in a previous post). I think it went remarkably well (OK \(-\) as I’m the organiser I may be over-enthusiastic but I really think it was a very good day!). Despite the fact that we’ve purposely limited advertisement as we wanted to keep it simple to start with we had a good turnout (about 30 people!).

All the speakers have agreed to make their talks available (which are here) and we’ve agreed that I’d try and summarise some of the main discussion points coming out of the presentations, so that we can perhaps move this forward, somehow. Here’s my attempt (any inconsistency is obviously due to my poor recollection of the events!):

1. Matt gave a very interesting talk focussing on the international comparison (of 5 main countries); he started by discussing the different approaches at defining the cost-effectiveness threshold (which was also reprised by James in the second talk). Then, he moved to give some evidence of how in general agencies (with the notable exception of NICE) tend to resist a clear definition of what the threshold is. He presented some interesting work that has tried to elicit the underlying value of the threshold by analysing a set of past decisions in the Australian setting. What I found interesting of the Dutch case is the idea that the threshold can be set as a function of the disease severity, which made me think of potential links to my talk on risk aversion (see later). One of the interesting points of his discussion of NICE’s case is the idea of a range of thresholds (rather than an absolute value). We had a little less time to discuss France and Japan, which are all interesting cases anyway (France because the formal use of cost-effectiveness methods is newly established; Japan because of the mix of private and public funding as well as the recent move towards formal inclusion of cost-effectiveness considerations, mainly driven by the need of capping health expenditure). Probably, in the end, the main message is to wonder whether strict adherence to a very specific guidance (such as NICE’s) is a good idea. As I was listening to the talk, I also thought that in a more expanded version of this, it would be perhaps interesting to look at North America.
2. James’s talk concentrated on different ways of determining the threshold, effectively (and quite amusingly) pointing out at the many flaws in virtually all of these (which in itself is evidence of how complex this problem is!). I found the discussion of the “historical precedent” method fascinating and the fact that this is actually used in some countries is kind of strange. Then he gave a very clear account of the “budget exhaustion” method, linking back to some applied examples (eg Netherlands and Ireland), showing the incredible range of variability present in many cases. I found his comments on why it may not be ideal to have a range of thresholds very sensible and while I was listening to the talk I kind of changed my mind about that 2 or 3 times… James also linked to the Claxton et al paper on value based assessment and its potential impact in setting some plausible values. I think this is a very important piece of work that we ought to consider very careful (if we are to discuss the problem in general terms). Finally, I think another very good point that James made is about the idea that in fact, all this procedure is dynamic — setting a threshold based on current budget and then ranking interventions accordingly may need continuous revisions, which in turn may also lead to modifications of the threshold itself, to add to the complexity of the problem.
3. I gave arguably the most confused talk of the lot. The first part discussed the meaning of PSA and what are the quantities that should be involved (and why so). This, I feel, is something that I have thought about quite a lot and I think I have a clear idea of what the problem is and how it should be tackled. I was trying to lead this back to the main issue of selecting the threshold but I think Chris Jackson‘s comment that, as statisticians we kind of take that for granted, sums up nicely the fact that it is difficult to embed this aspect in the statistical model. In the second part of my talk, I have tried to (still confusingly) address the problem of risk aversion and its implications to PSA in general and the threshold in particular. I have presented a couple of potential utility functions that could be used to include a risk aversion parameter, together with a brief description of their characteristics. I liked James’ comment that while difficult to think about it, it would be possible to elicit reasonable values of the risk aversion parameter, which in turn would make the analysis more applicable. While I was talking (and relating to the previous presentations) I also thought that may be risk-propensity is just as important (eg in cases where a disease is perceived to be so important as to be granted some sort of special treatment). Also, Matt asked the very fundamental question of whether decision-makers should be risk averse, in the first place \(-\) I think nobody knew the answer to that one…
4. Finally, Mike presented to us his model addressing the impact of the several assumptions underlying the standard definition of the threshold. The model attempts at transforming this into some more complex object, embedding the ideas of “value” and “information” available to the (different) decision-makers. The model is quite complex, but Mike explained it brilliantly (one of my PhD students came to me after the talk still shaking with the excitement!). The idea is to try and include formally several aspects that characterise the fixed assumptions underlying the definition of the threshold (eg the divisibility of the technologies involved, the level of information available to the decision-makers and, crucially, the fact that the process is driven by sequential decisions made, in general, by a set of actors, rather than the vaguely defined “decision-maker”). Model complexity aside (which I quite liked), the results are also very interesting and show that in fact, the straight line in the cost-effectiveness plane can be thought of as a special case of a more general threshold which can assume very different shapes (including kinks and steps). The very interesting point made by Mike is that if you start accounting for value and information, then you (read: NICE) may find yourself in the situation where it may be cost-effective to replace an existing technology with something that, while non cost-effective in the conventional sense is still a better option. This is kind of intuitive, but the model actually formalises the intuition and produces thresholds that have suitable shapes to accommodate this situation.

As I said, I don’t really know where we go from here. But it was a very interesting start to this discussion!