David Sackett had a good idea in the 1960s when he suggested that physicians in Canada could improve their diagnostic accuracy if they checked the literature. Archie Cochrane, an English physician, had a dynamite idea when he proposed that researchers start collecting the burgeoning medical literature at designated centers where it could be rigorously synthesized. The psychologist Gene Glass also added a piece to the puzzle. He was frustrated by the methods fashionable thirty years ago to summarized literature by counting the number of significant vs. insignificant results or reaching "professional consensus." He developed a statistical procedure now called metaanalysis, which quantifies the measure of effect in each research study and allows similar studies to be combined in quantitative ways.

These three components - practitioners blending personal experience and research evidence, the accumulation of best evidence in large databases that can be searched, and statistical procedures for estimating measures of treatment effect combining across studies (generally called systematic reviews) - are the foundation for evidencebased medicine and dentistry.

Sackett's original definition focused heavily on the individual practitioner: "Conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients." As the powerful tools of electronically searchable databases and systematic reviews have come to the fore and as a few hundred researchers have invested part of their careers in this field, a broader definition of EBD has emerged: "Integration of best research evidence with clinical expertise and patient values." (See the ADA position on EBD and useful links here.)

In this essay, I will focus on issues in EBD that arise from the concept of "best evidence" and the notion of combining sources of evidence. I will also comment on the fear that EBD can be used to limit the freedom of practitioners. This is a hard argument to dispel because there is some truth in it.

The Meaning of "Best Evidence"
There are three problems with the concept of "best evidence." First, the term is ambiguous. Let me try to make the point by mentioning that I only eat in the "best" restaurants. I too seldom dine at the French Laundry in Napa or the Inn at Little Washington, acknowledged exemplars of unique food impeccably prepared from the best ingredients; big wine lists; and attentive, knowledgeable service and plenty of it. Researchers at the University of Oxford in England have developed a list like this of the characteristics of "best evidence" in health care research, which is available at www.eboncall.co.uk/content/levels.html. This is the sense of best evidence that advocates of EBD have in mind- the best imaginable. Most of the EBD research has focused on establishing such standards and sorting the literature in search of a few good examples.

But there is a humbler and more frequently used definition of best. It means that none of the other available alternatives is better. I have had some excellent meals in Chapel Hill, Omaha and Portland. I pick the finest eateries in those towns, and, when necessary, at the best dives on the Interstate when I am tired and hungry. The best evidence available may not meet the Oxford standards, but it is still the best in a very real sense. When a practitioner bases his or her choice of therapy on a single randomized control trial report in JADA, years of successful personal experience, or the advice of some knowledgeable colleagues, the dentist may be using the best evidence available.

The advocate for EBD make a useful point when they urge practitioners to review the literature generally and investigate specific issues as part of their practices. They also make a useful point in insisting that we should give heavy weight to research results that are grounded in good science. William Gies said so eighty years ago when he founded the International Association for Dental Research, and it is still true. My second concern is that the "best evidence" in the EBD sense may not actually exist in sufficient quantities to be a useful guide to practice. Although individual randomized controlled trials are not the best evidence, they are near the top of the list. Between 1990 and 2000 there were 106 RCTs published in peer review papers regarding endodontic therapies, 438 in oral surgery, and 198 in orthodontics. The current supply of "nearly best evidence" is simply too thin to provide the kind of scientific base to practice that animates EBD enthusiasts. (See Richard Niederman's June 2003 paper in the Journal of Dental Research.)

The third problem with best evidence is subtle. Evidence should be valid, meaning that conclusions from the samples observed would not differ much from the conclusions drawn from observing every possible meaningful case. Thanks to EBD and many other researchers, the standards for internal validity (methodological rigor) have been elevated in oral health research. What remains largely unaddressed is the problem of external validity. This concerns itself with the representativeness of the sampling. EBD researchers seem focused on sophisticated conversations among themselves over extracting defensible findings instead of finding out how practitioners use information or what information they find valuable. Are carefully controlled RCTs representative of the typical practices to which their conclusions are meant to generalize? Best evidence cannot be defined without reference to the context in which it is to be applied. This issue has yet to be tackled by EBD.

The Meaning of Combining Evidence
There are also issues with the part of the definition of EBD having to do with combining evidence. The methodological advances that systematic reviews has made possible for combining multiple RCTs into a single estimate of the measure of effect for an intervention are impressive. The methodological advances in EBD have come in learning how researchers combine scientific data rather than in how practitioners combine research with their own experience and presenting patients. Beyond the obvious unresolved issue of utility, there is a logical problem here. Researchers do not "combine" data in the same sense that practitioners do. Research combining cannot take the place of the combining done in dental offices. Researches describe and practitioners act. When describing, various evidence is averaged, weighting for the credibility of the source. This is what EBD researchers do when they publish. But taking an action precludes taking alternative actions; it is a different logic and an "average" approach is not possible. Under such circumstances the appropriate decision rule is to commit all resources to the action favored by the most credible evidence and ignore the other data. The combining problem (or more properly the problem of selecting the best strategy) is apt to be even more complex than just suggested. Averages are reported in the literature because theoretical summaries are meaningful in theoretical contexts. We are moving away from p-values as tests of differences between averages (toward confidence intervals). But we haven't moved far enough to be meaningful when introducing research results into practice settings.

Practitioners are concerned with predictability of results as well as with average performance. They want products and procedures that are unlikely to fail. Thus robustness - satisfactory outcomes over a range of realistic situations in individual offices - becomes a desirable criterion for practice-ready evidence. Robustness has yet to enter the thinking of proponents of EBD.

As practiced today, EBD is predominantly an academic discipline trying to sell itself to dentists. In order for it to be successful, at a minimum, it must solve the problems associated with a useful definition of "best evidence" and what it means to "combine evidence" in the context of dental practice. EBD has been useful as a reminder that science has always been, and must continue to be, the foundation upon which progress in practice is grounded and as a development in the way data from multiple studies of good methodological rigor can be summarized and reported.

These are useful, perhaps even necessary improvements, but they are insufficient. More science needs to be done before EBD can be considered useful to the practitioner.

Policy-Based Dentistry
A good understanding of EBD has been made difficult by two red herrings. Evidence-basers have characterized practitioners who disagree with them as empirics; practitioners who fear the development have cried "third-party interference and control." Neither of these is a useful way to promote the profession. When a practitioner reports that "X works well in my hands" he or she may be reporting a scientific fact. Noting that "Y works better in some other people's hands" is not a satisfactory argument to the contrary. What must be compared is the outcome in context not the approach in the abstract.

The argument about outside control is as difficult to address because it is as ill defined as EBD is. Researchers and academics that support EBD have generally said that they are not advancing uniform standards for how care is to be provided. They are, however, sometimes caught talking about practice guidelines, parameters of care, clinical protocols, or critical paths. At issue here is who decides that care is appropriate for a particular patient in a particular office. The definition of EBD certainly implies that individual dentists do. But sometimes the EBD literature spills over from "best evidence" to "best practices."

The challenging case, especially for the insurance industry, is the middle ground. Isn't it reasonable for government, brokers and payers, and even professional associations to convert trends supported by evidence into guidelines and to make those guidelines mean something in practice? There are benefits to be gained from this approach, and standards of care and consensus conferences existed for years before the three-letter-buss-word. Precisely because the opportunity for individual practitioners to combine best evidence in their offices is reduced by this practice, it cannot be called EBD. I prefer to think of it as policy-based dentistry - somebody decided what is best generally and attempts to move, by various incentives, all practitioners in that direction. EBD can contribute to creating good policy. But in its current incomplete state of evolution, it can also mislead. As just one example, evidence-based dentistry (and dental research generally) has failed to develop a complete method for managing sampling variance and its impact on generalizability of research results. Valid policy must be robust enough so it does not preclude acceptable variation in practice that is appropriate or even best given prevailing practice circumstances.

Research methods designed to find the "best" are blunt instruments for finding the generally best. We need to know which variation matters. One can get in over his or her head trying to cross a river that is only three feet deep - on average.

Good science should be one of the foundations for PBD (policy-based dentistry), just as it is for EBD. Because PBD has not been identified as a respectable entity to be explored seriously, it is even less likely to be understood than EBD. There is a field of health policy research with a rich tradition but few practitioners. Bringing these experts into the discussion would be a useful protection against the view that research will provide sufficient answers to oral healthcare strategy.

Here is my summation. EBD has stimulated some valuable innovations in clinical dental research. Organized databases and meta-analyses are sound new tools, with more productive years of development in front of them. Discussions about "best evidence" have been useful for researchers, although they have not gone far enough in considering external validity or in identifying sources of variance and promoting technologies that are context and practitioner robust. Practitioners need to be shown how to verify and improve the outcomes in their offices, not told that they should practice in a certain fashion because others have found it effective.

Dr. David W. Chambers is Professor and Associate Dean for Academic Affairs and Scholarship at the School of Dentistry, University of the Pacific and Editor of the American College of Dentists.

You may also download this article as a PDF.