NUMBER JUMBLE CLOUDS JUDGMENT OF DRUG WAR

Differing Surveys, Analyses Yield Unreliable Data

As the election season began gearing up in late 1991, President George Bush
got an unsettling bit of front-page news:

The number of habitual cocaine users in the United States had jumped an
astounding 29 percent in a single year, from 662,000 to 855,000, according
to the National Institute on Drug Abuse (NIDA). Bush had aggressively
pushed his administration's anti-drug effort. Now, he had little to show
for it.

But the bad news, widely reported by newspapers across the country, was
wrong. NIDA had miscounted in its annual National Household Survey on Drug
Abuse, one of the nation's "leading drug indicators." A year later, without
fanfare, the number of habitual users was revised back down to 625,000.

"Problems with statistical imputation," the General Accounting Office
concluded in a 1993 report on the miscalculation that received little
public attention. "We certainly think that more adequate quality control
procedures could have caught findings of such significant policy relevance."

The 1991 cocaine mistake stands out as just one example of the tenuous
grasp scientists, politicians, the media and the public have in evaluating
America's 25-year crusade against drugs. Different methods of calculating
the number of drug users continue to produce widely gyrating estimates,
including those contained in the 1997 White House drug strategy report that
variously gives the number of habitual cocaine users as 582,000 and 2.2
million.

In spending a proposed $16 billion on the federal drug war in 1998 -- a 400
percent increase since 1986 -- lawmakers will rely on reams of data that
often attempt to impose statistical order on a chaotic social problem that
defies easy analysis. Extensive federally funded efforts to accurately
assess the subterranean drug world have led to contradictory findings and
occasional statistical curiosities, such as a 79-year-old female respondent
whose avowed heroin usage in one survey resulted in a projection of 142,000
heroin users, 20 percent of the national total.

"It's clear that these things are badly mismeasured and nobody cares about
it," said Peter Reuter, the former co-director of drug research for the
non-profit RAND think tank and now a University of Maryland professor.
"That's because drug policy isn't a very analytically serious business."

Measuring the drug war with any precision is a daunting task. Hard-core
drug users are hard to find, much less question, and people frequently lie
on drug-use surveys -- one study shows two-thirds of teenagers giving
deceptive answers. Since surveys typically receive only a small number of
positive responses, analysts risk making substantial errors in creating
projections for the entire nation. Survey results sometimes include
warnings acknowledging these obstacles, such as "subject to large sampling
error" or "great caution should be taken."

But the caveats often are downplayed or ignored, either by those issuing
the data or by journalists and others promulgating the information. In
reporting the apparent 1991 jump in habitual cocaine use, for example, the
White House's Office of Drug Control Policy noted that the statistics were
both "cause for concern" and "highly unreliable."

The difficulty in measuring and evaluating the nation's illegal drug
problem made it harder to set policy, stoked partisan rhetoric and confused
the public, drug analysts say. Many experts, for example, believe cocaine
and crack use are in decline, and the federal household survey indicates
that overall drug use is down 49 percent from its peak of 25 million
monthly users in 1979; yet many Americans still perceive the drug war as
perennially lost.

"You really can't tell from the big debate that goes on in public what the
big picture is," said David Musto, a Yale University medical historian who
has studied drug trends for three decades. "When I tell people about it,
they're completely surprised by the fact there has been a decline since 1980."

That big picture can be obscured by drug statistics that are "often
incomplete, erratic and contradictory," in the words of two RAND
researchers funded by the government to measure cocaine consumption. The
first problem of drug war analysis is the sheer number of measurements --
there are more than 50 federal drug-related "data systems" with hundreds of
"drug variables" produced by an array of federal agencies.

For cocaine alone there are national statistics on casual use (at least
once a year), current use (at least once a month), frequent or habitual use
(at least once a week), crack use and use broken down by age, race and sex.
There are stats on tonnage consumed, purity, price per gram, price per
kilo, patients reporting cocaine problems in emergency rooms, patients
seeking treatment and so forth.

"It's not that one thing is better than the other," said Eric Wish,
director of the Center for Substance Abuse Research at the University of
Maryland. "They all give a different piece of the puzzle, and they need to
be put together. But because of federal turf issues, it's more of an
adversarial process than a collaborative relationship."

Reuter said he has pointed out discrepancies in the habitual cocaine-use
figures in the national strategy report in the past, but the discordant
numbers keep appearing. On page 11 of the 1997 strategy, the count of
habitual cocaine users is given as 582,000, a number that "has not changed
markedly since 1985." But in a chart on page 227 of the strategy's budget
summary, the number of such users is given as 2,238,000.

"I can't seem to get the machinery that cranks out these reports to pay
attention to these inconsistencies," Reuter said.

An official with the Office of National Drug Control Policy blamed the 1997
inconsistency on "sloppy writing." But the precise reasoning behind it
gives a glimpse into the problem of gauging the drug war. The warring
numbers in this case come out of different measuring methodologies -- one
based on the household survey, the other on urine tests of jail inmates --
that give radically different results.

"The truth is probably somewhere in the middle," said Joe Gfroerer, who
manages the household survey for the federal Substance Abuse and Mental
Health Services Administration (SAMHSA). "It's just a difficult thing to
estimate."

Jared Hermalin, the GAO project manager who uncovered the 1991 cocaine
mistake, said: "There's every reason to believe that maybe the numbers are
not absolutely correct but the trends are correct. That's the main thing we
need to know."

In recognition of the need for better analysis, the office of national drug
policy director Barry R. McCaffrey has proposed a comprehensive Performance
Measurement System intended, for the first time, to standardize measurement
of the drug war.

"Facts should drive policy, but they haven't until very, very lately, with
McCaffrey," Sen. Joseph R. Biden Jr. (D-Del.), a longtime critic of the
household survey's measurement of hard-core cocaine use, said in an interview.

The proposed system shows just how complex measuring the drug war is. It
contains one mission statement with five goals, 32 objectives and 99
"targets" that will be tracked by more than 111 "measures."

Even when the data is not marred by obvious statistical flaws, the sheer
profusion of it can baffle those looking for simple answers on whether the
drug war has been a success or failure. There is consensus that overall
drug use, as well as marijuana and cocaine use specifically, have declined
dramatically since the 1970s. But that clarity soon clouds when researchers
delve deeper.

For example, according to the household survey, current (monthly) cocaine
use decreased in the 1980s -- and was often cited as a sign of success;
but, also according to the household survey, hard-core (weekly) use did not
drop, and that was cited as a sign of failure. More recently, even as the
household survey shows that the overall number of cocaine users has
declined (success), emergency room data shows that the number of people
seeking medical treatment for cocaine problems is rising (failure) as
chronic addicts age and their health deteriorates. And the household survey
may show that overall drug use is down (success), but a high school survey
shows that teenage marijuana use is up (failure).

For the past 25 years, the nation's most prominent gauge of illegal drug
use has been the national household survey, begun by NIDA in 1972 and taken
over by SAMHSA in 1992. Government workers annually conduct one-hour,
in-person interviews with a randomly selected sample of 18,000 people, age
12 and up. From the answers, statisticians extrapolate the size of the
nation's drug-taking population.

The second most-publicized measurement is the NIDA-sponsored, 22-year-old
"Monitoring the Future" survey. Each year, more than 51,000 high school
students at more than 400 public and private schools are polled about their
drug use.

In the 1970s and 1980s, the household and high school surveys were treated
as national news on the state of the drug war, particularly in tracking the
rise of marijuana and cocaine.

"I've been looking at the household survey and the high school survey for
years and years," said Eric Sterling, a former House Judiciary Committee
staff counsel now with the Criminal Justice Policy Foundation. "They have
an effect like electric shock on a dead frog's leg. There's a spasm people
have when they get this data. People, certainly on Capitol Hill, look to
respond."

In the mid-1980s, the advent of crack played havoc with the existing
measurement system. Simply put, there was no measurement in place for crack
use -- crack was so new that the household survey did not start asking
about it until 1987.

Faced with an unprecedented national outcry after the overdose death of
University of Maryland basketball star Len Bias on June 19, 1986, Congress
rushed through a law punishing crack cocaine possession at a rate 100 times
that of powder cocaine. Without hard data, lawmakers relied heavily on
high-pitched media accounts, some of which "were not supported by data at
the time and in retrospect were simply incorrect," the U.S. Sentencing
Commission later concluded in a comprehensive study on "Cocaine and Federal
Sentencing Policy."

"It was really the opposite of science," said Sterling, who wrote the draft
version of the crack law when he served with the Judiciary Committee. "It
was mythology-driven. It was said repeatedly that there were 3,000 new
crack addicts every day. These kinds of numbers would get thrown out and
repeated without anybody doing the arithmetic or asking: `How does this
number relate to anything we know about the usage?' "

The lawmakers believed -- erroneously, it would later turn out -- that
crack had killed Bias. (Testimony from someone who was with Bias when he
died pointed to powder cocaine.) Congress reacted so strongly to crack in
part because it believed it was dealing with a rapidly spreading "crack
epidemic."

Yet the household survey eventually estimated that crack use stabilized
almost immediately and never approached the levels that powder cocaine had
- -- crack stood at 668,000 monthly users in 1996 compared with more than 5
million for powder cocaine in 1985, according to survey figures.

But the statistical data eventually provoked just as much criticism as the
absence of data did. Crack use turned out to be harder to measure than
powder cocaine use. Like heroin, crack quickly concentrated among poor
urban addicts. Many of them lived on the streets, where they would not be
counted by the household survey.

"The household survey and the school survey are pretty useless for
measuring hard drug use in the population," said Wish, the University of
Maryland research center director.

By the late 1980s, drug researchers like Wish thought that the nation's
cocaine problem was breaking into two distinct groups: mainly white
suburbanites who used cocaine casually on weekends and mainly black urban
addicts who used crack or cocaine daily. For casual users, Bias's death
seemed to have the effect of scaring millions off cocaine; the household
survey indicated that after 1985 the number of monthly cocaine users
plummeted 70 percent.

Yet the trend in hard-core usage is still being sorted out.

In 1990, just as the Bush administration had begun touting the decline in
casual use, then-Senate Judiciary Committee Chairman Biden produced a
report counting habitual cocaine users at 2.2 million. That was nearly
triple the household survey's estimate.

Biden's numbers had come from what would eventually emerge as a third
leading indicator of the nation's drug use -- the Justice Department's Drug
Use Forecasting (DUF) program, started in 1987. The DUF program collects
voluntary urine samples from 30,000 jail inmates in 23 cities across the
country each year to test for cocaine and other drugs. Biden's figures were
extrapolations from these urine tests.

Mark Kleiman, a Harvard researcher who supervised the Biden committee's
work, subsequently acknowledged that the methodology was "not precise." But
he said conservative assumptions were used to come up with numbers that
gave a clearer picture of the nation's cocaine use.

But the GAO and household study researchers like Gfroerer say that the DUF
urine tests cannot be used to extrapolate larger numbers because they are
not part of a randomly selected scientific sample.

"DUF really isn't representative of anything," Gfroerer said. "The way it's
collected, you can't project it out to any population."

Although the household survey is based on a randomly selected sample, it
also has limitations, according to some researchers. Only a tiny percentage
of people admit to heroin and cocaine use, and they must then become the
basis for projections into the millions of users. For example, of 32,594
people surveyed in 1991, only 127 admitted to using heroin in the past
year, according to the GAO. From this number the survey projected 701,000
heroin users nationwide.

Thus, small errors in the way the survey is carried out can be magnified.
That means yearly shifts of a few hundred thousand in a projected user
population of a million are statistically insignificant because they could
be explained by possible errors in sampling, reporting or extrapolation,
Gfroerer said.

The GAO found such problems in the 1991 cocaine and heroin figures. For
heroin, further investigation revealed that 53 of the 127 users counted in
the survey were inappropriately "imputed" -- researchers made a subjective
decision to count them even though they gave contradictory answers. When
the error was later corrected, the number of heroin users dropped 46
percent to 381,000.

Moreover, of the 701,000 annual heroin users originally estimated in 1991,
142,000 were derived from the survey response of a lone 79-year-old white
woman. Her answer was weighted in an effort to make the survey result more
representative of the nation's population; but the resulting statistical
projection accounted for one-fifth of all the estimated heroin users in the
United States that year, according to the GAO.

"The bottom line is [that] to make projections from the household survey to
the number of heroin users in the country is probably not a good idea,"
said Hermalin, the GAO project manager. "Cocaine [estimation] is dangerous,
too."

In 1994, the household survey was revamped to make it more accurate at
counting hard-core drug use, but Gfroerer said the difficulty was "only
partially" corrected.

"The basic issue of understating of hard-core drug use, those problems are
exactly as they have been," Gfroerer said. "We still feel it's important to
collect these data as part of the survey. The real issue is how you report
them."

MEASURING THE DRUG WAR: For the last 25 years, progress in America's battle
against illegal drugs has been measured primarily by four "leading drug
indicators."

SURVEY: National Household Survey on Drug Abuse

BEGUN: 1972

SPONSOR: Substance Abuse and Mental Health Services Administration (SAMHSA)

FREQUENCY: Every three years since 1972, yearly since 1990

YEARLY FUNDING: $5.7 million

METHODOLOGY: Questionnaires given to a randomly selected sample of 18,000
households; answers are reported anonymously

STRENGTHS: Best measure of overall national drug trends, especially
marijuana and first-time users

REPORTED LIMITATIONS: Undercounts hard-core heroin and cocaine users who
don't live in households; some people lie on surveys; sampling errors
distort results

OVERALL TRENDS REPORTED: Overall drug use down 49 percent since 1979 peak;
cocaine down 70 percent since 1985 peak; marijuana down 58 percent since
1979 peak; crack stable since 1988

SURVEY Monitoring the Future Study

BEGUN 1975

SPONSOR National Institute on Drug Abuse

FREQUENCY Yearly

YEARLY FUNDING $4 million

METHODOLOGY University of Michigan researchers poll more than 51,000
eighth-, 10th- and 12th-grade students at more than 429 schools

STRENGTHS Best early warning system for drug use among youth

REPORTED LIMITATIONS Doesn't count dropouts; undercounts non-white
students; some students lie on surveys

OVERALL TRENDS REPORTED For high school seniors, overall use down 37
percent since 1979 peak; cocaine use down 70 percent since 1985 peak;
marijuana use up 84 percent since 1992 but still down 41 percent since 1978
peak; heroin use up sharply since 1985

SURVEY Drug Abuse Early Warning Network

BEGUN 1978

SPONSOR SAMHSA

FREQUENCY Yearly

YEARLY FUNDING $2.5 million

METHODOLOGY Data collected from patients in more than 500 emergency rooms
in 21 cities

STRENGTHS Best measure of people with chronic or acute drug problems

REPORTED LIMITATIONS Not a representative sample; counts people seeking
treatment along with people who overdose; counts suicide attempts with
legal drugs

OVERALL TRENDS REPORTED More and more drug users are ending up in emergency
rooms -- a record 531,827 in 1995, up 43 percent since 1990

SURVEY Drug Use Forecasting program

BEGUN 1987

SPONSOR Department of Justice

FREQUENCY Quarterly

YEARLY FUNDING $2.4 million

METHODOLOGY More than 30,000 jail inmates at 23 U.S. cities submit to
voluntary urine tests

STRENGTHS Best measure of drug use among the criminal population

REPORTED LIMITATIONS Isn't a scientific sample; can't be used to
extrapolate national figures

OVERALL TRENDS REPORTED Crack and cocaine use declining among U S.
arrestees; 17 cities reported declines in percentage of positive cocaine
urine tests in 1996

SIX WAYS OF LOOKING AT COCAINE USE

Over the past 10 years, different efforts funded by the federal government
have produced wildly different estimates of the number of hard-core
(weekly) cocaine users in the United States.

The most conservative estimates come from the National Household Survey on
Drug Abuse, which was revised in 1994 to better measure hard-core use. (A
mistake in 1991 led to the report of an erroneous single-year jump of
200,000 hard-core users.)

Other studies combining the household survey findings with urine-test data
from jail inmates have produced larger figures but differing trendlines.

A RAND study released in 1994 showed a slow upward trend.

A 1995 study by Abt Associates Inc. showed falling and rising trends. When
Abt improved and revised its methodology in a 1997 study, it counted more
hard-core users than ever.

SOURCES: National Household Survey on Drug Abuse, Office of National Drug
Control Policy, General Accounting Office, RAND, Abt Associates Inc.

THE BIG PICTURE

The number of illicit drug users has declined sharply since 1985.

IN MILLIONS

1996: 13 million

The amount of federal money spent on drug control efforts continues to
increase rapidly.

IN BILLIONS

1998: $16 billion requested

SOURCE: Office of National Drug Control Policy, National Household Survey
on Drug Abuse

© Copyright 1998 The Washington Post Company