The limitations of a new doctor-rating database, Surgeon Scorecard, are on full display in the Bend medical market.

Of the 21 local surgeon complication rates that Surgeon Scorecard published in July, 13 are based on fewer than 100 procedures and seven of those are based on fewer than 30 procedures.

Small sample sizes mean the complication rates calculated by Surgeon Scorecard aren’t very meaningful, said James Cochran, statistics professor at the University of Alabama. “It doesn’t mean they’re wrong. It just means we don’t have a whole lot of information to go on.”

Since the New York journalism organization ProPublica released Surgeon Scorecard, the medical community has zeroed in on its shortcomings, and a few critics have called for it to undergo a scientific peer review or to be redacted. While ProPublica plans to improve the database in the future, there are no plans to kill the project.

And the Oregon Health Care Quality Corp., a nonprofit that has done similar work in measuring the performance of clinical practices, says there’s no need to do so. Surgeon Scorecard has no fatal flaws, Q Corp. Senior Director of Informatics Steve Merryman wrote in response to an inquiry from The Bulletin.

Q Corp. would have used a slightly larger minimum sample size and grouped similar results into broad categories, such as “better,” “average” and “poorer,” he said. “Overall, is the methodology ProPublica used perfect?” Merryman said. “No, but we believe the methodology honestly attempts to address the common concerns brought up in health care quality measures.”

ProPublica created the scorecard from five years worth of Medicare data on common elective surgeries. Complication rates were based on deaths and readmission to the hospital for certain reasons, such as surgery-related infection. ProPublica says it was conservative in deriving the complication rates and consulted with medical experts in order to be fair to doctors.

Most of the published complication rates for the Bend area were in the “medium” range.

The question of where to draw the line on sample size was a judgment call. ProPublica published complication rates for doctors who had done at least 20 of any one of the eight elective procedures under Medicare. If the line had been drawn at, say, 50 procedures, the results would have been more credible, but the database would have included very few doctors in less populous areas like Central Oregon, Cochran said. “They’re balancing things here.”

Consumer advocates applaud Surgeon Scorecard because it’s one of the first efforts to make doctor-specific performance data available to the public. ProPublica has acknowledged the limitations but says limited data is better than none at all.

Cincinnati cardiac doctor Edward Schloss, a self-proclaimed fan of past ProPublica projects, disagreed on that point. He called for Surgeon Scorecard to undergo a scientific peer review and pointed out in a lengthy, critical blog post that ProPublica failed to explain its margin of error as prominently as it stated the complication rates. “I and others wonder if consumers will be able to interpret this complex data without a more up-front discussion by reporters,” Schloss wrote.

ProPublica reporter Marshall Allen said he believes the general public can understand Surgeon Scorecard’s limitations. “It’s a starting point for much more meaningful conversations with doctors,” he said. Patients should ask, “What’s your total complication rate?”

He continued, “You know what’s crazy is a lot of these people don’t even really know. A lot of surgeons do not track their complications.”

St. Charles Health System and Bend Memorial Clinic each track complication rates and are willing to share data on doctors who appear in Surgeon Scorecard.

Only one local doctor, Timothy Beard, was deemed to have a “high” complication rate for gallbladder removal in Surgeon Scorecard. But Surgeon Scorecard’s rate was based on 26 procedures, the number of in-hospital surgeries Beard performed on Medicare patients.

With such a small sample size there’s a chance that Beard’s overall complication rate is actually “medium,” and it’s possible it could be “low,” according to the database. Cochran, the statistics professor, doubted that most people would understand the nuance. “They’re not going to understand how little confidence we really should have in this result.”

Most gallbladder removal surgery is out-patient, and Pro Publica will update Surgeon Scorecard to reflect those larger numbers, Allen said. It could make a difference for Beard, who did 190 gallbladder removals at Bend Surgery Center over the same time frame considered by ProPublica, 2009 through 2013, and had just two complications, a rate of 1 percent, BMC Chief Medical Officer Dr. David Holloway said.

Holloway said he would favor making BMC’s doctor-specific performance data public. “I think that’s where all of this ends up, with total transparency,” he said. “Where I would like to see us go over the next several years is to become really transparent in a way that makes sense to the public.”

Surgeon Scorecard also prompted executives at The Center, an orthopedic and neurology specialty practice, to consider whether to publicize its outcomes. “What can we publish that will be useful for the general public?” CEO Mike Gonzalez said.

The medical community has also blasted ProPublica for trying to measure surgeons’ performance from Medicare billing records on hospital readmissions. Gonzalez pointed out that one of The Center’s surgeons, Dr. Timothy Bollom, had three complications among 187 Medicare patients undergoing knee replacement, a rate of 1.6 percent. Furthermore, Gonzalez said, two of those patients were readmitted for fever, something that’s not necessarily related to surgery.

Despite doctors’ complaints, hospital readmission rates continue to be used to measure quality, and a study published in the New England Journal of Medicine in September 2013 supports that practice. The authors looked at nearly 480,000 patient readmissions at 3,004 hospitals and found that high surgery volume and low mortality rates are associated with lower rates of surgical readmission.

ProPublica points out in a paper on its methodology that among the readmissions counted in Surgeon Scorecard, the average length of stay is five days, “more than one would expect if a patient was simply hospitalized as a precaution.”

Holloway supports transparency of quality metrics, but he said a single doctor shouldn’t shoulder the responsibility. “The state of medicine anymore, it’s really a team sport. Things are so complicated, it really is a team.”

Doctors have also worried that ratings will discourage them from taking on difficult cases.

Dr. Kathleen Moore, an orthopedic and spine surgeon with Desert Orthopedics, is one of several local doctors doing lumbar spinal fusion, and whom Surgeon Scorecard listed as having a “medium” complication rate. Moore said the database doesn’t reflect the fact that she and her partner are the only local doctors who take on complex scoliosis cases.

She said many of her patients are older, and that leaves them prone to complication, but their other option is to live with a lot of pain, and possibly heart and lung problems, she said. “It really skews the data tremendously when you have practices that are very different,” Moore said. “Those physicians who are willing to do the tougher work are going to look worse.”

And comparisons among doctors who are all ranked in the same category may not mean much in Surgeon Scorecard. All of the local complication rates for lumbar spinal fusion were based on 30 or fewer surgeries. Moore said she does 75 to 100 fusions per year. For all types of surgery, she said, her complication rate is around 2 percent.

Dr. Darrell Brett, whom Surgeon Scorecard listed as one of the best-performing spine surgeons in Oregon, doesn’t have a problem with the Surgeon Scorecard. “They have a right to know,” he said of patients.

Brett said he doesn’t do scoliosis cases. “However, I do a lot of difficult reoperations,” on people who had prior surgery with a different doctor, Brett said in an email. He added, “I take all-comers so I certainly don’t ‘cherry pick.’”

On his website, Brett says he does about 750 spine surgeries a year, mainly on the lumbar and cervical spine with “clinical outcomes” in the top 1 percent of national peers and an infection rate of less than .01 percent.

Doctors are extremely competitive, yet they’re sensitive to being called out, said Doris Peter, director of Consumer Reports’ Health Rating Center. Even within institutions, it can be controversial to share individual performance data, she said.

Peter applauded ProPublica for shedding light on what many people have suspected, which is that individual doctors affect patients’ outcomes. She hopes it will push the medical industry to release data that’s even more reliable than Medicare billing records. “We know there’s other data out there, held by the American College of Surgeons,” she said. “It’s not available to the public.”

— Reporter: 541-617-7860 kmclaughlin@bendbulletin.com

Editor’s note: This article has been corrected. The original version misidentified the city in which cardiac doctor Edward Schloss lives and practices medicine. The Bulletin regrets the error.