A Controversial Study on Heart-Disease Risk

By Jane Farrell

A new study is raising questions about a tool designed to help physicians prevent heart disease and stroke among people at high risk for those diseases.

At issue in the analysis published Monday in the Journal of the American College of Cardiology is the “risk calculator,” which aims to help identify people who may face a high risk for cardiovascular events within 10 years. The study authors found that the calculator overestimated risk in the specific population in which they chose to apply it.

However, some medical experts were quick to criticize the new study itself, saying the research methods used were flawed so that the findings about the calculator must be dismissed for the general population.

This isn’t the first time the risk calculator has been the focus of debate in scientific and medical circles. When it was first unveiled by the American Heart Association and American College of Cardiology in 2013, some critics questioned its accuracy, although subsequent studies have found it to be very useful in identifying high-risk patients.

The authors of the analysis published Monday said the calculator needs some reworking.

“Our study provides evidence to support recalibration,” they said. “Ongoing research and dialogue in this area remains crucial and should be encouraged to provide more rigorous, valid evidence in contemporary, diverse populations.”

However, Dr. Donald Lloyd-Jones, who served on the expert panel that helped create the risk calculator, took issue with the new study’s approach and findings. Lloyd-Jones, the senior associate dean for clinical and translational research and chair of the department of preventive medicine at the Northwestern Feinberg School of Medicine in Chicago, said the study authors erred by excluding hundreds of people who were at high risk.

The study examined the calculator’s results for about 312,000 people treated by Kaiser Permanente Northern California over five years addressing risk factors including cholesterol, blood pressure, weight, smoking status, physical activity, blood sugar and diet.

Researchers examined the health records of the participants, who were between the ages of 40 and 75 and did not have existing cardiovascular diseases at the beginning of the study.

The 312,000 study participants were chosen from 4 million people treated within Kaiser Permanente Northern California. Approximately 86,000 people who received cholesterol-lowering statins during five years of follow-up were excluded. Nearly 256,000 people were excluded because they hadn’t been in the healthcare system or had pharmacy benefits for more than a year.

This tended to skew the sample toward heavily engaged people who were already receiving intensive preventive care, Lloyd-Jones said.

“Every step they took moved the study towards finding the result they were looking for,” Lloyd-Jones said.

The strongest evidence that the analysis is skewed is that the study population had a 1.4 percent prevalence of diabetes, Lloyd-Jones said. That compares to about 10 percent for the general U.S. population, a disparity that Lloyd-Jones said indicates the people studied generally were far healthier and at lower risk than the usual primary-care population.

Lloyd-Jones said he has seen a pattern across published studies analyzing the risk calculator: It tends to overestimate the risk in people who are later found to exhibit more healthy behaviors and measurements; and it underestimates risk for people who are shown to have more health risks.

And that’s fine, Lloyd-Jones said, because the whole purpose of the calculator is to be an initial screening device that leads to the next step of an in-depth and more personal discussion with the patient.

“The cholesterol guidelines say use this tool as the starting point for a conversation, then bring your patient’s individual characteristics in to help you make a decision,” he said.

Other studies have found the calculator to be effective.

A December 2015 study  found it to be superior to two other methods in predicting the risks of atherosclerotic cardiovascular disease. The study used records from nearly 38,000 people participating in the Copenhagen General Population study, ages 40 -75 who did not have atherosclerotic cardiovascular disease, diabetes or take statins when they enrolled.

A 2014 analysis published in JAMA: Journal of the American Medical Association found that the calculator predicted risk very accurately in a broad, representative sample of the U.S. population and ranked people most likely to have cardiovascular events.

Researchers in that study, including Lloyd-Jones, applied the calculator to data from nearly 11,000 whites and African-Americans, ages 45-79 without existing atherosclerotic cardiovascular disease, and examined the overall group as well as those without diabetes or taking statins upon enrollment. This study population closely mirrored the U.S. population, Lloyd-Jones  said.

The paper describing the risk assessment equation used by the calculator was included as one of four new ACC/AHA prevention guidelines. The cholesterol guidelines recommend that the 10-year risk of heart attack and stroke should guide treatment, instead of a cholesterol number.

While Lloyd-Jones defends the calculator, he also pointed out that criticism is an expected part of scientific progress, and expects further improvements over the years.

“From a broad perspective, this is exactly what should happen,” he said. “Any time we use something new based on science, people should poke and prod at it, see what works and what may be a weakness. That’s how we keep improving ways to treat our patients. But any changes should be based on the best science.”