Disappointment with AI in radiology

Gary Smith and Jefferey Funk writing for qz.com:

Furthermore, only 40 of the more than 80 radiology algorithms currently cleared by the FDA, along with 27 in-house tools, were utilized by respondents. Only 34% of these were used for image interpretation; the other applications included work list management, image enhancement, operations, and measurements. The bottom line: only about 11% of radiologists used AI for image interpretation in a clinical practice.

The American College of Radiology survey agreed with Ng: “A large majority of the FDA-cleared algorithms have not been validated across a large number of sites, raising the possibility that patient and equipment bias could lead to the inconsistent performance.”

The disappointment is related to insufficient training of datasets while being cleared by the regulator. Therefore, while it can protect against indemnity, the underlying expectation is flawed. If the premise of having a magical AI algorithm sits back through the workload holds with every purchase, the buyers are going to be dissatisfied. It comes down to what algorithm is being pushed and for what purpose. Therefore, also the need to have a consistent focus on standardisation. They have quoted a study (published) on a survey which is highly inconsistent with capturing the truth. It only reflects the “mood”. Besides, who would ensure radiologists find time magically tick of questions in surveys?

There was a comment on this on another blog too:

Interesting. I’m not sure what to think here. AI will only get better, not worse, so it seems reasonable to suppose that in the not-too-distant future it will be useful, at the very least as aid to radiologists. A lot of work has to get into making any system be useful in practice, but there’s lots and lots of money in radiology so I’d think that someone could be put on the job of building a useful tool.

I don’t necessarily disagree, and I am not going to turn cynical about the potential. We merely need training datasets and “real-world” applications with built-in metrics to constantly iterate software in the background. Those buying it should understand its limitations too.