Research has shown the potential for artificial intelligence (AI) to assist in the diagnosis of conditions, like endometriosis, but what happens if the modelling is incorrect?
Recent research from PhD student Alison Deslandes, from the IMAGENDO team at the University of Adelaide's Robinson Research Institute, has begun to address this by developing a quality scoring system for gynaecological images used by diagnostic AI algorithms.
"The development of valuable AI tools to assist with ultrasound diagnosis is dependent on algorithms being developed with high-quality data," says Ms Deslandes.
"Previously, researchers have shown that the performance of deep learning systems is reduced significantly when applied to images from low-cost ultrasound machines with lower image quality.
"It's important we consider creating AI tools to assess image quality at the same time as exploring the use of AI algorithms to diagnose the condition to ensure satisfactory information is being used."
After a literature review, researchers developed a way to score the quality of transvaginal ultrasounds (TVUS), with six professionals assigning a score of one to four to 150 images of the uterus and ovaries.
The images were assessed against five factors - correct depiction of anatomy, view of anatomical structure in the field of view, image optimisation, ability of image to be interpreted for diagnosis or pathology, and overall clarity.
A score of four was given to optimal quality images, three to suboptimal images and two to images with a poor image quality; a score of one was assigned if the images were rejected.
The professionals were paired based on their fields and asked to examine the images together (inter-operator results), as well as individually (intra-observer results).
"What we found was only poor to moderate agreement (on the image quality) when our paired observers looked at the images and mostly weak to moderate levels when the individual observers looked at images again after more than a week," says Ms Deslandes.
"Some more refinement of the scoring system may be needed to improve agreement, however interpreting ultrasound image quality carries a level of subjectivity so it remains unclear whether quantification of image quality can be achieved.
"The inherent subjectivity in image quality assessment may explain our intra-operator results which, although better than our inter-operator agreement, were weaker than expected."
"Although some AI systems can tolerate labelling noise, most will favour clean (high-quality) data."
Ms Deslandes said although AI systems will likely be able to assess image quality (IQ) more objectively than humans, development of these systems will depend on human labelling which will likely feature noisy data due to the inherently subjective nature of ultrasound IQ.
"As such, reliable methods of IQ scoring are essential to further progress the development of AI systems in gynaecological ultrasound as are AI methods capable of accommodating noisy labelling," says Ms Deslandes.
