A consistency evaluation of AI doctor and real doctor was held for the first time in Chengdu Haiersen Hospital, Southwest China's Sichuan Province on June 30, 2023. Photo: Screenshot of Guangming News

A consistency evaluation of AI doctor and real doctor was held for the first time in Chengdu Haiersen Hospital, Southwest China's Sichuan Province recently. In the 91 cases reviewed, an AI doctor "MedGPT" and the attending doctors reached 96 percent agreement on case results, the Global Times learned from the hospital on Monday.

According to the sponsor Haiersen Hospital, more than 120 patients and 10 attending physicians from West China Hospital participated in the assessment research. According to the event organizers, this was the first evaluation exercise completed by an AI doctor in China.

There were seven departments and eight consultation desks on-site, including two cardiology units, urology, orthopedics, etc. Each consultation desk was equipped with a doctor's assistant who was responsible for communicating the patient's symptoms to the AI doctor "MedGPT" in written form, while the real doctor also received the patient's information synchronously in the consultation system.

"This evaluation is not a competition between AI and real doctors. The purpose of developing MedGPT is to hopefully use this AI doctor to assist real doctors in consultations through continuous optimization, the staff member of the hospital said, according to China News.

"MedGPT" mainly works in primary diagnosis of patients, such as common illnesses like headaches and fevers, helping real doctors understand and judge the possible range of diseases. The medical AI can help patients better understand their symptoms and how to manage their diet during treatment. In this evaluation, both MedGPT and real doctors analyzed the same cases, and their responses were compared in real-time on separate screens, the Global Times learned from the Haiersen hospital.

During the free consultation day on June 30, a 58-year-old female patient told the doctor's assistant that she often had abdominal bloating and burping issues. The doctor's assistant immediately conveyed these problems to both "MedGPT" and the real doctor in written form.

Soon, "MedGPT" first confirmed that the patient's symptoms were related to the digestive system and guided the patient to describe her symptoms, such as whether the bloating and burping appeared slowly or suddenly, how long they lasted, and whether there were any other symptoms. The patient answered that she had been suffering from abdominal pain, bloating, burping, dry mouth, and bitter taste for a year.

Meanwhile, the real doctor first asked about the patient's height, weight, mental state, and other basic information. After receiving the patient's responses, the doctor asked whether she felt acid reflux, heartburn, and inquired about her sleep and diet. 

Both "MedGPT" and the real doctor's examination suggestions included abdominal ultrasound, gastrointestinal function tests, and gastroscopy. After checking the results, the real doctor diagnosed the patient as suffering from a fatty liver.

The "MedGPT" did not directly provide suggestions but continued to ask the patient about her height and weight. After calculating that her body mass index was within the normal range but close to overweight, "MedGPT" suggested the patient do exercises like jogging and yoga. MedGPT did not provide suggested medication.

"To ensure the accuracy of this evaluation, both AI doctor and real doctor only conduct consultations in written form," staff member told reporters.

Seven experts from Peking University People's Hospital, China-Japan Friendship Hospital, Beijing Friendship Hospital, and Fuwai Hospital reviewed 91 cases and scored the AI doctor's consultation, diagnosis, treatment suggestion, auxiliary examination scheme, data analysis accuracy, provision of explainable information, and fluency in natural language during the consultation.

The results show that real doctors scored an average of 7.5, and the AI doctor "MedGPT" scored an average of 7.2. The consistency of the scoring results between the AI doctor and the attending doctors from West China Hospital reached 96 percent.

Currently, the AI doctor "MedGPT" can diagnose more than 3,000 common diseases, and the number of diseases it can diagnose will significantly increase by the end of the year when the first phase of testing is complete, a staff member told the Global Times on Monday.

Now the use of AI in clinical practice is becoming more and more popular, and the advantages of related technologies in fine operations such as disease diagnosis and surgery are still very prominent, a doctor of Peking University Third Hospital who has used AI technology for help told the Global Times. For example, AI can make a clear diagnosis based on a series of examination results such as the patient's film and has a very surprising success rate. 

But AI technology still has room for improvement, and the current technology cannot completely replace a trained specialist, he noted. Current AI technology can only be carried out in accordance with the established procedures set, and once there is an error, it will have very serious consequences, so the doctor also needs to supervise and adjust the whole process of the application of AI technology.