An NHS-partnered app that uses artificial intelligence to assess medical symptoms and boasts more than 2.5m users faces regulatory scrutiny after complaints from doctors, who warn it can miss signs of serious illness, reports the Financial Times.
Babylon Health has formed partnerships with the English National Health Service as well as tech giants Samsung and Tencent, expanding rapidly since it was founded four years ago. Investors include the founders of Google’s DeepMind AI unit while users include Matt Hancock, the UK’s new health secretary and former culture secretary.
The report says the UK medicines regulator asked questions about the app after one doctor complained it had failed to identify symptoms of a heart attack or deep vein thrombosis. Two other doctors are quoted as saying they had complained about wording on the website, which has since changed, that could confuse patients. Babylon, one of a number of new technology products being adopted by overburdened health services eager to cut costs, has two significant partnerships with the NHS.
Some patients in London can register to use the app’s video consultations to communicate with a doctor instead of registering with a traditional GP’s surgery. Babylon also delivers a telephone advice service called NHS 111 in north London.
The report says the complaints have shone a spotlight on a regulatory system that seems to lag far behind new innovations and classes Babylon’s service alongside items such as spectacles and sticking plasters. Uber was criticised last month for using a different health app, PushDoctor, to verify drivers’ vision with video calls, a length of string and home-delivered eye charts.
Babylon said in the report that it had responded to questions from the Medicines and Healthcare products Regulatory Agency, which had confirmed the company was not under formal investigation. The MHRA said it could not comment on specific companies but added that it “regularly carries out post-market surveillance and maintains dialogue with manufacturers”.
The report says it tested Babylon’s symptom checker to understand its response to the two conditions at the centre of the first complaint. When told a 66-year-old obese male smoker was experiencing sudden chest pain and excessive sweating, Babylon suggested 9 out of 10 people with similar symptoms were likely to be having a panic attack and made no mention of the risk of a heart attack. By contrast, the NHS website’s advice page for “chest pain” contains an alert box warning people to seek medical advice and call an ambulance if the pain begins with symptoms such as sweating. “You could be having a heart attack,” the box says.
According to the report, Babylon said: “Variable outcomes are still possible if a different symptom is selected, as each symptom could suggest an increase or decrease in the probability of a larger number of potential diseases.” Disclaimers on Babylon’s symptom checker and terms and conditions say that the chatbot’s responses are not diagnoses and do not constitute medical advice.
Dr Richard Body, a cardiologist and professor of emergency medicine at Manchester University, and Dr Annette Neary, a former consultant with the NHS, both expressed concerns about the results when contacted for their views. “Lay-people don’t realise that 90% of diagnosis is based on history,” Neary said.
The report says a rheumatologist from north London, who spoke on condition of anonymity, said he had been incorrectly referred three patients through Babylon: “With standard NHS GPs, the quality is variable, but on average the quality is higher,” he said. Babylon said: “Statistically your ‘consultant rheumatologist’ would only expect to see one or two of these referrals, so we don’t think this gives his/her perspective much credence beyond blatant self-interest … We regularly conduct clinical audits of our referrals, which consistently demonstrate they are of high quality and are clinically appropriate.”
Another doctor, GP and medical writer Margaret McCartney, has complained to the MHRA as well as the Care Quality Commission and Advertising Standards Authority about wording on Babylon’s support website, the report says. According to a screenshot she shared, the page about the NHS 111 non-emergency service “Is it safe?” used to read: “a study examining the app’s safety and accuracy found that it gave safe advice in 100% of the cases used in the test”. “I was concerned that the app was unsafe and was being oversold,” she said. The page no longer exists in the same form, the report says.
Hugo Farne, a respiratory doctor, also is quoted in the report as saying he had complained to the ASA about language on Babylon’s website. According to screenshots shared, the website used to say Babylon’s technology was certified as a medical device with the MHRA and that its video consultation service had been “comprehensively inspected by the Care Quality Commission”.
Correspondence from the MHRA said Babylon would be asked to amend the wording. The ASA’s website indicates that five complaints about advertising by Babylon have been informally resolved.
According to the report, Babylon said it constantly updates the content of its websites. “As an innovative company working at the forefront of technology and operating in an industry with many vested interests, Babylon operates under very close scrutiny,” it said. It added that the complaints about its chest pain and deep vein thrombosis responses had been made by an individual with “potentially vested interests”.
The report says the doctor rejected the allegation. “We go far further than traditional GPs when it comes to safety measures, as every patient interaction is recorded, encrypted and can be played back for later review,” the company said.
Andrew Haldenby, director of think-tank Reform, said technologies such as Babylon had huge and widely recognised potential to speed up access to health services and enable quicker diagnoses: “The potential is so great that it is easy to get carried away… it’s important to remember that the technology is still in its early days.”Financial Times report (subscribers only)