What is MedPaLM?
What it means for the future of Mental health and Conversational A.I?
Hey Everyone,
I was questioned by a journalist recently who was writing a story about A.I. at the intersection of mental health. And it got me thinking of Google’s Med PaLM.
Google Research and DeepMind have recently launched MedPaLM, a large language model aligned to the medical domain.
The new AI model is supposedly just 0.3% worse than doctors at answering medical questions. How can this be?
The tool is an example of a large language model or LLM, which are designed to understand queries and generate text responses in plain language, drawing from large and complex datasets – in this case, medical research. Med PaLM could evolve into BigTech being our A.I. therapist and friend, to also fish more personal mental health data to tailor us even more personalized Ads.
A.I writers will continue to use the HER (2013) analogy to garner clicks, for example the connection between journaling, ChatGPT and therapy. Elsewhere people have used ChatGPT in bots without consent in therapy apps like Koko.
I don’t think DeepMind is at all nefarious about MedPaLM, but without controls and regulation, you can see how ChatGPT’s API will be misused in health and mental health contexts.
The paper is dated December 26th, 2022 and in its they discuss PaLM (a 540-billion parameter LLM) and its instruction-tuned variant, FlanPaLM, on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA, MedMCQA, PubMedQA, MMLU clinical topics), including 67.6% accuracy on MedQA (US Medical License Exam questions), surpassing prior state-of-the-art by over 17%.
Therefore a Conversational A.I. could be pretty smart about health topics and issues. In 2023, RLHF will improve drastically and in 2024 it could be ready - as some ChatGPT descendent or derivative for mental health and friendship “reorientation” with A.I council.
Chatbots will get faster, more eloquent, more truthful (less prone to hallucinate) and more empathetic, relatively fast from 2023 to 2026. Conversational A.I. at the intersection of healthcare and mental health could therefore have a moment in the 2020s.
Microsoft could use technology like VALL-E a new text-to-speech AI model (VALL-E) that can closely simulate a person's voice when given a three-second audio sample, so putting people at ease using perhaps a familiar or a favorite voice. A “David Attenborough chatbot” giving me dating advice, you never know!
The point being Google, Amazon, Microsoft and even Apple are racing to combine A.I. -advantage with healthcare. A mental health/friendship personal assistant personalized just for us is the obvious answer with the way LLMs are progressing. Getting the voice right and being personalized to the consumer (patient) is the obvious key. Teaching ChatGPT like agents more empathy with human feedback in all types of mental health scenarios is key.
Google DeepMind’s MedPaLM is the best example that comes to mind.
Will the Smart Operating System Soon Offer us Therapy?
They propose a framework for human evaluation of model answers along multiple axes including factuality, precision, possible harm, and bias. Anthropic’s Constitutional AI and harmlessness from A.I. Feedback is another framework (the model is called Claude) that could easily result in a mental health - friendship product.
As ChatGPT’s API becomes available, it will be implemented into countless products and apps. Microsoft will leverage its antitrust worthy deal with OpenAI to “sell AI to the world”. And it won’t always be safe, but it will be better than anything we’ve had yet and put Amazon Alexa and Google Assistant to shame.
With DeepMind’s work on Med PaLM, the researchers concentrated on answering medical questions in order to evaluate the potential of LLMs in medicine. Reading comprehension abilities, the capacity to accurately retain medical information and the manipulation of expert knowledge are necessary for responding to medical queries. This frame of reference combined with DeepMind’s Sparrow or LaMDA would not be difficult to train. A Google Duplex (2018) for Mental health, if you will.
DeepMind’s showed that comprehension, recall of knowledge, and medical reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal important limitations of today’s models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLM models for clinical applications.
DeepMind introduced a new benchmark dataset called MultiMedQA, combining 6 existing open questions-answering datasets + HealthSearchQA, a new dataset of medical responses searched online. They used “instruction prompt tuning” to improve Flan-PaLM, creating a new model called Med-PaLM. To simplify, they focused on how to ask better questions to the model, providing well-crafted examples first.
Med-PaLM performs very well, especially compared to Flan-PaLM. It must still outperform the judgment of a human medical professional. Currently, a team of medical experts found that 92.6 percent of the Med-PaLM responses matched the answers provided by clinicians. Google could essentially marry Med-PaLM with Sparrow and voila, you’d have a half decent A.I. therapist or at worst, a potential counsellor with medical knowledge. Useful for around the home I imagine in 2024?
MedPaLM has 67.6% accuracy on the US Medical Licence Exam questions. To pass, it’s enough to have 60% accuracy. This was back in 2022, DeepMind could partner with Google Health and create a ChatGPT for our psychology and self-help, meanwhile having access to some of the data (because it is Google after all).
Like I was trying to explain to the inquiring journalist, this isn’t only possible, it’s probable that they are working on this right now in January, 2023. Microsoft and Google themselves have the greatest incentive to own such a Conversational A.I. With Amazon and Apple a close second.
Recent advances in large language models (LLMs) offer an opportunity to rethink AI systems, with language as a tool for mediating human-AI interaction. LLMs are “foundation models” , large pre-trained AI systems that can be repurposed with minimal effort across numerous domains and diverse tasks, says the paper. If Adept AI is going after AGI, it’s actually a startup like Inflection A.I. that I think would be tempted to work on such a mental health therapist or A.I. companion like product.
In May, 2022 Inflection AI, the new artificial intelligence start-up from DeepMind co-founder Mustafa Suleyman (pictured above) and LinkedIn co-founder Reid Hoffman, has secured $225 million in funding.
The intersection of our mental health data and A.I. is just getting started in 2023. The British entrepreneur says Inflection will look to develop a new suite of technologies that will eventually enable anyone to speak to a computer in plain language. Sometimes it’s not plain language that we want, it’s actual companionship from the A.I. And therein lies the golden ticket of projects like Med PaLM in my opinion.