Key Takeaways
Trust is still a major issue in the adoption of AI in healthcare. While numerous initiatives such as CHAI and the National Academy of Medicine’s recently launched Steering Committee for a Code of Conduct exist, a great deal of work remains to be done as an industry to demonstrate responsible AI practices and robust replicability of models.
More research is needed on the economics and implementation science of AI in clinical practice. Automation of administrative tasks has the potential for significant cost savings as well as impact on clinical care. However, more data is needed on the actual costs of implementation and for the use of large language models which, in some cases, (eg. OpenAI products) may have significant unit costs.
Case studies on the ethical choices developers face across performance, explainability, fairness, and bias are needed to better inform the field of the complexity of responsible or ethical AI. There is a market in virtue, they say, but merely deploying the terms will not help the field advance; there needs to be an understanding of the hard decisions developers and organizations will face in managing the various components of what constitutes responsible AI. Fears of liability issues often prevent transparency, however, a commons-based approach to governance that also sheds light on the “how” of responsible AI could advance the field significantly in coming years.
Introduction
Chilmark Research has been covering AI since around 2017, and we are now please to introduce our first-ever eBook, Building Responsible AI in Healthcare: A Journey into Automation, Ethics, and Trust, a curated selection of articles and reports from the past few years. While AI is the buzzword du jour, there are plenty of challenges and questions to answer before we begin to see significant impact at scale across most domains. From clinical decision support systems (CDS) to data analytics, imaging, operations, and administrative tasks to biopharma, the use cases for AI are quite broad.
However, as we are learning quite rapidly, AI is not traditional software and implementation is not always as straightforward as the hype would have it. The underlying economics of AI and its impact on organizational design are even less mature than the models themselves. Nevertheless, there are many issues to focus on between the poles of technosolutionism (there’s an AI for everything) and technopessimism that accompanies the recent growth of AI. Prudence will be the essential ingredient in developing AI strategies for organizations, which is why we have chosen a path heavily focused on responsible AI in our research.
Dominant Themes: Responsible AI
There has been a great deal of lip service paid to ethics and responsible AI, but not a great deal of case studies into how healthcare organizations are actually doing it so far. One of our key messages is the need to go beyond checklists and that responsible AI will, in many cases, be much less than a right or wrong answer, but about how choices and trade-offs are made. In the real world of developing AI algorithms, balancing the various needs for transparency, performance, and fairness/equity, there will almost definitely be compromises made for one over the other.
So how do we make these choices? Who benefits and who may be harmed? What do we do with the answers to these questions beyond sweeping them under the rug? We will build far more trust in the long run by learning more about these decisions and putting measures into place to address the fallout from a particular trade-off, when possible.
On a related front is the inevitable question of generative AI. The spring of 2023 was met with a tremendous amount of hype and overinflated promises that generative AI will transform medicine and healthcare. While the hype continues and some pilots have been launched, we have noticed a growing amount of caution as well as some interesting efforts to fill in some of the holes in our knowledge and modifications of architectures of large language models (LLMs) to address problems like hallucinations. Some insights that smaller models and open-source models can match the performance of the larger models are beginning to emerge as well.
One issue that remains bothersome to many critics of AI and LLMs is the ongoing tendency to anthropomorphize AI. A good example is the notion that LLM outputs can be more empathetic than doctors. The models obviously cannot feel or think; to what extent this observation is either the result of chance or the benchmarks used to evaluate performance remains to be seen in the long run. This issue — along with the growing controversy over the training datasets — and copyright issues, relative lack of attention to responsible standards, and a growing list of biases found due to the training data, are beginning to weigh heavier on OpenAI and tools like chatgpt4.
We also looked at Boeing’s challenges with the 737 Max to understand more about what healthcare AI can learn from other industries. The key lesson here is that a shift in corporate culture that gave safety less priority than profits resulted in crashes that killed hundreds of passengers and had a major negative impact on profits. Furthermore, this was a case study that showed how neglected end-users in the design process can have catastrophic results.
Automation: Ethics and Economics in Healthcare
We have had the opportunity to work with a number of vendors over the past year that proved to be insightful in how we think about automation. These relationships deepened the role of both AI and non-AI tools to automate components of service delivery or administrative functions such as utilization management. In our eBook, we have a non-AI example of automation from Lumeon, the care orchestration platform that automates processes around perioperative care that can reduce the work burden for clinical staff and allow more time for difficult cases that demand more care. Another case study is XSOLIS’ approach to utilizing AI for utilization management in a way that builds bridges between payers and providers and saves time.
In 2021 we looked at the AI4Ops space and were impressed with the maturity of the offerings in this space. In addition to addressing the administrative and operational side of hospitals, they also can have a major impact on clinical care as well, when use of hospital assets is optimized, which in turn leads to shorter wait times and higher patient satisfaction and quality of care. LeanTaaS, one of the leading vendors in this space, offers an excellent case study of how machine learning applied to optimization challenges in discrete, deterministic processes can make an impact on the bottom line in a relatively short amount of time. Machine learning can be applied to address the mismatch between supply and demand of assets in a highly effective manner. They are applying the same approach to adapting LLMs for user-tools that allow more convenient access to data dashboards for decision-making.
We are also paying more attention to the economics and implementation science of AI. To date, there have been relatively few studies of the economics of AI in healthcare, which may be due to the tricky nature of AI and the impact it has on decision-making and institutional design. We are clearly in the extremely early days of understanding how organizations may need to rethink organizational structures and decision-making with these new tools. The unit costs of LLMs — at least for the leaders like OpenAI — are still worrisome from the limited data available at the present time.
Recently Daron Acemoglu and Simon Johnson have written about AI and automation and the often found mismatch between AI and human-driven tasks. They draw upon the language from some of the original thinkers in AI in the 1950s, who preferred the term “machine usefulness” to describe how robust AI tools could be applied to tasks where humans are less adequate than machines. Frequently, however, we see weak AI tools applied to tasks that humans are well suited for; perhaps obviously, productivity does not improve. They refer to this as “so-so” automation that rarely leads to the productivity gains we expect from AI and other digital health tools. These insights are worth revisiting as we think about the design issues for AI in healthcare in order to avoid creating new mousetraps that merely create additional work burdens and burnout for clinical staff, as we have seen with the design of EHRs over the past 30 years.
Conclusion
We are publishing this eBook to provide insights and case studies that can facilitate adoption of the best AI tools. In a period of excessive hype, we feel it is important to think critically about where machine learning can play a role as well as where it is an inappropriate tool that could actually cause harm. As the field of AI has developed, we have seen a large number of examples of AI doing harm; once individuals or communities are harmed, it can be difficult to repair.
At the same time, there are more tools available to assist data scientists in addressing issues such as bias, explainability, and fairness. Nevertheless, as an industry, we need to develop more standards and benchmarks for responsible AI in healthcare that are pertinent to the range of use cases where AI can be applied. Some aspects of this may be partially determined by emerging regulatory frameworks, as we have seen in the European Union and to some degree at the state level in the US, as various jurisdictions begin to create frameworks for data governance and AI.
Stay tuned for much more coverage of AI in healthcare as well as our new offerings in the futures and forecasting space. We will be releasing a Future of Health IT Map that organizations can use as a planning tool across our multiple coverage areas. We may also be partnering for more applied consulting in the responsible AI space in the coming months as well.
0 Comments