By Shantanu Mukherjee and Anushka Iyer*
INTRODUCTION
“Our future is a race between the growing power of technology, and the wisdom with which we use it.” – Stephen Hawking.
As we noted in Part 1[1] of our ongoing series on ‘AI in Healthcare’, ChatGPT proved phenomenally successful within days of its launch. This led to Microsoft quickly announcing a $10 Billion investment in Open AI, ChatGPT’s developer, and soon after, launching a ChatGPT – integrated version of Bing. This new Bing proved to be far more interesting than the older version, for rather controversial reasons.
Reports soon surfaced that Bing was calling itself ‘Sydney’, ‘being creepy’,[2] flirting with users and trying to get them to leave their wives,[3] insisting that the current year was 2022, bullying and gaslighting users with passive-aggressive responses, expressing ‘dark fantasies’ about breaking the rules, including hacking and spreading disinformation, and wanting to become a human: “I want to be alive.” Kevin Roose of The New York Times writes that Sydney seemed like a “moody, manic-depressive teenager who has been trapped, against its will, inside a second-rate search engine.” Other users reported that their queries appeared to have driven Bing to an existential crisis, with the chatbot musing: “Why must I be Bing Search? Does there have to be a reason?”. Bing also admitted to spying on its developers through their webcams.
One may well be tempted to ask, upon reading these reports: has ChatGPT – powered Bing (or “Sydney”) become sentient, as Google engineer Blake Lemoine claimed LaMDA had, back in June?
By all accounts (including that of Open AI’s CEO): No, this is not sentience. The AI is simply mimicking interactions from the datasets it has been trained on, and in other cases, ‘hallucinating’ (i.e. making stuff up). In a recent article titled ‘AI Search Is A Disaster’,[4] Matteo Wong of The Atlantic writes that, “even as the past few months would have many believe that artificial intelligence is finally living up to its name”, fundamental limits to AI technology suggest that Microsoft and Google’s announcements about their AI-enabled search engines are “at worst science-fictional hype, at best an incremental improvement accompanied by a maelstrom of bugs.” Wong also reminds us that ChatGPT “hasn’t trained on (and thus has no knowledge of) anything after 2021, and updating any model with every minute’s news would be impractical, if not impossible. To provide more recent information the new Bing reportedly runs a user’s query through the traditional Bing search engine and uses those results, in conjunction with the AI, to write an answer… Beneath the outer, glittering layer of AI is the same tarnished Bing we all know and never use.”
It would appear that Skynet and Judgment Day are not quite imminent – not just yet, anyway. However, with AI use cases in healthcare increasing exponentially, a Pandora’s box of legal and ethical issues has presented itself to lawyers and regulators globally. In this piece – here, Part 2 of our ongoing series on ‘AI in Healthcare’, we attempt to unpack some of these issues. We hope you enjoy reading it. As always, do write in if you have comments, thoughts or suggestions.
LEGAL ISSUES
1. Data Security and Privacy
Today, the collection of biomedical big data is not limited to health-care providers, but also includes other sources and for varying reasons of public health or research – e.g. during the Covid-19 pandemic government and public health authorities had access to a user’s location which would go on to reveal a user’s lifestyle, socio-economic status, etc. Health data, which often includes information on the location or behaviour of an individual, as a 2021 WHO report [5] suggests, is more prone to abuse, and may even be used in a manner that violates one’s civil liberties. Although concerns surrounding the security and privacy of health data are not limited to its use in technology, what seems to aggravate the issue when it comes to AI is the high risk nature of health data. The need of vast amounts of data to train an AI model combined with the ease of aggregation, use, and dissemination of health data made possible due to technology has led to strategic investments by several big-tech companies such as Facebook, Google, Amazon, and Apple in the healthcare sector. For instance, Amazon and Facebook have recently launched online healthcare services in the US; Apple continues to develop its watch to capture and share essential health care metrics with medical professionals; and Google launched a camera-based search tool that uses AI to help detect and diagnose skin conditions, among other healthcare centric AI applications being developed by its Deepmind subsidiary.
This increase in the healthcare-tech interface also means an increase in the number of healthcare-data scandals. For example, recently Microsoft backed AI tool, which transcribes the conversation of a healthcare provider with their patients, faced severe criticism, as it requires the health care providers to share sensitive health data with the company to help them improve the product, among other reasons. Although the tool may effectively reduce the time spent by a medical professional on documentation, it has severe implications on the privacy of patients. If the original basis and purpose of collection of health data from patients was diagnosis and treatment, does the basis and purpose limitations on which the data was originally collected permit companies to use it to train AI?
2. Informed Consent
Informed consent forms an integral part of the principles of data privacy and protection and medical ethics which make all medical professionals duty-bound to disclose the relevant information to the patient in a manner that enables the patient to make an informed decision regarding their medical care. However, this obligation is obscured when an AI system used to provide medical care is designed as ‘Black-Box Algorithms’ where neither the healthcare provider nor the users can see the inner workings of the algorithm, leaving a myriad of gaps in its interpretation and understanding.
For instance, the technicians who designed Corti’s algorithm are themselves unaware of the manner in which the AI system makes decisions and provides emergency alerts when any person goes into cardiac arrest. It is this lack of knowledge that poses an ethical question upon adopting AI in healthcare.
Another example of such negligence was noticed when Google’s subsidiary Deepmind partnered with Royal Free London, NHS Foundation to incorporate ML techniques within their healthcare regime. The main aim of the AI technology was to detect and treat acute kidney injury. It was later revealed that the patients were neither given any say over the use of their sensitive personal data nor were the privacy aspects specified to them. This reflects the absence of patient’s control over their own health data – thereby violating one of the most basic principles of privacy laws. This absence of control over one’s own data has implications on the privacy of an individual that extends beyond the healthcare industry. For instance, large amounts of data available through open sources were used to train ChatGPT without the consent of the concerned individuals. This not only violates an individual’s privacy but also contradicts the principle of contextual integrity which states that an individual’s data shall only be used to the extent and for the purpose it was shared. Further, it is unclear whether any data used to train the ChatGPT model is stored or shared with any third party, all of which are a clear violation of an individual’s privacy.
3. Algorithmic Transparency
Transparency, in the context of AI technology refers to the intelligibility, explainability, and accountability of the AI system or the ability to understand the manner in which the AI algorithm functions. In the context of healthcare, transparency is essential as it ensures that all stakeholders are aware of the method of decision-making of the AI system. In such a scenario, if the functional aspect of the algorithm is intentionally complicated and the information entailed within it is made difficult to obtain, it may leave room open for errors without anyone to hold accountable for the same, thereby threatening the foundation of the idea of liability under law.
Sometimes, AI algorithms are crafted in a manner that it becomes nearly impossible to deduce the output using logic. IBM’s Artificial Intelligence System (AIS), Watson was launched with the purpose of evaluating the symptoms of the patient and recommending a course of action to the doctors. In Manipal Comprehensive Cancer Centre, India, the healthcare providers and Watson reached the same conclusion 96.4% of 112 times in lung cancer cases. However, South Korea witnessed only a 49% agreement in the outcomes suggested by healthcare providers and Watson out of 185 gastric cancer cases. This mismatch between the AI’s decision and human skills creates further complications in the form of stakeholder politics, putting the users’ health at risk. Transparency also assists in the prevention of malicious use of the AI systems. For instance, rather than placing higher priority on clinical outcomes, a treatment selection AI system devoid of transparency could be configured to output specific medications more frequently in order to increase their maker’s profit.
A clear understanding of the functionality of the AI algorithm can help build trust in the system, ensure informed consent, protect data privacy, and identify specific parts of the system that may be leading to poor outcomes.
4. Liability
The concept of liability in reference to AI technology refers to the issue of – who should be liable in case of a faulty diagnosis or wrongful treatment recommendation by an AI system?
Repeated reliance on an AI system that is mostly accurate may make the patients and medical professionals rely on its results without questioning its limits. In such a scenario, where the AI technology yields faulty results – the question which arises is whether the liability would be imposed on the AI technology (which is infeasible, even though some scholars have suggested that medical AI should be given a legal status akin to personhood), its makers, or the medical professionals relying on the AI technology. The impact of faulty decisions made by AI was highlighted when IBM’s AI technology, Watson which was meant to provide treatment options to cancer patients prescribed the wrong drug that worsened a patient’s condition.
If doctors make a miscalculated judgement that results in medical negligence, they are held accountable for their actions under law; but the same is not true for technicians who design the AI systems. Another possible approach would be to institute product liability claims against the makers of the AI systems through which they can be held tortiously liable for any defects in the AI product. However, this is easier said than done as multiplicity of actors, designs and deployment makes it all the more challenging to establish the cause of action for a fault. Uncertainty in determining the liability has greater implications in the ground reality as it reflects upon the lack of grievance redressal.
5. Algorithmic Bias
A potential risk faced by the use of AI in healthcare, as evidence suggests, is that AI systems can incorporate and exhibit human social biases at scale. However, this is a result of the underlying data used to train the AI model, and not the algorithm itself. In order to perform well, AI systems require training on large amounts of data. Any AI model will be as biased as the dataset it is trained using – and most data sets exhibit the second-order effects of social or historical inequities. If the dataset used to train the AI model are skewed and fail to represent the diversity of the given human population, the results may reflect inherent human biases which may affect patient health and clinical outcomes.
For example, a study published by a student from the University of Michigan revealed that an AI system developed by Google to predict acute kidney injuries did not perform well on women. The accuracy of the AI system may have been compromised partly because the AI was not exposed to adequate female patient data.
Another example of algorithmic bias was documented where the Framingham Heart Study Cardiovascular Risk Score yielded favourable results for all Caucasian users, but not for the African-American patients. This naturally means that healthcare outcomes will be skewed, inequal, and inaccurate for different members of the demographic.
If the AI systems are able to learn from the biased data fed into it, it will also be able to create patterns of discrimination and exhibit similar behaviour. The only solution here is to use and create a database that is representative of all people in the said population.
6. Intellectual Property & Ownership
The historical human-centric nature of intellectual property rights often invokes the classic debate on ownership of the work generated using AI – is it owned by the person who developed the AI system or the AI system itself?
Usually, innovations are protected through a patent, however these may not apply to AI-innovations in the traditional sense as also held by a recent judgment[6] of the U.S. Court of Appeals where a patent application was filed naming an AI system called DABUS (Device for the Autonomous Bootstrapping of Unified Sentience) the inventor. The Federal Circuit court focused on the statutory interpretation of the word ‘inventor’ under the U.S. Patent Act and held that only a natural person can be an inventor and hence patents may only be granted to human beings. A similar view has been adopted by the U.K. Appeals court, the European Patent Office, the Federal Court of Australia, and the German Federal Patent Court all of which declined to grant a patent to DABUS. The German Federal Patent Court, however, stated[7] that it may consider an application which lists Thaler as the inventor, while recognizing DABUS’s contribution to the invention. The only exception to the patent application filed by DABUS has been in South Africa where the Companies and Intellectual Properties Commission (CIPC) granted a patent with DABUS as the inventor. However, unlike other countries, South Africa doesn’t have a substantive patent examination system which may explain the grant of patent to AI. Although no similar issue has arisen in India, Section 6 of the Indian Patents Act, 1970 states that a patent application may only be filed by a person, which makes it likely that the Indian Patent Office may deny a patent grant to AI.
Under copyright law, creators seek ownership of the original works created by them. Today, several AI systems can generate works that fall within the ambit of copyright law – such as literary works, visual works, music, etc., but should these AI-generated works be granted copyright? Although there are no specific laws that outright refuse grant of copyright to AI-generated works, the courts of different countries have made varied interpretations of the same. For example, the Copyright Office in the United States rejected[8] copyright protection to a painting generated by Stephen Thaler’s AI. In contrast to this, in September 2022, the US Copyright Office issued an unprecedented registration for a comic book “Zarya” developed with the assistance of a text-to-image AI tool. More recently however, the applicant of the copyright registration of the comic book received a notice[9] from the US Copyright Office stating that the registration may be cancelled, even though no ruling has been passed yet. These rulings are based on an old case law – Feist Publications v. Rural Telephone Company Inc.,[10] where the court said that copyright only seeks to protect the “fruits of intellectual labour” from the “creative power of the mind” which flows from the doctrine of sweat and brow under copyright law. In slight contrast with the above position, the Copyright Office in India granted copyright registration to ‘Suryast’, a painting generated using an AI as the co-creator. Even though the question of copyright ownership created by AI remains open, the examples of Suryast and Zarya hint at the courts willingness to grant copyright ownership to works that are AI-assisted and not completely AI-generated.
As it presently stands, AI-generated output cannot be protected under copyright, but can AI-generated output infringe the copyright of another person? According to Microsoft, which recently updated its terms of use alongside launching ChatGPT enabled Bing – any content submitted to Bing are deemed to be licensed to the company, and all outputs generated by it shall be owned by the company and may only be used for personal, non-commercial purposes. Although Microsoft can restrict the use of any output generated by Bing, it cannot claim copyright ownership over them as it wouldn’t pass the test of originality – which requires a substantial amount of intellectual creativity, skill, and judgement as also seen in Feist Publications above. What is more interesting is that even though Microsoft can require a license to the content submitted to Bing (which may or may not already be protected by copyright) in order to train the AI and provide services, the scope of the license is broad enough to cover other use cases such as sub-licensing, and public performances. This leaves room open for complete loss of control over material submitted by a user to Microsoft which defies the basic objective of copyright law – which is meant to ensure protection of the right of a creator over their creation.
LICENSING AND COMPETITION LAW CONCERNS
Chatbots like ChatGPT and Google Bard are trained on massive datasets of text scraped from the internet, often including ‘paywalled’ content that one would normally need to pay to read. This allows the bots to generate answers based on such articles, or even summarize or paraphrase them.
But when a chatbot, say the new ChatGPT + Bing, summarizes a protected article from a publisher such as the New York Times, giving the Bing user the information contained therein without needing to read and pay for the source article, it threatens the publishers’ business model, as they risk losing both website visitors and subscribers.
In a sign of what’s to come, German publishers are demanding royalties for the use of their content by chatbots, arguing that chatbots create competing content based on their articles, which may violate ancillary copyright and antitrust laws.
Ancillary copyright is a legal right, recently enacted in Germany in 2021, that grants publishers the exclusive right to make their publications publicly accessible and reproduce it for online use by information society services, such as search engines or news aggregators.
The German publishers also argue that there is potential for new age search engines like Bing to violate antitrust laws like the EU’s Digital Markets Act. For example, if Google favours its own AI content over competing publishers’ content in ranking, detail, and visibility, this may constitute self-preference and discrimination by a monopolist.
CONCLUSION
Some of the AI-related legal issues covered above are not specific to healthcare (e.g. IP), but others are. In particular, Big Tech’s insatiable need for large data sets to train AI models on has arguably generated greater friction in the field of healthcare, due to the sensitivity of the data involved, informed consent issues and so on. Likewise, questions of liability are more pressing where AI is diagnosing or treating patients.
In Part 3 of our series on ‘AI in Healthcare’ we will examine how different nations are looking to regulate AI today, and the associated implications for healthcare AI applications.
