New opportunities and potential benefits of large language models (LLMs)

The natural language processing (NLP) field is advancing at an extraordinary pace in both academic performance and integration with the broader public [1]. Generative Pre-trained Transformer 4 (GPT-4) marks the latest in this field’s series of significant leaps [2,3]. OpenAI has released an extensive technical report (TR) in the style of a formal academic paper demonstrating exceptional results [2]. For example, GPT-4 scored in the top 10% of test takers across numerous bodies of knowledge in simulated examinations [2].

New horizons opened by LLMs have been extensively discussed, already with significant applications in the biomedical and pharmaceutical domains, such as protein design and drug discovery [4,5]. Many also envision LLMs will substantially impact beyond "AI for health sciences" and eventually transfer into "AI for healthcare practice." Namely, their potential use in reviewing clinical documentation, answering questions about electronic health record (EHR) data, augmenting clinical procedures, and helping radiologists to produce grounded radiology reports are all potential applications [6–8].

AI has greatly benefitted from and necessitates transparency

Like many other computer science publications, the OpenAI report is publicly available at arXiv. However, the capabilities and scale of GPT-4 have resulted in novel safety challenges that must be safeguarded against [9]. Like any machine learning algorithm, GPT-4 may have inherent biases that reflect biases in training data. In addition, the safety risks listed by the authors include the following:

1. Hallucinations, Harmful content, Harms of representation, allocation, and quality of service, Disinformation and influence operations, Proliferation of conventional and unconventional weapons, Privacy, Cybersecurity, Potential for risky emergent behaviors, Interactions with Other Systems, Economic impacts, Acceleration, and Overreliance.

Traditionally, the artificial intelligence (AI) field has greatly benefitted from the Open-first approach, from ImageNet to MNIST and MIMIC. Therefore, for GPT-4 to go against this grain, at present, where the data used for training and the necessary compute for training remain a trade secret, raises old concerns. Initial hesitance in releasing LLMs due to potential societal impact was common, yet it needs to be clarified if these obstacles have been overcome or what has changed in this period [10,11]. Especially when we review the model for bias, we are now speculative—as many researchers will not have access to the resources and datasets needed to train their own LLMs. Without systems and standards safeguarding against potential risks, self-audit can produce selective sharing and expose humans and patients to risks.

In the context of GPT-4, the OpenAI report is highly selective in disclosing foundational elements, such as training data, necessary to scrutinize critical aspects, including representativeness, robustness, and quality of the datasets. To ensure fairness and equity, it is essential to identify and address such biases and ensure that the model is trained on diverse and representative data. This self-selection is further demonstrated by the number of redactions released from the final draft, including areas such as toxic language. While this also indicates that the TR was meant as less of a rigorous technical evaluation and more of a demonstration of current performance, it highlights the potential for omitting areas deemed damaging. Indeed, review of the LaTex source code on initial drafts of the paper has highlighted the internal dynamics of this process, such as significant reduction of sections on “Toxic Content.”

The rising number of applications that have links to GPT-4 across education [12,13], finance [14,15], government [16], and healthcare [17] is exacerbating concerns. This is compounded by the proliferation of startups worldwide utilizing the technology and the growing visibility of the AI economy to consumers [18,19]. The TR proactively addresses this omission by referencing the competitive landscape and notes possible further disclosure later. While this may justify some of the nondisclosure, the gravity of potential harms in the interim that could result from lack of access to this information justifies an expectation of greater efforts toward more adequate stopgap measures.

This study reviews the GPT-4 report and summarises critical strengths and weaknesses

The importance of technical details in foundational models cannot be recognized, particularly when applying to high-risk settings. OpenAI states they could release such pending a legal process. Herein, we aim to set a blueprint for future technical descriptions of LLMs that maximize benefits and mitigate risks while respecting the competitive landscape. We highlight areas of strength in the current report and key areas of omission and set this in the broader context of LLMs. We focus on the implications to the healthcare domain due to our expertise and the importance of the systems evaluation before use in an extremely high-risk application. We frame our review of this highly consequential topic by emphasizing the importance of developing and evaluating appropriate uses for LLMs and mitigating the risks associated with applications [10,20].

This study aims to provide an impartial review of this technology, specifically this TR and its associated systems card. We conducted a peer review of this report using guidelines from previous publications and domain expertise spanning multiple fields, including engineering, computer science, law, and healthcare. Finally, we highlight critical questions raised and summarize a list of recommendations to address before this technology is deployed at large.

Technical report

For ease of reviewing and presentation of results, the document was downloaded from arXiv on March 22, 2023, converted to Microsoft Word in Adobe Acrobat, added line numbers, and converted back to PDF format. No other alterations were made to the published document. The full report with line numbers used can be found in S1 File.

Reviewer composition

A group with geographical and disciplinary diversity was necessary to deliver the most comprehensive perspective on this technology. Reviewers from across the globe with a wide range of professional training were sought to provide a global perspective on this innovation. The following areas were decided as critical to the success of the review: AI, NLP, public health, law, policy, social science, healthcare research, and bioethics. A full breakdown of the author’s expertise and background can be found in S2 File.

Review criteria

The authors reviewed the report individually before all feedback was assimilated into a final document. Reviewers were asked to consider review criteria in S3 File based on domain expertise and previous literature; however, reviewers were free to comment on any aspect of the manuscript [21,22]. Responses were then grouped into key categories, and all reviewers reviewed the final responses. No responses were omitted.

Strengths of the technical report

Limitations of the technical report

Concerns of bias and potential for LLMs to harm

While this technology can potentially revolutionize many fields, there is also concern about the potential for these models to spread misinformation or perpetuate bias at scale. This mirrors broader concerns for AI deployment in healthcare, where biases in electronic health records can be transmitted into downstream model performance. However, GPT-4 also brings substantial new concerns to bear in AI debates, such as the tendency to "hallucinate" (or the possibility to generate nonsensical or irrelevant responses), act in unpredictable ways, or struggle with processing complex and nuanced language, such as sarcasm, irony, or metaphor [32]. Furthermore, as language models become increasingly powerful, there is a growing concern about their potential misuses, such as generating hate speech, propaganda, or deep fakes [11].

A significant concern is that the sources used to generate GPT-4 are primarily refractory to scrutiny, leaving users to rely on an output, the basis for which they cannot verify. In addition, the lack of public versioning or documentation of back-end changes renders it difficult to consistently audit model behavior. GPT-4 must be designed and used responsibly to prevent such unethical practices. While it is often the case that it is difficult to foresee all of the risks of new technologies, in the case of GPT-4, some of the most severe risks are obvious, obviating any excuse for not seeking to mitigate them from the outset. We appreciate that Section 2, GPT-4 Observed Safety Challenges, in the GPT-4 System Card presents the harms identified by a large and somewhat diverse group of testers. We also applaud the measures taken towards safety and discouraging problematic use or outcomes, such as warnings against harmful behaviors contained in prompts (see, e.g., smoking) and disallowance of manifestly harmful prompts (e.g., how to make a bomb). These are valuable controls, but, as the authors acknowledge, the models that sound plausible and make fewer mistakes may be more dangerous due to the increased trust that develops with familiarity.

It must be acknowledged that other LLMs in high-risk settings such as healthcare have demonstrated a greater degree of openness in characterizing their development processes. Healthcare-specific LLMs, including MedPaLM from Google [33] and GatorTRON from researchers at the University of Florida and Nvidia [6], have offered extensive detail regarding their training datasets and evaluation processes. In the case of MedPaLM, the Google researchers further developed and validated a quality assurance task suite (MultiMedQA) upon which performance could be compared across models. While OpenAI and Microsoft have performed research subjecting GPT-4 to post hoc evaluation through the MultiMedQA task set and other specific evaluative benchmarks [20], the closed nature of the model architecture heavily restricts the possibility of effective external audit. While the authors give summary statistics and individual example outputs, no systematic dataset of prompts and responses is available. Particularly given ongoing proposals for the inclusion of GPT-4-related models in point-of-care electronic medical records, openness in this regard is important to ensuring safety and clinical buy-in [34].

Accountability, responsibility, and liability

GPT-4 raises several legal issues in addition to privacy, data protection, and intellectual property, all of which may arise in the production and deployment of the technology. Once the technology is made available to users, potential legal issues may arise due to its use. Generating output that misinforms regarding a high-risk activity or discourages more effective solutions to an urgent need may cause serious harm to a user who has relied on this information. Depending on the disclaimers that accompany consent to the use of GPT-4, there must be some accountability for the provision of foreseeably misleading information on which a user might rely to their detriment. With a technology of this scope, there are limits to caveat emptor (Let the buyer beware). For example, basic considerations underlying product liability require that a product actually be fit for purpose. Failing that, the seller is liable for the cost of the product and foreseeable damages [35].

In the case of GPT-4, a simple (or complex) disclaimer may be all that is needed to satisfy considerations of legal accountability. However, as an ethical matter, the foreseeable and, indeed, expected reliance of users on GPT-4 outputs would suggest that more is needed in the way of accountability. Because of the pioneering nature of several aspects of the technology, it would be reasonable to expect that such a technology would be accompanied by its own framework for accountability in which, at a minimum, it lays out precautions that the OpenAI has taken, precautions that the user should take, and a clear accounting of both direct and indirect risks that could arise in various contexts, including warnings that the technology should not be used for specific purposes [36]. Such warnings do not prevent undesirable uses, but as with any potentially harmful misuse of a product, the warning should serve to elevate caution and foster understanding of the limits of the technology. In doing so, the manufacturer takes on a degree of accountability that ultimately serves both the user and the manufacturer. Importantly, such a framework does not supplant a need for regulatory regard but rather acts as an important complement that can enhance the kind of trustworthiness that supports innovation. Additionally, issues pertaining to copyright infringement through the use of publicly available data and images may require address for any dignitary, proprietary, and economic harm. Notably, the proposed EU AI Act also takes aim at the risk to copyright interests brought about by ChatGPT and has amended the proposed regulation to require transparency specifically regarding any copyrighted material in the data used to develop the model [37]. Furthermore, energy consideration and impact on climate and environment need to be considered and responsibility allocated in sensible and ethically defensible ways.

In the context of generative AI, a framework for accountability also acknowledges a sense of responsibility for the operation and use of the product [38]. Such a framework could identify appropriate measures taken to mitigate risks at various stages in production. In devising schemes of responsibility, one approach looks to the party who is in the best position to mitigate harm (e.g., by design, development, or explicit use limitations), to assume the responsibility to do so. Thus, data brokers, developers, deployers, and providers may carry some degree of responsibility, and, under the revised EU AI Liability rules, exposure to civil liability cannot be avoided simply by the complexity of locating responsibility. The advent of this high-powered technology may also change how liability plays out where a person voluntarily but foreseeably relies on information to their detriment. While there has been little success in holding the providers of information liable for harm caused by such reliance (except in contexts such as healthcare where the point of providing the information is reliance), the particular configuration of sourced, combined, and integrated data in specific contexts, e.g., educational, healthcare, or transport, may result in a need to revisit responsibility and, eventually, liability. The most recently proposed amendments to the EU AI Liability Directive that effectively introduce a presumption of causality may signal an eventual shift in how proof of liability is viewed in the context of AI systems [39]. This may, in turn, inspire new schemes of responsibility. While GPT-4 may not fall under the Liability Directive, the fact of legislative recognition of the complexity of proving causality in the context of the use of AI devices could raise the specter of innovative interpretations of causality.

Regardless of possible legal configurations of allocation of responsibility, in putting GPT-4 on the market, OpenAI must engage with a level of accountability that reflects the awareness of potential use and misuse and meaningful engagement with how harmful sequelae can be minimized. Accountability and responsibility must be understood to extend beyond that which is currently legally mandated to ensure that the trustworthiness that is so essential to the successful uptake and integration of generative AI is not compromised. The proposed EU AI Act, the first comprehensive attempt at regulation of AI, has identified key considerations that should be noted beyond the EU regardless of jurisdiction. Ensuring appropriate disclosures both for proprietary as well as safety reasons, along with documentation of adequate testing and mitigation measures for foreseable risks, including to the environment, are among the necessary points of reference in developing a responsible approach to the provision and deployment of GPT-4.

Disruptive technology increases the demand for interdisciplinary reviews

Peer review is critical to the scientific process, ensuring research is rigorously evaluated before publication and safeguarding knowledge before implementation. This is especially important for reports on large AI models like GPT-4, which have the potential to impact a wide range of fields and applications. Given the interdisciplinary nature of AI and its applications, experts from different disciplines must conduct peer reviews for these models.

Firstly, large AI models like GPT-4 have applications in various engineering, data, and computer science fields. Each of these fields has its own set of specialized knowledge and techniques. Experts from each area must be involved in the peer review process to evaluate the model from all relevant perspectives. Experts should also realize that it is their duty to conduct such reviews. Secondly, AI models disrupt technical progress and societal norms, cultures, and processes. Given the potential for these models to impact people’s lives, experts from legal and social studies, ethics, anthropology, sociology, history, knowledge transfer, and health services research must be involved in the peer review process. This will ensure that the ethical and social implications of the model are considered alongside its technical capabilities. Finally, the interdisciplinary peer panel involved in this review will help to ensure that the TR receives a holistic, transdisciplinary assessment to recognize the strengths and identify potential issues with the TR that may have been missed if experts from a single field conducted the evaluation.

Setting up systems for high-risk applications

The translation gap for AI in healthcare is well acknowledged, and LLMs could fall into the same trap [40–42]. Particularly, if we plan to use closed models in this high-risk setting, even more so if we use models that consistently change and update over time. With the lack of version numbers and constant ongoing changes to the model, it is difficult to evaluate the models or for regulatory bodies to certify them properly. The field of MLOps has proposed several approaches for wider healthcare AI applications [43,44]; however, all will require or greatly benefit from increased transparency. The standardization of data collection and reporting should be foundational to its translation, starting in the development process. It is clear that while no group can fully understand all risks and biases or explore these in one document, there should be a structure that permits their exploration. What must be present is an acknowledgment that a systematic process to detect and improve these problems, even after deployment, must be developed; this is a continuous process. To safeguard against biases in clinical data, humans used in RLHF, or model design, an Open-first approach must be a default position.

Limitations of this peer review

We recognize that this TR was not produced for formal peer review but to provide insight into a novel technology with world-changing potential. This review recognizes GPT-4’s significant advance in the field and cannot criticize its potential to do significant good. However, its far-reaching implications warrant further discussion of essential areas of concern that involve an industry, not just GPT-4 or OpenAI alone. A diverse group conducted this review with respect to expertise and geography; however, in no uncertain terms is this complete. We aim for this review to be a springboard for further open dialogue on this revolutionary technology. Additionally, we encourage other groups to provide their unique insight to maximize the good this technology can do while safeguarding against its potential dangers.

Further, we seek to emphasize that peer review of a document for publication represents only a small part of what is needed for a thriving, open scientific engagement with such an important advance. Just as all outside of a few select researchers, our group is limited in our degree of access to the GPT-4 model, and we must rely primarily upon the materials produced by OpenAI researchers. Beyond mere expansion of these materials or prepublication peer review, we encourage a culture of openness, which will enable evaluation of LLMs and improvement of this technology by a broad research community with diversity befitting the diversity of GPT-4’s potential applications.