Theoretical framework

Since we are studying the governance of collaborations between Canadian public health/health care organizations and private technology companies, we have drawn from a governance framework to inform the methodologies and analysis of our study. Specifically, we used Vayena and Blassime’s framework of systemic oversight [45–47].

Systemic oversight draws upon the extensive literature on adaptive governance, originally developed for application in the environmental sciences. Adaptive governance is a means of coordinating resource management within uncertainty-laden and rapidly changing socio-ecological systems [48,49]. Vayena and Blassime argue that health data, due to its sources, capabilities, and stakeholders, can be characterized as an “ecosystem”, presenting significant implications for its management [45]. The framework is aimed at promoting responsible innovation across biomedical research, patient care, and public health with three aims: 1) to draw attention to ethically controversial areas to address risks, 2) to direct innovation towards outcomes beneficial for society, and 3) to promote public trust and accountability [46]. In keeping with these objectives, systemic oversight: 1) leverages adaptive and flexible mechanisms to cope with uncertainty, 2) is both inclusive and reflexive to engage a wide range of stakeholders and consider how they may be impacted by a given solution, and 3) monitors for and is responsive to emerging risks and harms [45–47].

There are numerous conceptual strengths of this framework for our study. First, it recognizes the ecosystemic and highly interconnected nature of both the digital technologies and the institutions involved in COVID-19 response. Second, it is appropriate for understanding how organizations respond to the COVID-19 pandemic, which is inherently a situation in which actors must act under tremendous uncertainty, time pressures, and constant change (e.g., in the epidemiological and clinical dynamics of the disease, spillover impacts to other aspects of society such as supply chains, etc.). Adaptive governance has been used in similar areas such as managing and mitigating risk due to natural disasters [50]. Third, adaptive governance also aptly relates to the notion of resiliency as both draw from the environmental sciences [48]. Finally, the use of this framework lends itself towards a hypothesis that is grounded in what has been seen in adjacent literature: when the structure of an ecosystem is challenged by crisis, then the mechanisms of governance that were designed optimally to support the ecosystem are at the heart of the crisis and must adapt in ways for which they were not designed.

We opted to employ Blasimme and Vayena’s framework of systemic oversight over other frameworks such as Char’s pipeline [51] or Leslie’s process-based governance framework for AI workflows [52] due to its complementarity and specificity with this study (i.e., we are focused on the governance of a health response to a crisis rather than the ethics surrounding a response). We also focus on the use of digital technologies to meet this response rather than artificial intelligence tools, strictly speaking. We also drew from Gasser et al.’s navigation aid to guide our data collection through semi-structured interviews with key informants [20]. We structured our interviews in accordance with the aid’s conceptual phases of developing and deploying digital technologies: 1) Preparation, 2) Planning, 3) Risk Assessment, 4) Development, 5) Deployment and Evaluation.

Document review

Document analysis included documents that provided details on the structure and nature of the sampled partnerships. Documents were included if they included details of the partnership, leadership perspectives, outcome measures, regulatory landscape, or product features. The search for these documents was done through multiple Google searches and manual searches in sampled organization webpages by following links. This process was done independently by two individuals (VH, TS) while keeping track of search terms (Appendix A in S1 Text). Searching and data abstraction was done by two independent reviewers (VH and TS), with conflicts resolved by discussion. We performed two rounds of document review (summer 2021, winter 2022) to account for new updates regarding the partnerships or organizations involved during the study period.

Key informant interviews

An initial list of key informants was first identified through the document analysis. We then used snowball sampling to identify additional participants at the end of each interview. Participants were recruited by email and were sent a standardized consent form (Appendix B in S1 Text). We sought to recruit participants in the roles of organizational leadership, end-users, and project/product managers, among others. To promote participants speaking freely, we ensured that their identities would be kept anonymous throughout the course of the study and requested that managers not ask if employees participated in the study.

The semi-structured interview guide (Appendix C in S1 Text) served to explore a range of topics from Vayena and Blasimme’s framework of systemic oversight as well as Gasser and colleagues’ navigation aid for the ethical use of digital public health tools [20,45–47].

Using this semi-structured interview guide, we conducted six pilot interviews prior to interviewing key informants to refine the content of the interview guide as well as the length and structure of the interview. We piloted with senior members of the research team and colleagues in similar roles to our key informants (e.g., clinicians, data scientists, etc.), and thus, we did not include the pilot interviews in the analysis.

The consent form was emailed to participants ahead of time, and interviews were conducted over internet-based audio/video chat on Zoom. Interviews were an hour on average and consenting participants had the option to be audio/video recorded. Interviews were then transcribed verbatim using Zoom’s built-in, automated transcription feature and the adequacy of the transcriptions were assessed by VH and TS using recordings. Recordings were then deleted upon ensuring the adequacy of transcription.

Coding and data analysis

To abstract and synthesize data from key informant interviews and publicly available documents, data coding was done iteratively and through the qualitative method of abductive analysis [53]. We selected abductive analysis to move between the data and our a priori theories. The goal of the abductive analysis is to generate novel theoretical insights that reframe empirical findings in contrast to existing theories. The transcribed interviews and document analysis were described using the coding structure-specific in Appendix D in S1 Text. Additional coding domains were developed as needed during the analysis, and the key themes were identified as nodes within the qualitative software package.

As mentioned above, the primary framework used to guide analysis was Blasimme and Vayena’s framework of adaptive governance through systemic oversight because it highlights a range of recommendations that promote responsible big data innovation across biomedical research, patient care, and public health. It also emphasizes the importance of directing attention to ethically controversial areas to address risks while also promoting public trust and accountability. Throughout the analysis, we also allowed thematic content to evolve independently of Blasimme and Vayena’s framework.

All coding was done in duplicate by VH and TS with conflicts resolved by discussion for each document and transcript until consensus was reached. Data collection, organization, and analysis was done using the NVivo qualitative software (version 12, QSR International, Melbourne, Australia). Finally, the data from each case study was gathered and synthesized into a single brief case report to facilitate cross-case synthesis. The results of the cross-case synthesis are presented here. Recruitment and data analysis were done in an iterative process between July 2021 and February 2022.

Sample size adequacy

In keeping with case study methodology, our goal was not to speak to every single stakeholder in each case but rather to understand an issue, process, or phenomenon. We used information power as the criteria for stopping further recruitment in each case once we believed we had a clear and sufficient understanding of the phenomena in each case [54]. Due to our narrow aim (i.e., explore specific partnerships between private technology companies and public sector health organizations), specificity in experiences of our key informants (i.e., had direct involvement with partnerships during COVID-19), use of well-established theory (i.e., Blasimme and Vayena’s theory of adaptive governance), and richness of interview dialogue, we found that five to ten interviews was sufficient to gain an understanding of each case in combination with the documents obtained. We then applied these requirements across our three cases.

A health emergency ultimately creates tension with time and resources

COVID-19 has been all-consuming for public sector organizations, especially during surges when staff are pulled away from non-pandemic-related activities to maintain core functions.

“… you’re dealing with a pandemic that is pulling every and all possible resources towards it like a big giant star.” (Case 1—Interview -Sales)

The ever-changing nature of the pandemic also created an environment of uncertainty for the partnerships as the health organizations we studied had to respond on very short timelines to developments both in the dynamics of the pandemic itself (e.g., the emergence of new variants such as Delta) or in pandemic-related policies (e.g., changes in testing criteria). These immense demands either catalyzed partnerships with private technology companies to form de novo or for existing partnerships to expand into new areas in reaction to evolving needs, such as keeping track of which patients were COVID-positive in the hospital or who received COVID-19 vaccinations in the community.

“So one of the challenges was that patients who were positive were a needle in the haystack amidst all the negative test… [the solution we built] ensured that everyone was on the same page within an hour of the resulting positive.” (Case 2—Webinar—Chief Medical Information Officer)

These pressures were felt across the different stages of the partnership (Table 2). At the beginning of the partnership, many participants felt that the current procurement regulations for public organizations, including hospitals, were not sufficiently agile in response to the time pressures of the pandemic.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Mapping the barriers and enablers identified in our three cases of study to the recommendations in Gasser’ et al.’s navigation aid for the ethical development and use of digital tools during the COVID-19 pandemic.

(Gasser et al., Lancet Digital Health 2020).

https://doi.org/10.1371/journal.pdig.0000164.t002

“So if you’re in a crisis, I just don’t know that you have time for that kind of lengthy [procurement] process because, there are there are pretty significant consequences for a loss of time and, you know, time is non-renewable as a resource you don’t get it back—you can’t rewind the clock… And so that’s the question: do you want to spend that time in that process? I think there needs to be a more agile process.” (Case 3—Interview—Chief Executive Officer)

However, there was a dichotomy to the effect of the pandemic on public-private partnerships: the same time pressures that posed challenges for the partnership also served as a catalyst for them. Participants in the private sector across our three cases felt that the business value of their solutions was made much more apparent during the pandemic. For example, virtual care proliferated in March 2020 because providers needed the means to connect with patients when outpatient clinics were shut down. Moreover, the public sector organizations were able to move much faster during the pandemic to form partnerships partly because leadership had a “burning platform” to act, especially if inaction would also cause harm. Leadership buy-in was an enabler to removing “red tape” that would otherwise stall partnership formation.

“… that pressure of time, it cuts through really debating what’s real and what is like, not a real risk … we’ll sometimes spend months debating like an indemnity clause in a contract about like a potential catastrophic event that’s very unlikely to occur… We were able to move past what would have been months and months of discussion… We were able to make a lot of ground in an emergency situation because we felt that any potential risk was outweighed by the risk of losing lives to COVID-19.” (Case 1—Interview—Vice President #1)

Early and sustained alignment around the problem is core to success

We found that a clear understanding of a central problem was critical to the success of our studied partnerships. Participants noted that successful partnerships were the ones where effort was made by the vendor to deeply understand the client’s pain points during the sales cycle and longitudinally through the partnership.

“So each party I think really has to… figure out where their little Venn diagrams can overlap in the middle to the success of both… in a pandemic that same approach would be even more profoundly important, because speed is not on your side there. You gotta move quicker, faster, and so you better define that common space in the middle, much faster than you would in normal times.” (Case 1—Interview—Incubator Lead)

Once a partnership was underway, participants consistently stressed the importance of continuous transparency around motivations, deliverables, progress, and roadblocks. Transparent communications were viewed as a core enabler of trust that both parties were aligned towards solving the same problem.

“…having that open, honest, and clear communication will allow just, you know for that trust to be built.” (Case 2—Interview—Customer Success)

The success of the partnerships in our case study is also reflected in the establishment of governance structures that support early and continuous alignment through transparency and trust. Logic diagrams demonstrate the multi-level connections between partner organizations, both in healthcare (Fig 2) and in public health (Fig 3). In both cases, the client success role has high centrality within the governance structure and serves as a vital “bridge” between the partners.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Generalized partnership logic diagram between technology companies and hospitals.

https://doi.org/10.1371/journal.pdig.0000164.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Generalized partnership logic diagram between technology companies and public health organizations.

https://doi.org/10.1371/journal.pdig.0000164.g003

“It’s my role within the company to be the voice of the client so I need to understand what the client wants, their needs, their priorities and advocate for them within the company… We are basically at the cross-section of almost every single department, because we are the voice of the client and the client drives the product.” (Case 3—Interview—Customer Success)

Beyond customer success, participants also spoke to the importance of ensuring alignment between different levels of seniority across the partnership and being able to escalate issues appropriately. One participant called this the “zipper:”

“I zipper people in and I make them develop relationships… Those contact points determine success or failure… So as an example [if] you were the director level or PM, I’d have you and your other person report monthly to [tech company A CEO] and myself… And you live and die together…[you need] the zipper, the alignment of interest, the processes to cause reporting, and the honesty to escalate when something’s going wrong [to] deal with it at the right level, you can’t deal with a strategic issue at an operational level.” (Case 1—Interview—Chief of Staff)

The emergency context led to the triage and streamlining of governance processes

Table 3 shows the wide range of tools that were used to govern the partnerships (i.e., governance processes) across our three cases of study. Participants noted that the time and resource pressures of the pandemic led to shifts in certain governance processes, such as identifying goals and allocating tasks, from formal mechanisms (e.g., project charters) to more informal ones (e.g., rapid, huddle-style meetings).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Common government processes utilized across our three cases of study.

https://doi.org/10.1371/journal.pdig.0000164.t003

“… clearly identifying what the goals are and articulating what the accountabilities are, my sense is that in an emergency this happened, far more informally than someone literally drafting up a charter or terms of reference … [it would] be like, … “All right, we’re gonna have a huddle, we’re going to spend 30 minutes this way, you’re going to do this, we’re going to do this. All right, let’s go””. (Case 3—Interview—Chief Executive Officer)

While many governance processes were modified to be more efficient, participants also stressed certain processes that were vital and needed to be maintained. For example, stakeholder assessment could not be skipped in its entirety but leveraging the public organization’s ownership and deep understanding of their problem and context allowed for one private organization to identify and prioritize “directly-impacted” groups during an accelerated stakeholder assessment. Moreover, mechanisms around what Blasimme and Vayena term “monitoring” such as maintaining risk registries or keeping track of key performance indicators, could not be dropped altogether because they helped the partnership remain aligned and navigate emerging risks. This is especially true in the context of privacy or security issues.

“I’d say that getting approval and acknowledgment from stakeholders… is a step you just can’t skip… if the stakeholder groups asked you to do something, you should have acknowledgment and awareness at a senior team level that it’s being done.”(Case 1—Interview—Chief Security and Privacy Officer)

An unexpected finding was that participants did not prioritize publicly sharing details of the partnership. Leaders stressed that, especially in the “information overload” context of the pandemic, publicly-directed communications about a given partnership should be meaningful and prioritized around the key needs of stakeholders.

“…especially in the context of COVID for example, [it’s] not always the most appropriate thing to be necessarily like flooding a hospital’s Instagram account to try to market this type of program when like they’re trying to get vaccinations in arms.” (Case 1—Interview—Implementation Manager)

Social learning can offset some emergency pressures

Social learning, defined as learning through observing others [55], was identified by Blasimme and Vayena as a key enabler of governance by systemic oversight and was identified across all three study cases. For the hospitals, social learning occurred both informally (e.g., Chief Information Officers would reach out to each other for advice) and formally (e.g., the local university’s medical school co-ordinated city-wide COVID-19 response tables for hospital leadership that met as often as daily). For some participants, social learning was viewed as a bare minimum level of due diligence needed before embarking on a partnership with a vendor.

“As much as a crisis makes people make rapid decisions—don’t [do that]. Take the time to at least do a, you know, an environmental scan and say like, “hey, what is everybody else doing?”” (Case 1—Interview—Chief Medical Information Officer)

The public health organization we studied also had a small team dedicated to learning from jurisdictions worldwide, but it was less clear how to take these learnings and apply them to their own context compared to the hospitals.

“I think there was an effort to constantly learn from other countries… there’s a group whose only task is to look at other countries, what they’re doing, and how they’re performing as a result of what they’re doing… and that project has been ongoing for at least probably since the summer of 2020.” (Case 3—Interview—Senior Epidemiologist)

For the private technology companies, social learning took the form of press releases, when deemed appropriate, to signal key milestones (e.g., launching new products or services). Conversely, it also involved scanning for other offerings in the market to understand and crystallize their unique value proposition as a company. For one partnership, they noted that social learning was formalized by the nature of their membership in an innovation accelerator, where they delivered and heard presentations about successful and unsuccessful healthtech partnerships from other participants in the program.

Startups have a vital role to play in emergency response, but growth can introduce new challenges

Since the companies studied were startups, we found that they were highly attuned to their local contexts and could often be highly flexible and adaptable to create bespoke solutions for their partners. Several participants from the hospitals contrasted their previous experiences with major electronic medical record (EMR) companies based in the United States, positively reflecting on how their perspectives as end-users were much more valued and regularly informed product updates compared to the EMR companies.

“I think with the big vendors, the other thing is making sure that I’m not getting the short end of the stick where I’m paying a lot but actually not getting service… whereas I think those discussions are easier with a smaller company right because they just have more flexibility” (Case 2—Interview—Chief Information Officer)

All three startups studied scaled dramatically (i.e., doubled, tripled) during the pandemic due to the increased demand for their solutions. While on the one hand, the increased demand and revenue is positive and reinforce their value, aspects of pandemic-fueled “hypergrowth” were also challenging. For example, one dimension of growth was around staffing and building capacity to complete core tasks in response to increased demand, such as responding to customer support tickets or onboarding new users.

“[Virtual care company] experienced a 500% increase in usage and added over 200 healthcare providers to its platform in one month alone.” (Case 1—Marketing brochure)

Such growth often demands changes to governance processes to improve their quality or feasibility.

“So for example… deployments, before it was just [the] wild west, we have this hospital deploying whatever and we think about things off the top of our head… and mistakes happen… With our new process… when we have a hospital deployment, we have every single task created… okay these are things that we have to do, we can delete ones that are not applicable and that makes it much more repeatable and the quality is much better on a wider scale”.—(Case 2—Interview—Chief Executive Officer)

Given the urgent, and oftentimes bespoke, nature of client demands during the pandemic, the companies we studied also had to navigate how to maintain their core value proposition.

“Each client wants a slightly different version of what you’re trying to offer. And so you start to get this just like infinite list of things that you can build against but it only applies to one client at a time, and building like bespoke things at scale is just not possible.” (Case 3—Interview—Product Manager)

There’s no partnership without healthy, motivated teams

The intensity of the pandemic timelines and workloads universally created burnout and a high degree of turnover among teams in both the public and private sectors. Effective interpersonal skills and emotional intelligence, especially empathy, was viewed as an enabler of effective partnerships.

“I’ll typically relate back to you know I get it, I’m a healthcare professional myself, you know, things come up, emergencies come up… I think that that perspective allows me to be there in a more supportive way because I do understand that their environment can be very hectic and can be ever-changing.” (Case 2—Interview—Customer Support)

Participants also spoke to the need for visibility into governance processes. Leaders felt the onus to be active throughout the partnership and diagnose areas of potential misalignment quickly.

“I understood that in that forming stage, it required a tremendous amount of effort and time. And the stakes were high. So the reality is… I’d be talking to that CEO, three times a day. And I’m setting up a steering committee with the executives there. And I’d be monitoring it… And [I would] be there when it’s going wrong. People like to be there when it’s going right.” (Case 1—Interview–Chief of Staff)

Having key governance processes, such as discussions on ethical issues or company direction, being highly visible and inclusive helped to keep teams motivated and engaged, especially in virtual and thus potentially isolating or siloed work environments.

“It really speaks to you know being in that supportive environment where we can listen to everyone’s concerns…So the fact that we are able to bake in, you know, being considerate of a client’s time and everyone’s time but still take the due diligence in order to be able to say can we [navigate this challenge] differently and more responsibly, I’m very happy to be a part of something like that. (Case 3—Interview—Data Engineering)

Ultimately, one of the best remedies against burnout and promoters of effective team functioning was that participants on both the public and private side of the partnerships we studied found that their work during the pandemic was valuable and impactful.

“Working in healthcare as a non-clinician… you want to be able to draw that line between what you’re doing and what the patient impact is in as short a distance as possible. And I think what we’ve been able to do during the pandemic is as short a distance between those two points as you can get.” (Case 1—Interview—Sales)