Standard

Department for Health and Social Care and NHS Digital: QCovid algorithm

Published 1 June 2022

Name	Tier and category	Description	Entry (please enter all the required information in this column)
Name	Tier 1 - Overview	Colloquial name used to identify the algorithmic tool.	바카라 사이트¢ COVID-19 Clinical Tool/ QCovid algorithms
Description	Tier 1 - Overview	Give a basic overview of the purpose of the algorithmic tool. Explain how you바카라 사이트™re using the algorithmic tool, including: 바카라 사이트¢ how your tool works 바카라 사이트¢ how your tool is incorporated into your decision making process Explain why you바카라 사이트™re using the algorithmic tool, including: 바카라 사이트¢ what problem you바카라 사이트™re aiming to solve using the tool, and how it바카라 사이트™s solving the problem 바카라 사이트¢ your justification or rationale for using the tool 바카라 사이트¢ how people can find out more about the tool or ask a question - including offline options and a contact email address of the responsible organisation, team or contact person	바카라 사이트¢ The aim of the clinical tool is to create a way of predicting how at-risk individuals might be from coronavirus. The tool was developed for use by clinicians in support of conversations with patients about personal risk. 바카라 사이트¢ The tool utilises algorithms developed by the University of Oxford, called QCOVID. These combine a number of factors such as age, sex, ethnicity, height and weight (to calculate BMI), and specific health conditions and treatments in order to estimate the combined risk of catching coronavirus and being hospitalised or catching coronavirus and dying. 바카라 사이트¢ More information is available at . 바카라 사이트¢ The original version of the QCOVID algorithms were also used as part of the Population Risk Assessment to add patients to the Shielded Patient List in February 2021. These patients were advised to shield at that time were provided support for doing so, and were prioritised for COVID-19 vaccination. The Shielded Patient List has since been closed.
URL of the website	Tier 1 - Overview	If available, provide the URL reference to a page with further information about the algorithmic tool and its use. This facilitates users searching more in-depth information about the practical use or technical details. This could, for instance, be a local government page, a link to a GitHub repository or a departmental landing page with additional information.	바카라 사이트¢ See 바카라 사이트¢ See
Contact email	Tier 1 - Overview	Provide the email address of the organisation, team or contact person for this entry.	risk.strat@nhs.net
1.1 Organisation/ department	Tier 2 - Owner and responsibility	Provide the full name of the organisation, department or public sector body that carries responsibility for use of the algorithmic tool. For example, 바카라 사이트˜Department for Transport바카라 사이트™.	바카라 사이트¢ Department for Health and Social Care 바카라 사이트¢ NHS Digital
1.2 Team	Tier 2 - Owner and responsibility	Provide the full name of the team that carries responsibility for use of the algorithmic tool.	바카라 사이트¢ Commissioner: Department for Health and Social Care 바카라 사이트“ Shielding / Enhanced Protection Programme 바카라 사이트¢ Provider: NHS Digital
1.3 Senior responsible owner	Tier 2 - Owner and responsibility	Provide the role title of the senior responsible owner for the algorithmic tool.	바카라 사이트¢ Senior Responsible Owner during delivery: Phil Harper (DHSC)
1.4 Supplier or developer of the algorithmic tool	Tier 2 - Owner and responsibility	Provide the name of any external organisation or person that has been contracted to develop the whole or parts of or the algorithmic tool.	바카라 사이트¢ BJSS Limited/Sparck 바카라 사이트¢ University of Oxford 바카라 사이트¢ Hippo 바카라 사이트¢ Kainos
1.5 External supplier identifier	Tier 2 - Owner and responsibility	If available, provide the Companies House number of the external organisation that has been contracted to develop the whole or parts of or the algorithmic tool. You can get a company바카라 사이트™s Companies House number by finding company information or using the Companies House API	바카라 사이트¢ BJSS Limited - company number 02777575 바카라 사이트¢ Hippo - 09877239 바카라 사이트¢ Kainos 바카라 사이트“ company number 09047312
1.6 External supplier role	Tier 2 - Owner and responsibility	Give a short description of the role the external supplier assumed with regards to the development of the algorithmic tool.	바카라 사이트¢ The University of Oxford developed the algorithm used for the Population Risk Assessment, and those housed within the clinical tool for clinicians. 바카라 사이트¢ BJSS Limited developed the architecture, processes and tools for delivering the clinical tool. 바카라 사이트¢ Kainos provided resources to lead on requirements and design development for the tool as well as owning the approvals process. 바카라 사이트¢ Hippo provided User Researchers and produced reports and guidance that informed development.
1.7 Terms of access to data for external supplier	Tier 2 - Owner and responsibility	Details the terms of access to (government) data applied to the external supplier.	바카라 사이트¢ Client/supplier agreements to cover use of data. 바카라 사이트¢ The Statement of Work covers suppliers to NHS Digital who must be Baseline Personnel Security Standard (BPSS) confirmed. 바카라 사이트¢ The Oxford University team were given temporary access to central government data in order to develop the algorithm (for example, NIMS and HES) to combine with GP data already held. 바카라 사이트¢ 바카라 사이트 data was not used in the development of the wider tool that housed the algorithm from Oxford University. 바카라 사이트¢ 12 team members (for example Data Architects and Testers) also have Disclosure and Barring Service clearance. 바카라 사이트¢ .
2.1 Scope	Tier 2 - Description	Describe the purpose of the tool in terms of what it바카라 사이트™s been designed for and what it바카라 사이트™s not been designed for. This can include a list of potential purposes that the tool was not designed to fulfil but which could constitute possible common misconceptions in the future.	바카라 사이트¢ The original version of the QCOVID algorithms were used as part of the Population Risk Assessment to add patients to the Shielded Patient List in February 2021. These patients were advised to shield at that time, were provided support for doing so, and were prioritised for COVID-19 vaccination. 바카라 사이트¢ The tool has been designed to allow clinicians to input information about individuals in order to calculate risk of suffering adverse outcomes in the event that an individual contracts Covid. 바카라 사이트¢ The tool estimates the likelihood of a particular outcome. It does not guarantee outcomes. 바카라 사이트¢ The current version of the tool makes use of separate algorithms for vaccinated and unvaccinated patients, but the original research methodology was not designed to establish the effectiveness of vaccines. 바카라 사이트¢ The absolute risk estimates produced by the tool were ultimately derived from the study period used in the research. This predates new variants of the virus and other developments in population immunity, so the main purpose of the tool remains as a guide to personal relative risk.
2.2 Benefit	Tier 2 - Description	Describe the key benefits that the algorithmic tool is expected to deliver, and an expanded justification on why the tool is being used.	바카라 사이트¢ The key benefits of the tool when it was used in the Population Risk Assessment was to identify individuals at a high risk who should  be advised to shield and/or be placed into a high priority cohort for the vaccine rollout. The ultimate aim was to ensure protective interventions were targeted at the most vulnerable, to reduce mortality and protect the NHS. 바카라 사이트¢ The clinical tool for clinicians is intended to support individual conversations with patients about risk. Originally, the goal was to help patients understand the reasons for being asked to shield and, where relevant, help them do so. Since the end of shielding requirements, it is hoped that better-informed conversations about risk will have supported patients to make appropriate decisions about personal risk, either protecting them from adverse health outcomes or to some extent alleviating concerns about re-engaging with society. 바카라 사이트¢ Furthermore, the technology introduced a new capability, created at speed during the early stages of the pandemic, as it adds input to aid more targeted clinical interventions. 바카라 사이트¢ The underlying principles that drive the tool may in the future lend themselves to application outside of COVID-19, particularly given that the technology has been proven in a real world application.
2.3 Alternatives considered	Tier 2 - Description	Provide, where applicable, a list of non-algorithmic alternatives considered, or a description of how the decision process was conducted previously.	바카라 사이트¢ Before the development of QCOVID, the Shielded Patient List was developed by identifying patients for protection on the basis of whether or not they had any of a list of medical conditions. This did not take into account demographic factors or combinations of co-morbidities which QCOVID was then able to accommodate in its decision-making process. 바카라 사이트¢ England바카라 사이트™s Chief Medical Officer commissioned a predictive model for a data-driven approach to COVID-19 clinical risk assessment which resulted in the development of QCOVID.
2.4 Type of model	Tier 2 - Description	Indicate which types of methods or models the algorithm is using. For example, expert system, deep neural network and so on.	바카라 사이트¢ In essence, the tool creates a risk calculation based on scoring risk factors across a number of data fields pertaining to demographic, clinical and social patient information. This process is detailed below: 바카라 사이트¢ The algorithm creates a risk prediction model - QCovid - that provides a weighted, cumulative calculation of absolute risk using the variables associated with poor COVID-19 outcomes.  The research on which the model is based uses a Cox proportional hazards survival model using a range of health and demographic variables, excluding those not significant from final models used in decisions. . 바카라 사이트¢ The factors incorporated in the model include age, ethnicity, level of deprivation, obesity, whether someone lived in residential care or was homeless, and a range of existing medical conditions, such as cardiovascular disease, diabetes, respiratory disease and cancer. 바카라 사이트¢ For the latest clinical tool, separate versions of the QCOVID models were estimated for vaccinated and unvaccinated patients. When using the clinical tool, clinicians enter patients바카라 사이트™ vaccination status, and the tool uses this to determine which algorithm will be used in the calculations.
2.5 Frequency of usage	Tier 2 - Description	Provide information on how regularly the algorithmic tool is being used. For example the number of decisions made per month, the number of citizens interacting with the tool, and so on.	바카라 사이트¢ The Clinical Tool has been used within and outside of the NHS. In the period between 1st January 2022 and 31st March 2022, there were 2,180 completed assessments. Increased uptake was driven, in part, due to increased access, as technical restrictions on who could access the tool were removed in January 2022. 바카라 사이트¢ Assessment numbers often move with relative infection rate (e.g. higher infection rate leads to more usage of the tool).
2.6 Phase	Tier 2 - Description	Describe the phase in which of the following stages or phases the tool is currently situated: - idea - design - development - production - retired This field includes date and time stamps of creation and any updates.	Production/run and maintain
2.7 Maintenance	Tier 2 - Description	Give details on the maintenance schedule and frequency of any reviews. For example, specific details on when and how a person reviews or checks the automated decision.	바카라 사이트¢ Maintenance schedules for the algorithm are dictated by the emerging understanding of COVID and any new variants and therefore do not fit into a routine schedule. 바카라 사이트¢ Changes are processed through a RFC (request for change) procedure and enacted by the NHS Digital delivery team. 바카라 사이트¢ To date, maintenance has included the addition of extra data points to the algorithm (e.g. Vaccine status).
2.8 System architecture	Tier 2 - Description	If available, provide the URL reference to documentation about the system architecture. For example, a link to a GitHub repository image or additional documentation about the system architecture.	See graph.
3.1 Process integration	Tier 2 - Oversight	Explain how the algorithmic tool is integrated into the decision-making process and what influence the algorithmic tool has on the decision-making process. Give a more detailed and extensive description of the wider decision-making process into which the algorithmic tool is embedded.	바카라 사이트¢ The use of the tool does not override any clinical decision making but is a supporting device in the decision making process. 바카라 사이트¢ The tool amalgamates known factors about Covid at the time the tool was released. The update schedule for the tool isn바카라 사이트™t confirmed as it is dependent on scientific understanding of new variants as they emerge. 바카라 사이트¢ The tool promotes shared decision making with the patient and is an extra point of information to consider in the decision making process. 바카라 사이트¢ The tool helps with risk/benefit analysis around decisions (e.g. recommendation to shield or take other precautionary measures). 바카라 사이트¢ The tool is treated, in essence, as a clinical device (class 1) to aid decision making, not to make automated decisions by itself. 바카라 사이트¢ As of April 2022, there are no further planned updates to the tool.
3.2 Provided information	Tier 2 - Oversight	Describe how much and what information the algorithmic tool provides to the decision maker.	Example outputs from the tool are below: Risk of dying of COVID-19 following a positive PCR test Absolute risk: 0.0102% We바카라 사이트™d expect that 10 out of 100,000 people with the same characteristics and risk factors as this patient would die of COVID-19 following a positive PCR test. Relative risk: 1.1 We바카라 사이트™d expect that the patient is 1.1 times more likely to die of COVID-19 following a positive PCR test than someone who shares the same age and sex registered at birth as them, but none of their other characteristics and risk factors. Result generated on 25 February 2022 and valid for 90 days. Release R1 Version 2.0.5.9292
3.3 Human decisions	Tier 2 - Oversight	Describe the decisions that people take in the overall process, including human review options.	바카라 사이트¢ The tool is designed to help guide discussions but does not mandate the treatment or next steps agreed for the patient. 바카라 사이트¢ The tool only provides one strand of information as guidance. In order to make a decision on vaccination or treatment, many strands of information to determine those who are eligible would be required.
3.4 Required training	Tier 2 - Oversight	Describe the required training those deploying or using the algorithmic tool must undertake, if applicable; For example, the person responsible for the management of the tool had to complete data science training.	바카라 사이트¢ The tool is designed for use by clinicians who are reminded to look through clinical guidance before using the tool
3.5 Appeals and review	Tier 2 - Oversight	Provide details on the mechanisms that are in place for review or appeal of the decision available to the general public.	바카라 사이트¢ Patients will first review results with their clinician and for any further clarification can refer to .
4.1 Source data name	Tier 2 - Information on data	If applicable, provide the name of the datasets used.	바카라 사이트¢ Data is entered by the user and is not stored in a database. No external datasets are used/loaded. 바카라 사이트¢ Townsend deprivation index is looked up by postcode and fed into algorithm (provided by the team at Oxford University) 바카라 사이트¢ Current daily infection rate for England is taken from the UK Coronavirus dashboard.
4.2 Source data	Tier 2 - Information on data	Give an overview of the data used to train and run the algorithmic tool. It will also specify whether data is used for training, testing, or operating. It should include which categories of data - for example 바카라 사이트˜age바카라 사이트™ or 바카라 사이트˜address바카라 사이트™ - which were used to train the model and which are used as input data for making a prediction.	The team used the QResearch database (version 46) of 12 million patients with personal, clinical, and drug data that have been used for clinical and drug safety research. QResearch is linked to multiple datasets at the individual patient level. For this analysis, we used the National Immunisation Database of Covid-19 vaccinations to identify data on vaccine date and doses for all people vaccinated in England. For hospital admissions, we used the linked Hospital Episode Statistics dataset supplemented by the more regularly updated Secondary Users Service data. We also obtained and linked the following datasets: national data for mortality; SARS-CoV-2 infection; systemic anticancer treatment; radiotherapy treatment datasets; and national cancer registry data.
4.3 Source data URL	Tier 2 - Information on data	If available, provide a URL to the dataset.	n/a
4.4 Data collection	Tier 2 - Information on data	Gives information on the data collection process, including the original purpose of data collection.	바카라 사이트¢ No data is collected other than by the clinician at point of use 바카라 사이트¢ The algorithm undertakes a calculation on patient data and the result of that calculation is retained by postcode. Patient level data is not retained. 바카라 사이트¢ Data is not patient identifiable at the point of input.
4.5 Data sharing agreements	Tier 2 - Information on data	Provide further information on data sharing agreements in place.	바카라 사이트¢ Anonymised data is stored in the logs for reporting purposes (i.e. patient data input by clinician 바카라 사이트“ age, gender, risk factors, etc.).
4.6 Data access and storage	Tier 2 - Information on data	Provide details on who has or will have access to this data, how long it바카라 사이트™s stored, under what circumstances and by whom.	Management Information (MI) is retained for 90 days and then a summary version is retained for 8 years. This is only available to approved NHS Digital staff and access is controlled in line with the following governance standards: 바카라 사이트¢ SO 13485:2016: Medical devices - Quality management systems - Requirements for regulatory purposes. 바카라 사이트¢ DBC0129: Clinical Risk Management: its Application in the Manufacture of Health IT Systems. 바카라 사이트¢ DCB0160: Clinical Risk Management: its Application in the Deployment and Use of Health IT Systems 바카라 사이트¢ IEC 62304:2006 Amd 1:2015: Medical device software 바카라 사이트“ Software lifecycle processes.
5.1 Impact assessment name	Tier 2 - Risk mitigation and impact assessment	Provide the name and a short overview of the impact assessment conducted.	바카라 사이트¢ A clinical risk assessment in accordance with mandated Safety Standards DCB0129: Clinical Risk Management and MDD Medical Device Directive 93/32EEC.
5.2 Impact assessment description	Tier 2 - Risk mitigation and impact assessment	Give a description of the impact assessments conducted.	바카라 사이트¢ Risk Assessment conducted considered Risk Analysis, Risk Evaluation, Risk Control and implementation. 바카라 사이트¢ In accordance with DCB0129: a Safety Case Report and Hazard Log describing the hazard(s) scenario, understanding risk impact and suitable mitigation to alleviate risks through design, test, business process.
5.3 Impact assessment date	Tier 2 - Risk mitigation and impact assessment	Provide the date in which the impact assessment was conducted.	바카라 사이트¢ A risk assessment is an iterative process throughout the lifecycle of the tool. As new software components are introduced, fresh assessments are undertaken, and the Hazard Log is iteratively updated to align with the latest version of the software.
5.4 Impact assessment link	Tier 2 - Risk mitigation and impact assessment	If available, provide a link to the impact assessment.	Example outputs from the Hazard log (when the shielding programme was active) are included below: 바카라 사이트¢ Hazard: A person who should not be identified as meeting the threshold for higher risk using QCOVID are identified inaccurately 바카라 사이트¢ Cause: Information held in GP/hospital records is clinically inaccurate, out-of-date or not recorded, i.e. poor quality of coding. 바카라 사이트¢ Effect: Output from COVID-19 Predictive Risk Model misidentifies person as meeting the threshold for higher risk
5.5 Risk name	Tier 2 - Risk mitigation and impact assessment	Provide an overview of the common risks for the algorithmic tool.	바카라 사이트¢ As part of patient safety risk assessment, Hazardous scenarios are documented, yet haven바카라 사이트™t occurred as suitable mitigation is introduced and implemented to alleviate the risk.
5.6 Risk description	Tier 2 - Risk mitigation and impact assessment	Give a description of the risks identified.	Example potential scenarios include: 바카라 사이트¢ Hazard - COVID-19 Predictive Risk Model Tool not generated for a Patient when expected. 바카라 사이트¢ Hazard - Patient not provided with the most accurate risk score (either excessive or insufficient score) from the algorithm. 바카라 사이트¢ Patient does not have an output generated from the COVID-19 Predictive Risk Model
5.7 Risk mitigation	Tier 2 - Risk mitigation and impact assessment	Provide an overview of how the risks have been mitigated.	바카라 사이트¢ Defined within the Hazard Log. In summary, each hazard identified is reviewed, risk classified and- through design, test, business process or training- the appropriate mitigations are defined and implemented. 바카라 사이트¢ Hazard log inputs outlined in 5.4