Demonstrating the value of digital health solutions with real-world evidence

Prova Health, in collaboration with Unity Insights, conducted an evaluation of Accurx's Self-Book product. This article shares some of the lessons we learned in the process.

Key messages:

Collaboration between innovators, end-users, and evaluation partners is essential for comprehensively evaluating digital technologies.
Mixed-method evaluations add significant value over utilising a single methodology.
Demonstrating impact in multiple areas is necessary to show the value of a solution to a health system. Real-world evaluations of innovations like Accurx Self-Book can capture the diverse value that a solution brings.

Introduction

Using Accurx's Self-Book product, practices can choose which patients require a face-to-face or telephone appointment and send these patients a unique link via text.

This project's success depended on the collaboration between Accurx, NHS GP surgeries, and the evaluation partners. Using a mixed methods approach, the evaluation produced valuable qualitative insights and a detailed quantitative health economics analysis. This methodology allowed the evaluation to showcase the impact of Self-Book in various areas, proving its benefits to the health system.

Collaboration with stakeholders is crucial to success

Producing robust results from an evaluation depends on the study's design. The correct methodology (or combinations of methods) depends mainly on the specific type of technology being evaluated and its real-world use cases. As an evaluation partner, developing a deep understanding of this context is crucial to identify valid outcomes and to produce a realistic plan for achieving these.

We undertook a co-design approach to the evaluation, working closely with the innovator and the partner sites to gain a deep understanding of the product and how it fits into the workflow of general practice. To ensure the best design possible, we consulted with in-house clinicians and subject matter experts early in the evaluation process to guide our methodology.

Accessing experts via the Prova Health Network and working directly with NHS staff through Accurx enabled this co-design approach. As an evaluation partner, we noted that NHS sites were keen to help with the evaluation, despite being under significant winter pressures and staff shortages. Staff were eager to give feedback to Accurx and contribute to the evaluation of Self-Book. This is a testament to the efforts that Accurx has made to foster its relationship with its end users.

Maintaining independence as part of the evaluation is crucial in producing results that minimise bias and provide real-world utility. Accurx made it clear early on in the project that this was a key priority for them.

Mixed-method evaluations enabled richer insights

Figure 1. Different methodologies can be used at different stages in the evaluation of digital health solutions.

A mixed methods approach is an evaluation design that combines qualitative and quantitative data collection and analysis methods. The quantitative part of this evaluation involved a detailed health economics assessment and provided a broad overview of the financial impact of the product. The qualitative part involved interviewing GP staff and surveying patients from 14 GP surgeries nationally looking after over 250,000 patients.

The qualitative data allowed for a more in-depth and nuanced exploration of Self-Book’s benefits and provided insights that quantitative data alone may not have revealed. Key themes that emerged from the interviews included insights into workflow improvements within practices, reduced stress levels in staff and improved efficiencies within practices.

The qualitative data also allowed corroboration and depth to be given to the quantitative insights, for example, the reduced dependence on phone lines. The evaluation indicated that patients had a positive experience of using the product. Staff emphasised how it had improved morale during day-to-day work and resulted in improved operational efficiencies within practices. These benefits would have been lost in a purely quantitative evaluation.

This mixed methods approach provides a comprehensive understanding of the deployment of Accurx, as it captures the complexity and nuance of how the Self-Book service is being used in real-world settings.

Figure 2. Qualitative data can reveal nuanced insights that quantitative evaluation alone may not provide.

Solutions need to demonstrate value in multiple areas

Health economic impact is crucial evidence that an innovator needs to demonstrate to a health system or payer. However, it is equally important to show benefits over and above health economics, particularly in areas that will affect uptake and adoption. A solution with favourable health economics is only valuable if users adopt and use it. This evaluation demonstrated that Self-Book had significant economic benefits alongside improving practice efficiencies and staff morale.

Figure 3. Digital health solutions must show value in specific areas for key audiences.

Conclusion

The collaboration between Accurx, GP surgeries, and evaluation partners was instrumental in ensuring a transparent and comprehensive evaluation of Self-Book. This approach helped to obtain valuable feedback from end-users, ensuring that the evaluation was grounded in the real-world experiences of those using the product.

A crucial factor in the evaluation's success was the mixed methods approach used. The combination of quantitative and qualitative methods enabled the gathering of different data types, providing a more comprehensive understanding of the product's impact. This approach also helped triangulate the data, validate the findings and provide a complete understanding of Self-Book's impact.

Demonstrating impact in multiple areas is necessary to prove the value of a product. The evaluation of Self-Book identified several areas of benefit, including financial savings, improved operational efficiencies and positive user experience. These insights are all essential for decision-makers who must be convinced that a product is worth investing in.

For more information, read the full report of this mixed-methods evaluation.

Prova Health supports digital health innovators with evidence generation. To discuss how we can help with evidence generation for your digital solutions, email hello@provahealth.com

Dr Mehul Patel is a GP and Health Informatician. He has worked in Clinical Lead and Clinical Safety roles for multiple digital health companies. He holds an MSc in Health Informatics from UCL/University of Manchester. He has completed a Health Education England Fellowship in Informatics involving national-level informatics projects within the NHS.

Gianluca Fontana is Co-founder and Chief Executive Officer of Prova Health. He has overseen the delivery of high-profile research, education and consulting projects, including the NHS Digital Academy, the REACT Covid-19 prevalence study, the World Innovation Summit for Health and the Centre for Health Policy of Imperial College London. He started his career as a management consultant at McKinsey & Company.

Dr Saira Ghafur is Co-founder and Chief Medical Officer of Prova Health. She is an honorary consultant Respiratory Physician at St Mary’s Hospital, London, and a digital health expert who has published on topics such as cybersecurity, digital health adoption and reimbursement, data privacy and commercialising health data. She is Co-founder of mental health start-up Psyma and holds a MSc in Health Policy from Imperial. She was a Harkness Fellow in Health Policy and Practice in New York (2017).

Demonstrating the value of digital health solutions with real-world evidence