Our first blog post of 2021 already covered the topic of "DiGA" (Digital Health Applications) and data collection. In this post, we'd like to delve deeper into the preparatory work, and therefore this time we'll focus on the evaluation concept. This concept, along with the study protocol for the DiGA study, must also be submitted with the application for inclusion in the reimbursement directory. This blog post explains what it entails, how best to create it, and what needs to be considered.

Abbreviations

BOB (higher federal authority)

BtB (Business-to-Business)

BtC (Business-to-Customer)

DiGA (digital health application)

MDR (medical device regulation; EU Regulation 2017/745)

Underlying regulations

Digital Healthcare Act (DVG)
Digital Health Applications Ordinance (DiGAV)
DiGA Guidelines
EU Regulation 2017/745 (MDR)
ISO 14155

1 Introduction

To be included in the reimbursement directory (DiGA directory) as a Digital Health Application (DiGA), various requirements must be met and the review process at the Federal Institute for Drugs and Medical Devices (BfArM) must be successfully completed. This includes, among other things, an evaluation concept and a subsequent clinical study if no study meeting the DiGA criteria has yet been conducted. The January blog post important information on the DiGA study.

This article deals with the evaluation concept for the positive healthcare effect of digital health applications (DiGA). The DiGA guidelines discuss this in more detail in section 4.5.2

„Furthermore, the manufacturer must submit with the application an evaluation concept developed according to generally accepted scientific standards, which adequately considers the results of the systematic data analysis. The study protocol of the proposed study should be part of the evaluation concept. The choice of outcomes and the study design of the selected comparison and the real-world healthcare setting must be justified. It must be demonstrated why and how the evidence for the targeted clinical outcome measures (pVE) is derived from the chosen evaluation concept. This concept must have been developed by a manufacturer-independent scientific institute.“

"The scientific evaluation concept to be submitted should, in accordance with § 15 DiGAV, take appropriate account of the results of the systematic data evaluation."

Excerpt from: Brönneke, Jan B. “DiGA VADEMECUM: What you need to know about digital health applications (German Edition).

The manufacturer is therefore not required to create this evaluation concept themselves, as the legislator mandates its creation by an independent scientific institute, both in the guidelines and already in the Digital Healthcare Act (DVG). Nevertheless, the manufacturer makes a significant contribution to its development, as the study design, including the endpoints to be documented for the positive healthcare effect, requires in-depth examination of this topic. This article will explain what this means in more detail. At the same time, it will demonstrate how the evaluation concept can be developed for a DiGA (digital health application) that has been on the market for some time, for a DiGA that is currently under development, or for one that has just been approved under the Medical Devices Directive (MDD).

2. DiGA Evaluation Concept

The evaluation concept is defined as follows in the DiGA regulations:

„If an application for trial approval is to be submitted, it must be accompanied by a scientific evaluation concept. This must be prepared by an independent institution to demonstrate the positive healthcare effect according to generally accepted scientific standards

(Source: DiGA Guidelines )

This includes, in particular, the following information on the planned study project in order to demonstrate the positive healthcare effect of DiGA:

• the specification of the trial period (maximum 12 months)
• systematic data analysis using the DiGA itself
• Description using the PICO framework in summary of the positive supply effect
• Specifying the patient group by providing the corresponding ICD codes
• Type of positive healthcare effects of DiGA: medical benefits and/or patient-relevant procedural and structural improvements
• Information on research design and results
• Information on the quality-assured application of DiGA and on exclusion criteria
• Information on the scientific and manufacturer-independent institute involved

For example, we structure our evaluation concept as follows:

Fig. 1: Content and structure of the evaluation concept

A key component of the scientific evaluation concept is systematic data analysis during the application of the digital health application (DiGA). Therefore, this data analysis must be conducted with the approved medical device. This is not a problem if the DiGA is already on the market and data has already been collected through its use. However, for products still under development or those that have just been approved, no data for such an analysis may yet be available. Such a data collection phase must therefore be planned after approval, before an application for provisional inclusion in the directory can be submitted.

Regardless, the manufacturer must consider the planned care pathway to be followed with the DiGA (Digital Health Application), for which the positive care effect is to be demonstrated. As part of the data collection for the evaluation concept, various endpoints should already be defined, which can then be checked for validity in this context.

The aim and purpose of the data analysis should be to define the endpoints of the DiGA study, based on which the positive care effect along the intended care pathway can be demonstrated. Therefore, a selection option from the following areas makes sense

• medical benefits
• patient-relevant procedural and structural improvements

But how should this data be collected so that it can then be evaluated for the evaluation concept?

2.1 Already approved DiGA

Many of the already listed digital health applications (DiGAs) are approved medical devices that were previously on the market. Therefore, it is possible to use a previously conducted study that meets the DiGA criteria to be immediately and definitively included in the directory. In this case, consultation with the Federal Institute for Drugs and Medical Devices (BfArM) is strongly recommended. This is especially important if there is uncertainty as to whether the data collected is sufficient and whether it is unclear whether it is heading in the right direction, and whether an application for provisional or final inclusion should be submitted.

If no study has yet been conducted, it is advisable in this case to retrospectively evaluate data that was collected during the application of the DiGA and is available to the manufacturer via the app.

Note: This is easily possible if a B2C relationship exists between the manufacturer and the user. If this relationship does not exist, for example, because the DiGA is not provided directly by the manufacturer but by a therapist (with whom a B2B relationship exists with the manufacturer), the data is not held by the manufacturer.

If the manufacturer has access to the data automatically collected by the digital health application (DiGA), this data can be analyzed anonymously. In this case, this is done via a so-called monitoring plan. The data collection is based on "real-world data," and because it is completely anonymous, it can be collected without involving an ethics committee or authority. The monitoring plan defines the parameters to be collected, which relate to the aforementioned aspects

• medical benefits
• patient-relevant procedural and structural improvements

You should refer to Real World Data. We also recommend our March article “Medical Devices and Real World Data as well as Real World Evidence” (Link: https://www.medxteam.de/index.php/medxteam-blog/15-medizinprodukte-und-real-world-data-sowie-real-world-evidence).

2.2 Not yet or only recently approved DiGAs

No data may yet have been collected for these products. In this case, the clinical evaluation is generally carried out using performance data (see also Article 61, Section 1 of the MDR), as clinical data can be waived for these Class I or IIa products in order to meet the essential safety and performance requirements.

Therefore, approval studies are not usually conducted. Guidelines for state-of-the-art chapters in clinical evaluation generally only exist for the underlying indications and alternative applications, as DiGAs tend to be innovative and are not yet comprehensively integrated into guidelines.

However, for the clinical evaluation, claims regarding clinical performance, safety and clinical benefit must already be defined in the clinical evaluation plan and then substantiated with data in the clinical evaluation report.

Note: This is precisely where the use of the interface to the DiGA topic "medical benefit" or patient-relevant structural and procedural improvements is recommended, because this data can then be used in the update of the clinical assessment after the DiGA study.

Furthermore, the DiGA process doesn't end with listing in the DiGA directory. Negotiations with the health insurance companies then begin.

Note: Therefore, when defining the intended purpose of the medical device and in the claims, it is best to include keywords relating to medical benefits or patient-relevant structural and procedural improvements, if possible. This facilitates later negotiations. It is also best to refer to a digital health application rather than "software."

Therefore, after the medical device has been approved, data collection for the evaluation concept must be planned.

Here too, it is advisable to collect the data gathered during the manufacturer's use of the DiGA (Digital Health Application). However, this collection is not retrospective, but prospective, looking towards the future.

However, it is also possible to retrospectively evaluate the data after a defined application period of the DiGA.

Here too, it is important that the data

• with the DiGA itself and
• anonymous

This data is collected. This works best in the B2C case. However, even in the B2B case, if the manufacturer doesn't have direct access to the app data, an observational study and collection of real-world data can be conducted. Essentially, only the anonymized provision of data in the B2B context needs to be ensured.

2.3 Summary

A key aspect of the evaluation concept is data on the chosen care pathway and the necessary evidence of its effectiveness

• medical benefit
• or patient-relevant procedural and structural improvements.

Therefore, parameters are defined in advance that will be evaluated accordingly after data collection. These should come from the following areas:

Medical benefits:

• Improvement in health status,
• Reduction of the duration of illness,
• Prolongation of survival or
• Improving quality of life

Patient-relevant structural and procedural improvements:

• in the context of the detection, monitoring, treatment or alleviation of diseases or the detection, treatment, alleviation or compensation of injuries or disabilities and
• geared towards supporting patients' health-related actions or integrating processes between patients and service providers and
• include in particular the areas of

1. coordination of treatment processes,
2. Alignment of treatment with guidelines and recognized standards,
3. adherence,
4. Facilitating access to care,
5. patient safety,
6. health literacy,
7. patient sovereignty,
8. Coping with illness-related difficulties in everyday life or
9. Reduction of therapy-related expenses and burdens on patients and their relatives.

3. What we can do for you

We function as a scientifically independent, manufacturer-independent institution (CRO). As such, we develop your evaluation concept and advise you, even in the early stages of development, on compliance with interface regulations regarding your medical device claims or the definition of its intended purpose. All of this is done with the DiGA requirements in mind, allowing you to kill two birds with one stone.

If your technical documentation is already complete, we will review potential DiGA endpoints, either based on your documentation or on the care pathway subsequently implemented with the DiGA, and make a sensible preliminary selection to ensure that the data analysis for your evaluation concept is focused and doesn't become unwieldy. Ultimately, we want clarity, not to be guessing.

4. How we can help you

Whether and, if so, which clinical trial needs to be conducted, under what conditions and according to which requirements, is clarified by us at medXteam during the pre-study phase: In three steps, we determine the right and cost-effective strategy for the clinical data collection required in your case. This also applies to your evaluation concept and your DiGA study!

Do you already have some initial questions?

You can get a free initial consultation here: free initial consultation