The first blog post of 2021 was about the topic of “DiGA” and data collection. In this article we would like to go into the preparation task in more detail and that is why this time it is about the evaluation concept. This must also be submitted together with the test plan for the DiGA study with the application for inclusion in the reimbursement list. This blog post explains what it is all about, how it is best created and everything that needs to be taken into account.

Abbreviations

BOB (higher federal authority)

BtB (Business to Business)

BtC (Business to Customer)

DiGA (digital health application)

MDR (medical device regulation; EU regulation 2017/745)

Underlying regulations

Digital Care Act (DVG)
Digital Health Applications Ordinance (DiGAV)
DiGA Guide
EU Regulation 2017/745 (MDR)
ISO 14155

1 Introduction

In order to be included as a DiGA in the reimbursement directory (DiGA directory), various requirements must be met and the review process at the BfArM must be successfully completed. This includes, among other things, if a study that meets the DiGA criteria has not yet been carried out, an evaluation concept and a clinical study based on it. January blog post important information about the DiGA study

This article deals with the evaluation concept for the positive care effect of the DiGA. The DiGA guidelines go into this in more detail in Chapter 4.5.2:

„With the application, the manufacturer also submits an evaluation concept drawn up in accordance with generally recognized scientific standards that appropriately takes into account the results of the systematic data evaluation. The study protocol of the intended study should be part of the evaluation concept. The choice of outcomes and study design of the selected comparison and the reality of care must be justified. It must be explained why and how the evidence of the desired pVE emerges from the selected evaluation concept. This must have been created by a manufacturer-independent scientific institute .“

“The scientific evaluation concept to be presented should appropriately take into account the results of the systematic data evaluation in accordance with Section 15 DiGAV.”

Excerpt from: Brönneke, Jan B. “DiGA VADEMECUM: What you need to know about digital health applications (German Edition).

The manufacturer does not have to create this himself because the law requires it to be created in the guidelines but also in the DVG by an independent scientific institute. Nevertheless, it contributes a significant part to the creation, because the study concept, including the endpoints to be proven for the positive care effect, require in-depth study of this topic. This article would like to explain in more detail what this means. At the same time, it is shown how the evaluation concept can be used for a DiGA (software as a medical product) that has been on the market for a long time or for a e.g. B. a DiGA that is currently in the development process or one that has just been approved under the MDD can be created.

2. DiGA evaluation concept

In the DiGA regulations, the evaluation concept is defined as follows:

„If an application for testing is to be submitted, it must be accompanied by a scientific evaluation concept. This must be prepared by a manufacturer-independent institution to demonstrate the positive care effect according to generally recognized scientific standards.”

(Source: DiGA Guide )

This includes in particular the following information on the planned study project in order to demonstrate the positive care effect of the DiGA:

• an indication of the testing period (maximum 12 months)
• systematic data evaluation with the DiGA itself
• Description using the PICO scheme in the short version of the positive supply effect
• Specifying the patient group by specifying the corresponding ICD codes
• Type of positive care effects of DiGA: medical benefits and/or patient-relevant procedural and structural improvements
• Information on research design and results
• Information on the quality-assured application of the DiGA and exclusion criteria
• Information about the scientific and manufacturer-independent institute involved

For example, we structure our evaluation concept as follows:

Fig. 1: Content and structure of the evaluation concept

An essential part of the scientific evaluation concept is systematic data evaluation as part of the application of the DiGA. Therefore, this data evaluation must be carried out with the approved medical product. This is not a problem if DiGA is already on the market and data has already been collected through its application. However, for products that are still in development or have just been approved, there may not yet be any data available for a corresponding evaluation. Such a survey phase must therefore be planned following approval before an application for provisional inclusion in the register can be submitted.

Regardless of this, the manufacturer must think about the planned supply path to be followed with the DiGA, for which the positive supply effect is to be proven. As part of the data collection for the evaluation concept, various endpoints should already be established that can be checked for validity in this context.

The aim and purpose of the data evaluation should be to define the endpoints of the DiGA study, which can be used to demonstrate the positive care effect on the intended care path. It therefore makes sense to have a selection option from the areas

• medical benefits
• patient-relevant process and structural improvements

But how should this data be collected so that it can then be evaluated for the evaluation concept?

2.1 Already approved DiGA

Many of the DiGAs already listed are approved medical devices that were already on the market. This means that it is possible to use a study that has already been carried out and that meets the DiGA criteria in order to be immediately and definitively included in the directory. In this case, we strongly recommend that you seek advice from the BfArM. Especially when there is uncertainty as to whether the data collected is sufficient and it is therefore unclear whether it is going in the right direction and whether an application should be submitted for provisional or final inclusion.

If no study has yet been carried out, in this case it is advisable to retrospectively evaluate data that was collected as part of the application with the DiGA and is available from the manufacturer via the app.

Note: This is possible without any problems if there is a BtC relationship between the manufacturer and the user. Doesn't this exist because the DiGA, for example, is not directly from the manufacturer, but e.g. B. is provided by therapists (with whom there is a BtB relationship with the manufacturer), the data is not held by the manufacturer.

If the manufacturer has access to the data that is automatically collected by DiGA as part of the application, it can be evaluated anonymously. In this case, this is done via a so-called observation plan; the collection refers to “real world data” and since it is completely anonymous, it can be collected accordingly without involving an ethics committee or authority. The observation plan defines the parameters to be collected, which relate to the above-mentioned aspects

• medical benefits
• patient-relevant process and structural improvements

should refer. Our March article “Medical products and real world data as well as real world evidence” is also recommended for real world data (link: https://www.medxteam.de/index.php/medxteam-blog/15-medizinprodukte-und-real-world -data and real-world evidence).

2.2 DiGAs that have not yet been approved or have just been approved

No personal data may have been collected using these products. In this case, the clinical assessment is usually also carried out using performance data (see also Article 61 Section 1 of the MDR), since clinical data can be dispensed with for these Class I or IIa products in order to meet the basic safety and performance requirements .

This means that no approval studies are usually carried out. Guidelines for state-of-the-art chapters in clinical assessment are usually only available for the underlying indications and alternative applications, as DiGAs are more innovative in nature and are not yet comprehensively anchored in guidelines.

However, for the clinical evaluation, claims for clinical performance, safety and clinical benefit must already be defined in the clinical evaluation plan and then supported with data in the clinical evaluation report.

Note: This is exactly where it is recommended to use the interface on the DiGA topic of “medical benefits” or patient-relevant structural and procedural improvements, because this data can then be used to update the clinical assessment after the DiGA study.

In addition, the DiGA process does not end with the listing in the DiGA directory. Negotiations with the health insurance companies then begin.

to include keywords from medical benefits or patient-relevant structural and process improvements, if possible, when defining the intended purpose of the medical product and in the claims This makes later negotiations easier. It’s also best not to talk about “software” but rather about digital health applications.

Therefore, once the medical device has been approved, data collection for the evaluation concept must be planned.

Here too, it is advisable to collect the data collected with the DiGA as part of its use by the manufacturer. This survey is not retrospective, but prospective into the future.

However, the data can also be evaluated retrospectively after a defined period of application of the DiGA.

Here too, it is important that the data

• with the DiGA itself and
• anonymous

be collected. This also works best in the BtC case. But even in the BtB case, if the manufacturer does not have direct access to the app data, an observation study and collection of real world data can be carried out. Essentially, all that needs to be ensured here is anonymized provision in the BtB ratio.

2.3 Summary

A key aspect of the evaluation concept is data on the care path taken and the necessary evidence of this

• medical benefit
• or patient-relevant procedural and structural improvements.

Parameters are therefore defined in advance that should be evaluated accordingly after data collection. These should come from the following areas:

Medical Benefits:

• improving health status,
• shortening the duration of illness,
• Prolonging survival or
• Improving the quality of life

Patient-relevant structural and procedural improvements:

• in the context of the detection, monitoring, treatment or mitigation of diseases or the detection, treatment, mitigation or compensation of injuries or disabilities and
• aimed at supporting the health care activities of patients or integrating the processes between patients and service providers and
• include in particular the areas of

1. coordination of treatment processes,
2. Alignment of treatment with guidelines and recognized standards,
3. adherence,
4. Facilitating access to care,
5. patient safety,
6. health literacy,
7. patient sovereignty,
8. Coping with illness-related difficulties in everyday life or
9. Reduction of therapy-related expenses and burdens on patients and their relatives.

3. What we can do for you

We act as a scientific, manufacturer-independent institution (CRO). As such, we create your evaluation concept and advise you in an interface-compliant manner in your early development phase with regard to your claims for the medical product or determination of the intended purpose. All of this is done with DiGA requirements in mind, so you can kill two birds with one stone.

If your technical documentation is already available, we look at possible DiGA endpoints either based on your documentation or based on the care path subsequently taken with the DiGA and make a sensible preliminary selection so that the data analysis for your evaluation concept is targeted and does not get out of hand. After all, we want clarity and not to fish in clouds.

4. How we can help you

At medXteam we clarify whether and if so which clinical trial needs to be carried out under what conditions and according to what requirements during the pre-study phase: In 3 steps we determine the correct and cost-effective strategy in relation to the clinical trial required in your case Data collection. This also applies to your evaluation concept and your DiGA study!

Do you already have some initial questions?

You can get a free initial consultation here: free initial consultation  

As part of our Christmas special, we wrote in Part 2 about the application procedure and the approval process for clinical trials with CE-marked products and assumed that this and Article 74 apply to all PMCF studies with medical devices that have already been placed on the market. The virtual event by BfArM together with the Federal Ministry of Health and the working group of ethics committees shed light on this article as part of a virtual event as far as the regulation is concerned, at least in Germany.

Abbreviations

BOB (higher federal authority)

EK (Ethics Commission)

KP (clinical examination)

MDR (medical device regulation; EU regulation 2017/745)

MPG (Medical Devices Act)

MPAnpG-EU (Medical Devices Adaptation Act)

MPDG (Medical Devices Implementation Act)

BO (professional regulations for doctors)

Underlying regulations

EU Regulation 2017/745 (MDR)

MPEUAnpG (the Medical Devices EU Adaptation Act was passed as law by the Bundestag on May 25, 2020. This MPAnpG-EU describes the Medical Devices Implementation Act (MPDG) in Article 1)

MPDG (the MPDG will gradually replace the Medical Devices Act (MPG) from May 26, 2021 and will be legally binding for all manufacturers and operators of medical devices in Germany).

1 Introduction

The European Medical Device Regulation (MDR) has been in force since 2017 and its entry into force was scheduled for May 26, 2020. Due to Corona, this start of application and that of the Medical Devices Law Implementation Act (MPDG) was postponed to May 26, 2021. As is well known, this means that many legal requirements and practical framework conditions for the approval and conduct of clinical trials of medical devices are changing. The BfArM provided information about this on May 5, 2021 together with the Federal Ministry of Health and the working group of the ethics committees as part of a virtual event.

Assuming that Article 74 of the MDR :

“Clinical trials relating to products bearing the CE marking

1. If a clinical trial is carried out to further evaluate a device which already bears the CE marking in accordance with Article 20(1) within the scope of its intended purpose (hereinafter “post-market clinical trial”) and would be subjects within the framework of that trial subject to additional procedures to those carried out under normal conditions of use of the device, and if those additional procedures are invasive or burdensome, the sponsor shall inform the Member States concerned via the electronic system referred to in Article 73 at least 30 days before the start of the test. The sponsor shall submit the documents set out in Chapter II of Annex XV as part of the notification. For post-marketing clinical trials C1, Article 62(4)(b) to (k) and (m), Articles 75, 76 and 77 and Article 80(5) and (6) and the relevant provisions of Annex XV apply.
2. If a clinical trial designed to evaluate a device which already bears the CE marking in accordance with Article 20(1) is carried out outside its intended purpose, Articles 62 to 81 shall apply."

applies to all medical devices that bear the CE mark, the majority assumed that from May 26, 2021 an ethics vote is required for PMCF studies and that there is no longer any professional advice according to Section 15 of the Professional Code for Doctors (BO). This is not the case now.

2. Different types of PMCF studies

2.1 Definition of PMCF studies

There is no actual definition of the PMCF studies. Neither in the directive, MPG or in one of the regulations or MEDDEVs, nor in the MDR or MPDG. The MDR only says in Article 74 that such a clinical trial is called post-market clinical

"If a clinical trial is carried out to further evaluate a device which already bears the CE marking in accordance with Article 20(1) within the scope of its intended purpose (hereinafter 'post-market clinical trial') [...]."

A PMCF study is a clinical trial that is carried out on the CE-marked product and provides clinical data on the product as part of the post-market clinical follow-up (PMCF). Clinical follow-up is regulated in the MDR in Annex XIV Part B.

The clinical follow-up does not only include PMCF studies, but also other possible activities to collect clinical data on the product. An example is real-world data, which was described in the last blog post. Or register data and other activities. The following figure provides an overview of this and of the entire market surveillance (post-market surveillance) according to MDR according to Articles 83 - 85:

Figure 1. General methods and procedures PMS and PMCF (Source: Keene A. Leveraging Post-Market Surveillance and Post-Market Clinical Follow-Up Data to Support EU Medical Device Regulation (MDR) Compliance, Whitepaper)

2.2 PMCF studies within the scope of the intended purpose and without burdensome examinations

Previous regulation:

To date, these PMCF studies have been regulated in accordance with Section 23b MPG :

Ҥ 23b Exceptions to clinical trials

§§ 20 to 23a do not apply if a clinical test is carried out with medical devices that may bear the CE marking in accordance with §§ 6 and 10, unless this test has a different purpose for the medical device or it Additional invasive or other stressful examinations are carried out.”

Such a study previously fell under the category of other studies and was carried out outside the MPG. On the website of the Ethics Committee of the Bavarian State Medical Association, for example: E.g. like this:

Figure 2: Types of EC studies at BLÄK

In Baden-Württemberg, a “free application” is submitted for such a study:

Figure 3: Free application for such a study at LÄK BW

For PMCF studies within the intended purpose of the medical device, professional advice was required in accordance with Section 15 of the Professional Code for Doctors. To date, an application has been sent directly to the Ethics Committee. Some ethics committees required multiple paper copies and one copy as a CD-ROM. Others (e.g. Hesse, Bavaria) have a portal through which applications can be uploaded electronically. Then all that is required is a simple paper copy.

For these studies, in addition to certain study documents (test plan, patient information and consent form, questionnaires, etc.), the examiner's CV must be submitted. A qualification proven by the examiner having at least two years of experience in clinical trials with medical devices, as required for an ethics vote, is not checked here.

Regulation from the MDR’s entry into force

So far our acceptance and interpretation of Article 74 of the MDR has been; that this includes all PMCF studies and therefore an ethics vote would be required from May 26, 2021.

At the above-mentioned event, Article 74 for Germany was interpreted as follows:

• It only applies to clinical trials with CE-marked products that are carried out outside of their intended purpose.
• It also applies to clinical trials with CE-marked products if additional stressful examinations are carried out as part of these trials.

This means that the procedure described above for other studies and free applications (professional legal advice in accordance with Section 15 BO) still applies!

Figure 4: Article 74 of the MDR (Source: presentation slides, BfArM event, https://www.bfarm.de/DE/Service/Events/Dialogveranstaltungen/2021/210505-klinische_Pruefungen_von_MP.html)

Article 74 of the MDR thus separates the PMCF studies within the intended purpose and continues to view them as an exception to clinical trials, as in the MPG.

So nothing changes here except for designations in the test plan for MPG and, if applicable, Directive 92/43/EEC (change in MDR, as it is no longer valid). The process remains the same and manufacturers continue to have the opportunity to collect their clinical data as part of the PMCF study in a more straightforward manner.

2.3 PMCF studies with stressful examinations

Previous regulation:

To date, these PMCF studies have also been regulated in accordance with Section 23b MPG :

Ҥ 23b Exceptions to clinical trials

§§ 20 to 23a do not apply if a clinical test is carried out with medical devices that may bear the CE marking in accordance with §§ 6 and 10, unless this test has a different purpose for the medical device or it Additional invasive or other stressful examinations are carried out .”

In the case of additional incriminating investigations, Sections 20ff of the MPG and in this case in conjunction with Section 7 of the MPKPV with Section 1 Sentence 3 applied to such a study:

may bear the CE marking in accordance with Sections 6 and 10 of the Medical Devices Act , unless this testing has a different purpose for the medical device."

In this case, an application had to be submitted to the BfArM for exemption from the approval requirement via the Medical Device Information System (MPI, formerly DIMDI) and an application for a statement (vote) had to be submitted to the Ethics Committee via the MPI.

Regulation from the MDR’s entry into force

When the MDR comes into force, this procedure will be replaced by the new one regulated in Article 74 of the MDR:

• The sponsor informs the higher federal authority (BOB, in Germany: BfArM) about the MPI (in Germany) at least 30 days before the start of the examination.
• Articles 62(4)(b) to (k) and (m), Articles 75, 76 and 77 and Article 80(5) and (6) apply
• The relevant provisions of Annex XV also apply

This means that the BOB must be informed and an ethics opinion must be obtained from the ethics committee in accordance with Article 62 paragraph 4 letter b.

In addition, Chapter 4 with Sections 1 and 2 and in the latter with Subsection 1 apply in the MPDG with regard to the application, approval and respective deadlines. See also the blog post in the Christmas special part 2 .

2.4 PMCF studies outside of intended purpose

Previous regulation:

To date, these PMCF studies have also been regulated in accordance with Section 23b MPG :

Ҥ 23b Exceptions to clinical trials

§§ 20 to 23a do not apply if a clinical test is carried out with medical devices that may bear the CE marking in accordance with §§ 6 and 10, unless this test has a different purpose for the medical device or it Additional invasive or other stressful examinations are carried out.”

If the clinical trial with the CE-marked product previously referred to a new intended purpose (e.g. also to new indications), Section 20ff of the MPG also applied again to such a study. This means that a classic approval study was carried out in accordance with Sections 20ff of the MPG:

• Application via the MPI to BfArM and EK

Regulation from the MDR’s entry into force

With the entry into force of the MDR, Article 74 sentence 2 now applies:

"(2) If a clinical trial designed to evaluate a device which already bears the CE marking in accordance with Article 20(1) is carried out outside its intended purpose, Articles 62 to 81 shall apply."

This means that in this case too, a classic clinical trial must be carried out in accordance with Article 62ff of the MDR.

In addition, Chapter 4 with Sections 1 and 2 and in the latter with Subsection 1 apply in the MPDG with regard to the application, approval and respective deadlines. See also the blog post in the Christmas special part 1 :

Application for approval of a clinical trial in accordance with Article 70 paragraph 7 of the MDR:

Shortened procedure paragraph 7a

For medical devices without CE marking and for CE marked products if the clinical test is carried out outside the intended purpose (Class I and IIa non-invasive)

The application must contain the documents from Annex XV Chapter II of the MDR as well as the positive opinion of the EC.

Full application process paragraph 7b

For medical devices without CE marking and for CE marked products if the clinical test is carried out outside the intended purpose (Class IIa invasive, IIb* and III)

The application must contain the documents from Annex XV Chapter II of the MDR as well as the positive opinion of the EC.

* Exception in Germany, the shortened procedure normally also applies to products in Class IIb, only in Germany does the full application procedure apply here.

3. What we can do for you

We initially support you in finding the right data collection method within the framework of the PMCF. If a PMCF study is to be carried out, we will work with you to find the right way to implement it.

4. How we can help you

At medXteam we clarify whether and if so which clinical trial needs to be carried out under what conditions and according to what requirements during the pre-study phase: In 3 steps we determine the correct and cost-effective strategy in relation to the clinical trial required in your case Data collection.

Do you already have some initial questions?

You can get a free initial consultation here: free initial consultation  

The topic of “real world data” and “real world evidence” is now also gaining momentum for medical devices thanks to digital health applications (DiGA). What is this about? And to what extent can this topic be transferred to data collection for medical devices? When does it make sense?

Underlying regulations

Digital Care Act (DVG)
Digital Health Applications Ordinance (DiGAV)
DiGA Guide
EU Regulation 2017/745 (MDR)
ISO 14155

1. What are Real World Data (RWD) and Real World Evidence (RWE)?

“ Real world data refers to data about the use or potential benefits or risks of a drug obtained from sources other than traditional clinical trials .” This definition comes from Jacqueline Corrigan-Curay, JD, MD, director of the Office of Medical FDA Policy Centers. It shows that this topic has already found its way into the pharmaceutical industry and is already being used, particularly in the USA.

So what is “real world data”? This refers to data collection that relates to actual clinical routine. The evidence provided by this data from routine clinical practice is referred to as “real world evidence”.

2. Real World Data – Collection and Use

2.1 Real World Data for Pharmaceuticals

Real world data is usually collected as part of observational studies. These are regulated for pharmaceuticals. For example, BfArM published the following in December 2019

“Joint recommendations of the Federal Institute for Drugs and Medical Devices and the Paul Ehrlich Institute on application observations in accordance with Section 67 Paragraph 6 of the Medicines Act and on the notification of non-interventional safety tests in accordance with Section 63f of the Medicines Act”

published.

There are currently no such regulations for medical devices.

2.2 Data from routine clinical practice for medical devices

For DiGAs, an evaluation concept is required before the DiGA study or the application for inclusion in the DiGA directory. This should include a “ systematic data evaluation in addition to a systematic literature search and evaluation as well as the inclusion of our own systematically evaluated data that were obtained using the DiGA .”

Therefore, these are data from routine clinical use of DiGA.

Roche Diabetes also comments on this topic:

„Evaluating the benefits of digital health applications using real-world data: When evaluating the benefits of digital health applications, it should be taken into account that in the area of ​​pharmacological approval procedures the perspective is increasingly gaining ground that randomized, controlled studies represent an incomplete reflection of the reality of care. Randomized, controlled studies are suitable for establishing valid causality between an intervention and its effect. Real-world data (RWD) is seen as a potential source to gain insight into how certified medical devices and approved medications impact patient outcomes in real-world care. The European Medicines Agency (EMA) is therefore intensively discussing how RWD can be integrated in the future to solve complex issues ...“

(Source: Roche Diabetes Policy Portal , accessed on March 30, 2021)

The advancing digitalization of the healthcare system and the resulting increasing availability of digital data sets form the basis for more intensive use of RWD and RWE in the future. These developments open up potential opportunities for new players in the system: platforms for data exchange between service providers and institutions are necessary in order to generate and process RWE data (Meinert et al., 2018).

But it's not just the DVG that requires such data, the MDR also requires clinical follow-up (Post-Market Clinical Follow-up, PMCF). This is intended to continuously collect clinical data on the medical device, with the primary aim of checking whether its use in normal or routine care is effective for a specific patient or user. This data must therefore well reflect routine everyday life and routine care.

In Annex IXV of the MDR, the first sentence of Part B states:
“ Post-market clinical follow-up shall be understood as an ongoing process to update the clinical assessment in accordance with Article 61 and Part A of this Annex and shall be reflected in the manufacturer's surveillance plan before being placed on the market. During post-market clinical follow-up, the manufacturer proactively collects and assesses clinical data resulting from the use in or on humans of a CE-marked product placed on the market or put into service in accordance with the relevant conformity assessment procedure for its intended purpose Body to confirm the safety and performance over the expected life of the product, to ensure the continued acceptability of the identified risks and to identify emerging risks on the basis of relevant evidence ."

Since the conditions in routine clinical practice are usually different from those in a randomized, controlled clinical trial, which takes place within a defined framework, randomized, controlled clinical trials (randomized controlled trials, RCTs) are only partially suitable as PMCF studies. Their results can only be applied to actual routine applications to a limited extent. In addition, new risks and opportunities as well as off-label use cannot necessarily be identified.

2.3 Regulation of medical devices?

But how can such studies be classified from a regulatory perspective in relation to medical devices? Here we should first make an excursion into evidence-based medicine.

Figure 1: Evidence hierarchy according to evidence-based medicine (EbM), source: DiGA Vademecum)

First, a distinction is made between interventional and non-interventional studies, so-called observational studies. In interventional studies, if the use of the medical device is planned and carried out in a specific population and all the conditions for this are specified, it is referred to as an interventional study. Results here can always be traced back to the intervention. Interventional studies are therefore often comparative and always prospective. The intervention studies include the much-cited, much-demanded and probably much-feared randomized controlled trial (RCT), the “gold standard” in evidence-based medicine.

In observational studies, no planned intervention is carried out, which is why they are also called non-interventional studies. Here the application and the further course of the patient are observed and appropriate conclusions are drawn.

In observational studies, no intervention is carried out in accordance with a clinical test plan; treatment is carried out exclusively according to therapeutic practice. Observational studies can also be conducted both comparatively and non-comparatively; They can also be based on retrospective data. The most well-known non-interventional types with a control group include the cohort study and the case-control study. But registries also collect data from routine clinical practice and are then evaluated retrospectively.

Since the results of observational studies can be influenced by a number of biases and confounding factors, their internal validity is lower than that of intervention studies. In any case, when it comes to answering the question of the clinical effect of a specific intervention, its evidence is generally lower than that of an intervention study, since this is precisely the one that assesses internal validity. (Anvil, 2020)

Correlations can be determined through observation; However, a causal connection cannot be proven. However, compared to intervention studies, observational studies can usually be carried out more quickly and cost-effectively and have higher external validity than intervention studies. Without a defined framework for the application to be evaluated, the observational study has lower internal validity (and therefore lower significance with regard to effectiveness), but can therefore provide a better insight into the effectiveness in the context of the actual circumstances of everyday clinical practice .

The data collected in this way is “real world data” (RWD). The evidence obtained from it is referred to as “real world evidence” (RWE).

From a regulatory perspective, the medical device can only be used in routine clinical practice if it has a CE mark. The observational study is not based on a clinical test plan, but rather an observation plan. Therefore, Article 74 of the MDR does not apply (§ 74 is the basis for post-marketing clinical trials, for which the documents required in Annex XV Chapter II must still be drawn up, e.g. the trial plan).

Until now, observational studies were regulated via Section 23b MPG (exceptions to clinical trials) and professional advice under Section 15 of the Professional Code for Doctors (BO). This paragraph is now no longer applicable with the MDR. In §82 (2) the MDR refers to the option of member states to regulate other clinical trials at local level. The German Medical Devices EU Adaptation Act – MPEUAnpG does this by defining “other clinical trials that already bear the CE mark” in Section 47. It is also clearly stated there that neither a report to the federal authority nor an approval vote from the ethics committee is necessary if the observation study meets the following two criteria:

• The participants are not exposed to any additional stress/therapies (to routine therapeutic treatment).
• the medical device is used within the scope of its intended purpose.

What remains is professional advice in accordance with Section 15 BO from the doctor who is carrying out the observation study with the CE-marked product in accordance with the observation plan.

3. What we can do for you

Since such data collection by RWD is no longer regulated as of May 26, 2021 and does not fall under the umbrella of the MDR, it offers another option for data collection in order to in turn support the P(ost) M(arket) C(linical) F( ollow-up) requirements of the MDR.

We not only support manufacturers in finding the right survey method, but can also assist with all aspects of conducting an RWD observational study.

4. How we can help you

At medXteam we clarify whether and if so which clinical trial needs to be carried out under what conditions and according to what requirements during the pre-study phase: In 3 steps we determine the correct and cost-effective strategy in relation to the clinical trial required in your case Data collection.

Do you already have some initial questions?

You can get a free initial consultation here: free initial consultation

Literature sources

Anvil (2020) Medical Research Study Types. URL: https://www.amboss.com/de/wissen/ Medical research study types (accessed on March 30, 2021)

Meinert E, Alturkistani A, Brindley D, Knight P, Wells G, Pennington N. The technological imperative for value-based health care. British Journal of Hospital Medicine. 2018;79(6):328-32

Keywords: sample size planning, clinical study, clinical trial

Secondary keywords: sample size, sample size calculation

1 Introduction

When planning a clinical trial, sample size planning plays an important role. This determines how many test subjects must be included in order to demonstrate a relevant effect - and thus ultimately the success or failure of a study. What considerations play a role here?

To demonstrate the effectiveness of any clinical trial, e.g. B. PMCF or in registration studies, hypotheses are tested based on a primary endpoint. A hypothesis to be proven (called alternative hypothesis) can e.g. B. the superiority of a product over a standard therapy. The confirmation or rejection of a hypothesis is assessed based on collected data and the results are then transferred to the population, i.e. to the entire target group. For this to be meaningful, there must be enough data from observations from the target group. If there are too few observations, actual treatment effects cannot be demonstrated and the study fails. On the other hand, a large sample size leads to high costs, is difficult to justify ethically, ties up resources and extends the duration of the study.

The sample size planning is used to determine the minimum number of patients or test subjects to be included in order to prove an actual effect. A number of preliminary considerations are crucial for this.

2. Reasons for case number planning

The aim of every confirmatory clinical trial is to statistically prove a hypothesis. If the sample size is too small, an actual difference between two treatment groups cannot be demonstrated. The result is a non-significant statistical test, even though effects actually exist.

On the other hand, data collection requires a lot of time, human resources are tied up and costs arise for each additional patient included. If too many patients are recruited, this also leads to even small, medically irrelevant effects being detected.

Sample number planning for a clinical trial ensures that:

1. An effect present in the target group is detected with the statistical test, i.e. the test delivers a significant result
2. If the statistical test does not show a significant result, a sufficient sample size ensures that there is no effect in the target group (population) with a sufficiently high degree of certainty.

The need for sample size planning in the planning phase of clinical trials is also required by law and is reviewed by the ethics committee. Calculating the sample size is an essential part of the clinical trial plan and the statistical analysis plan.

For prospective study designs, sample size planning before the start of the study is essential, but in pilot studies or retrospective studies it should also be considered in advance how high the minimum sample size must be.

Aspects of caseload planning

Doctors, investigators, statisticians and CRO work closely together when planning the number of cases. The starting point is always the primary endpoint and the hypothesis of the clinical study to be tested.

3. Selection of statistical test

On the one hand, the type of question is essential for selecting the appropriate statistical test. Depending on whether superiority or equivalence of a treatment is to be demonstrated, different testing procedures are required. The scale level of the primary target variable also plays a crucial role. Different methods are used for nominal characteristics (yes/no, success/no success) than for ordinal (e.g. Likert scale) or continuous characteristics (e.g. visual analogue scale (VAS), sum scores, etc.).

3.1 Effect size

The effect size indicates the relevant difference to be detected. Depending on the test method used, different measurements are used. The best-known effect size for continuous variables is Cohen's d, which indicates the difference between two independent groups in relation to the common variance.

For categorical endpoints, the effect size W is used, which is the root of the squared relative difference in proportions.

According to Cohen (1988), the following general rules of thumb apply:

Effect size ≈ 0.2: small effect

Effect size ≈ 0.5: medium effect

Effect size ≈ 0.8: large effect

To determine the effect size, preliminary information that is as precise as possible from the literature or our own pilot studies is required. The medically and practically relevant difference that can be proven is also taken into account. A mean blood pressure reduction of a few mmHg, i.e. a very small effect size, can be statistically proven with a sufficiently large sample size, but is practically irrelevant for patients and doctors.

3.2 Significance level of the statistical test

The significance level a must be determined in advance and written down in the study protocol and the statistical analysis plan (SAP). The significance level indicates the probability of obtaining a statistically significant test result if there is actually no effect in the target group. A further distinction is made as to whether the test is carried out on one or two sides. One-sided tests test superiority hypotheses. Two-sided questions that compare the difference between two therapies are common. The value a = 5% has been established as the significance level; if the question is one-sided, a = 2.5% is often used.

3.3 Power or might

The power of the study is also determined in the planning phase. This refers to the probability that a statistical test will prove the actual difference, i.e. deliver a significant p-value. The power of a study should therefore be as high as possible. Values ​​between 80% and 90% are common here. The higher the power of a study, the higher the resulting number of cases.

4. Example from our NOVUSTAT consulting practice

As part of a clinical trial, the improvement in quality of life, measured by the score of the “Physical Functioning” scale of the SF-36 questionnaire, should be demonstrated after 3 months of therapy. The scale ranges from 0 to 100 points. The measuring instrument is well documented, validated and there are numerous publications with this measuring instrument. From the standard value table of the Federal Health Survey ^[1] it can be seen that healthy people in the age range 40-70 years have a mean score of 80-90 with a standard deviation of around 20 score points. For the study population, this physical function at inclusion (before therapy) will be 50 score points (standard deviation 25 score points), as results of a pilot study have shown. After three months of therapy, the goal is to achieve an improvement in physical functioning by 30 score points, so that the average functional ability after therapy corresponds to that of healthy people of the same age. A low value of 0.2 is expected for the correlation between the first measurement before therapy and the second measurement after 3 months of therapy (and confirmed with the data from the pilot study) due to the time interval.

If you enter these values ​​into G*Power, a software for calculating the sample size, you get the following result:

Fig. 1 Calculating the effect size

Based on the information and prior information, you get an effect size of 0.949, i.e. around 1. This information is now needed to calculate the minimum sample size required to detect an effect of d = 0.949.

For a normally distributed characteristic, a two-tailed t-test for paired samples can be used to demonstrate this. With a 5% significance level and a power of 90%, at least 14 observations are required for detection (see Figure 2).

Fig. 2 Sample size calculation for a two-tailed t-test with paired samples.

Taking into account a drop-out rate of 10%, at least 1.1*14 = 15.4, i.e. 16 patients, must be recruited.

As part of a sensitivity analysis, we check how sensitively the number of cases reacts to deviations from the assumptions. On the one hand, the effect size can be varied within reasonable limits, and on the other hand, the sample size can also be carried out using a non-parametric alternative. Reducing the power results in a reduction in the number of cases required.

A graphical sensitivity analysis can be seen in Figure 3.

Fig. 3 Sensitivity analysis: number of cases depending on the effect size and power of the study

5. Sources/literature

• Case number planning in clinical trials
• Chow S, Shao J, Wang H. 2008. Sample Size Calculations in Clinical Research. 2nd Ed. Chapman & Hall/CRC Biostatistics Series.
• Bock J., Determining sample sizes for biological experiments and controlled clinical trials. Oldenbourg 1998

6. What we can do for you

Before the start of a clinical trial, sample size planning is an important part of the preparation. The sample number calculation ensures that the actual effect can be proven. Professional sample size planning ensures that the sample size remains as small as possible. The sample size planning is tailored to the respective test, taking into account the study design, the primary target variable, the hypothesis to be proven and the required security.  

That's why our study planning basically and always includes case number planning as a first step. The entire study concept is based on this. And thus further planning (e.g. How many test centers are needed? How long do I need to recruit? etc.) can build on this.

At this point we would like to thank our partner Novustat for the guest article, as we think that this topic in particular is often underestimated.

About the author: "Dr. Robert Grünwald has been self-employed with the statistical consultancy Novustat for 6 years and and his team primarily advises customers from the pharmaceutical, medical technology and industrial sectors on all questions relating to statistical evaluations."

Statistics consultancy Novustat

8. Outlook

In one of the next blog posts we will take up the topic of “study types” again and take a closer look at the approval study according to MDR Article 62.

9. How we can help you

At medXteam we clarify whether and if so which clinical trial needs to be carried out under what conditions and according to what requirements during the pre-study phase: In 3 steps we determine the correct and cost-effective strategy in relation to the clinical trial required in your case Data collection.

Do you already have some initial questions?

You can get a free initial consultation here: free initial consultation 

[1] https://www.thieme.de/statics/documents/thieme/final/de/documents/zw_das-gesundheitswesen/gesu-suppl_klein.pdf