Draft analysis grid intended to be used by CNEDiMTS to contribute to its evaluation of medical devices embedding decision systems based on automatic learning processes ("Artificial Intelligence")
- Public Consultation -
logo.gif
HAS believes that a collective approach is necessary to elaborate an analysis grid, which will be used during its evaluations to understand the operation/functioning of medical devices algorithmic part. This is the subject of this public consultation.

The proposed analysis grid will not replace the clinical data required to document the regulatory evaluation criteria, but is an essential descriptive panel for the technologies involving automatic learning processes.

We thank you in advance for your collaboration.
Context of your response: *
Last name: *
First name: *
Email:
Phone:
Position: *
Type of organisation: *
Give details:
Name of the structure : *
Correspondent's last name: *
Correspondent's first name: *
Email:
Phone:
Position: *
These data are confidential and will only be used for the analysis of the results in the context of this consultation.
NB: The questionnaire must be completed in one go. No field is required, but the form cannot be saved before validation.
Before completing this questionnaire, please read the instructions that can be downloaded here. Have you read them?
In order to optimize your contribution, it is essential to read the instructions (which can be downloaded here).
Purpose
1. Specify the benefit of the information that will be provided by the system based on automatic learning processes.

For example, is the aim to:
* help the patient to adjust the dosage of his/her treatment?
* help the doctor to make a diagnosis or select the examinations to be carried out?
* predict the occurrence of an event?
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
2. Describe the characteristics of the target population for the MD indication.

These may be:
* demographic
* medical, etc.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
Description of learning
The term learning here refers to the entire process.
It includes in particular the training, testing, validation and feedback phases.
3. Specify the type of learning.

Is it:
* continuous (self-learning algorithm)?
* initial (algorithm designed by self-learning then non-evolutive)?
* or incremental (algorithm whose structure and/or parameters are updated by learning)?
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
4. Describe the type of model used.

Is this automatic learning process:
* supervised?
* semi-supervised?
* unsupervised?
* by reinforcement?
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
5. Describe the type of algorithm used.

* classification
* regression
* clustering
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
6. Describe the model selection method.

For example, based on models tested, or due to the priority given to the explicability of the method.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
7. Describe the various learning and control phases.

Specifically indicate those based on learning from individual or collective data.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
8. Describe the re-training strategies.

For example, the frequency of re-training, the variables involved and the data inclusion period.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
9. Where applicable, state in which cases a human being is involved in the re-training process.

For example, in the case of active learning, specify the frequency and qualification of the person involved.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
Description of input data involved in initial learning or re-learning
Description of the population samples used to develop the model
10. Specify the characteristics of each sample.

Expected: their function, size and composition. Included variables must be cited. The manner in which rare events are taken into account must be described.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
11. Specify their terms of incorporation or, where applicable, the separation or segmentation methodology.

For example, specify how to separate training, test and validation datasets.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
12. Justify the representativeness of the samples used for the algorithm learning in relation to the samples to which this algorithm will be exposed once deployed.

A justification of the representativeness criteria is expected.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
Description of variables
13. Specify the characteristics of the variables.

* variable type
* distribution
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
14. Indicate the method of acquisition of the variables and their origin during the learning process.

For example, was a variable entered by a patient? Does it come from a sensor? Was it generated from virtual patient models?
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
Description of the processing of these data prior to use during learning
15. Describe the tests performed on the data.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
16. Describe the transformations applied to the data.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
17. Describe the missing data management process.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
18. Explain the outlier detection methods and describe their management.

In particular, how are the outliers of the least likely values discriminated
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
Description of input data involved in the decision
19. Indicate the method of acquisition of the variables and their origin.

For example, was a variable entered by a patient? Does it come from a sensor?
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
20. Specify the characteristics of the variables.

* type
* distribution
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
Performance
21. Describe and justify the choice of the metric, i.e. the method used to measure performance.

For example: RMSE, AUC, F1-score, etc.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
22. Describe the potential impacts of the processes performed.

For example, in the event of class rebalancing.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
23. Characterise over- and under-learning.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
24. Describe the methods used to manage over- and under-learning.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
Approval
25. Describe the validation methods.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
26. Report performance on different datasets.

For example, the results of the test and validation bases, according to the distribution used.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
System resilience
27. Describe the mechanisms implemented to measure model drift.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
28. Specify the chosen thresholds (limit values, maximum error rate, etc.)
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
29. Specify whether there is an anomaly detection system for the input data involved in the decision.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
30. Describe the potential impacts of these input data.

For example:
* In the event of non-correction of outliers
* If declarative values are used
* Due to the level of uncertainty associated with the input data (physiological, environmental data, etc.)
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
31. Specify the measures implemented in the event of model or input data drift.

For example: degraded mode, substitution of the learning algorithm by an expert system, etc.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
32. Describe the situations identified as likely to impair system operation.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
Explicability / interpretability
33. Specify whether the algorithm could benefit from an explicability/interpretability technique.

In particular, for self-learning algorithms, is an explicability technique applicable to allow the clinician or patient to understand the main factors that led to the decision made or proposed?
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
34. Indicate the explicability/interpretability elements provided.

For example, specify whether the explicability of decision making is recorded for ex post analysis by experts in the event of model drift.
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
35. Identify influential parameters.

For algorithms with initial or incremental learning, are these parameters identified and consistent with scientific knowledge?
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
36. Specify whether the system's mode of action is consistent.

For example, is the decision-making process consistent with professional recommendations, where they exist?
Is this wording explicit?
Comments:
To understand or evaluate the part of the medical device relying on machine learning, is this information...?
Overall, what do you think of collecting this information in addition to the other elements related to the description of the MD and the clinical results collected? Indeed, these are already the subject of a specific collection through the existing frames of applications for reimbursement filed by manufacturers.
1 (This is a waste of time)..... 5 (Do not know) .... 9 (This is essential)
If you wish to share more general comments on this approach or on the elements sought, you can do so in this field: