Evaluation
metrics for the classifiers and named entity recognition
Random Forest: Evaluation metrics permit to verify the efficiency of the classification algorithm (random forest) and of the algorithm for identifying therapies and drugs (NER). The following data were obtained by training of the random forest algorithm using 4,000 articles that were annotated with value 0 for those reporting treatment failure and value 1 for those reporting treatment success. This number corresponds to 2.95% of the 338,445 articles available for classification. The training parameters of the algorithm were defined as k-fold=10 for cross-validation and construct validation of 300 trees in the random forest algorithm. The value N=300 was obtained after analysis of the out-of-bag (OOB) error that stabilized at 300 trees in the ensemble. With the number of trees defined (N=300), the algorithm was executed and the metrics obtained are shown in Table 1.
Table 1. Metrics of the random forest classifier
| Classifier | Precision | Sensitivity | Specificity | Accuracy |
|---|---|---|---|---|
| Random Forest | 91.8% | 85.39% | 84.70% | 85.05% |
Precision indicates that 91.8% of the articles that were classified as reporting treatment “success” were indeed articles that reported success. Accuracy indicates how often the classifier is correct (85.05%).
NER: The data below were obtained from a training database of 10,000 articles annotated according to the specification of the Apache OpenNLP library.
Table 2. NER metric
| Classifier | Precision | Sensitivity | Specificity | Accuracy |
|---|---|---|---|---|
| (A) Max Entropy | 71.20% | 66.39% | 66.72% | 67.05% |
| (B) Max Entropy + regular expressions | 87.12% | 86.61% | 86.57% | 86.09% |
As can be seen in Table 2, two models were implemented. This was necessary after the low performance obtained with algorithm (A). Interestingly, the precision of 71.2% is the result of a large number of false positives, i.e., the model identifies correctly drugs and therapies, but a series of other words are recognized erroneously. Model (B) considerably improved the final result. The combination of the maximum entropy algorithm implemented in the Apache OpenNLP library with regular expressions improved the final result by 22.35%. Considering that more than 338,000 articles were submitted to recognition by the model, the improved algorithm using the combination of techniques permitted considerable elimination of noise in the final result.