Publications

Text Mining to Support Abstract Screening for Knowledge Syntheses: A Semi-Automated Workflow

Ba’ Pham and Jelena Jovanovic and Ebrahim Bagheri and Jesmin Antony and Huda Ashoor and Tam T. Nguyen and Patricia Rios and Reid C. Robson and Sonia M. Thomas and Jennifer Watt and Sharon E. Straus and Andrea C Tricco
Reference:
Ba’ Pham; Jelena Jovanovic; Ebrahim Bagheri; Jesmin Antony; Huda Ashoor; Tam T. Nguyen; Patricia Rios; Reid C. Robson; Sonia M. Thomas; Jennifer Watt; Sharon E. Straus and Andrea C Tricco Text Mining to Support Abstract Screening for Knowledge Syntheses: A Semi-Automated Workflow. In Systematic Reviews, 2021.
Links to Publication: [www][pdf]
Abstract:
Background: Current text mining tools supporting abstract screening in systematic reviews are not widely used, in part because they lack sensitivity and precision. We set out to develop an accessible, semi-automated “workflow” to conduct abstract screening for systematic reviews and other knowledge synthesis methods. Methods: We adopt widely recommended text-mining and machine-learning methods to 1) process title-abstracts into numerical training data; and 2) train a classification model to predict eligible abstracts. The predicted abstracts are screened by human reviewers for (“true”) eligibility, and the newly eligible abstracts used to identify (using near-neighbor methods) similar abstracts, which are also screened. These abstracts, as well as their eligibility results, are used to update the classification model, and the above steps are iterated until no new eligible abstracts are identified. The workflow was implemented in R and evaluated using a systematic review of insulin-formulations for type-1 diabetes (14,314 abstracts) and a scoping review of knowledge-synthesis methods (17,200 abstracts). Workflow performance was evaluated against the recommended practice of screening abstracts by 2 reviewers, independently. Standard measures were examined: sensitivity (inclusion of all truly eligible abstracts), specificity (exclusion of all truly ineligible abstracts), precision (inclusion of all truly eligible abstracts among all abstracts screened as eligible), F1-score (harmonic average of sensitivity and precision), and accuracy (correctly predicted eligible or ineligible abstracts). Workload reduction was measured as the hours the workflow saved, given only a subset of abstracts needed human screening. Results: With respect to the systematic and scoping reviews, the workflow attained 88%/89% sensitivity, 99%/99% specificity, 71%/72% precision, an F1-score of 79%/79%, 98%/97% accuracy, 63%/55% workload reduction, with 12%/11% fewer abstracts for full-text retrieval and screening, and 0/6 1.5% missed studies in the completed reviews. Conclusion: The workflow was a sensitive, precise, and efficient alternative to the recommended practice of screening abstracts with 2 reviewers. All eligible studies were identified in the first case, while 6 studies (1.5%) were missed in the second that would likely not impact the review’s conclusions. We have described the workflow in language accessible to reviewers with limited exposure to natural language processing and machine learning, and have made the code accessible to reviewers.
Bibtex Entry:
@article{slr2021, title = {Text Mining to Support Abstract Screening for Knowledge Syntheses: A Semi-Automated Workflow}, journal = {Systematic Reviews}, author = {Ba’ Pham and Jelena Jovanovic and Ebrahim Bagheri and Jesmin Antony and Huda Ashoor and Tam T. Nguyen and Patricia Rios and Reid C. Robson and Sonia M. Thomas and Jennifer Watt and Sharon E. Straus and Andrea C Tricco}, abstract = {Background: Current text mining tools supporting abstract screening in systematic reviews are not widely used, in part because they lack sensitivity and precision. We set out to develop an accessible, semi-automated “workflow” to conduct abstract screening for systematic reviews and other knowledge synthesis methods. Methods: We adopt widely recommended text-mining and machine-learning methods to 1) process title-abstracts into numerical training data; and 2) train a classification model to predict eligible abstracts. The predicted abstracts are screened by human reviewers for (“true”) eligibility, and the newly eligible abstracts used to identify (using near-neighbor methods) similar abstracts, which are also screened. These abstracts, as well as their eligibility results, are used to update the classification model, and the above steps are iterated until no new eligible abstracts are identified. The workflow was implemented in R and evaluated using a systematic review of insulin-formulations for type-1 diabetes (14,314 abstracts) and a scoping review of knowledge-synthesis methods (17,200 abstracts). Workflow performance was evaluated against the recommended practice of screening abstracts by 2 reviewers, independently. Standard measures were examined: sensitivity (inclusion of all truly eligible abstracts), specificity (exclusion of all truly ineligible abstracts), precision (inclusion of all truly eligible abstracts among all abstracts screened as eligible), F1-score (harmonic average of sensitivity and precision), and accuracy (correctly predicted eligible or ineligible abstracts). Workload reduction was measured as the hours the workflow saved, given only a subset of abstracts needed human screening. Results: With respect to the systematic and scoping reviews, the workflow attained 88%/89% sensitivity, 99%/99% specificity, 71%/72% precision, an F1-score of 79%/79%, 98%/97% accuracy, 63%/55% workload reduction, with 12%/11% fewer abstracts for full-text retrieval and screening, and 0/6 1.5% missed studies in the completed reviews. Conclusion: The workflow was a sensitive, precise, and efficient alternative to the recommended practice of screening abstracts with 2 reviewers. All eligible studies were identified in the first case, while 6 studies (1.5%) were missed in the second that would likely not impact the review’s conclusions. We have described the workflow in language accessible to reviewers with limited exposure to natural language processing and machine learning, and have made the code accessible to reviewers.}, year = {2021}, webpdf = {http://ls3.rnet.ryerson.ca/wiki/images/1/16/Slr2021.pdf}, url = {https://systematicreviewsjournal.biomedcentral.com/} }




Powered by WordPress