Case study: combining AI tools with human expertise for manuscript evaluation

Example of a Conflict of Interest check, one of over 25 Technical Checks provided by UNSILO

Kyorinsha is an innovative Japanese service provider, working with over 400 medical associations and societies to aid researchers get their articles published. Their services range from expert editorial support, production and printing of research journals, proceedings, newsletters and other publications, to technical consultation, production and management of online journals and development of research databases and software.

There are two problems with this model. First, it is very expensive in terms of human labour to check each manuscript that has been submitted.

In addition, humans are not good at repeated, low-intensity operations, such as formatting checks. An automated approach can help in both of these areas, if implemented in the right way.

For example, to check an academic article for a conflict of interest statement requires a lot of time, as such statements can be placed anywhere in the article. A human reader has to skim through the article from start to finish to be sure they have identified any relevant statements, and often researchers may not be aware of the potential conflicts of interest revealed by their work. They may reveal a conflict of interest without necessarily using the words “conflict of interest”, for example, one recent manuscript included the following two statements:

There are no conflicts of interest relating to this paper.

This research and publication received funds from XYZ Pharmaceutical Corporation

Clearly, both statements are potentially relevant to determining whether a conflict of interest exists. Simply searching for the string “conflict of interest” would not reveal the second statement. Using UNSILO’s AI tools, both statements are reliably identified from the article and presented to the editor. This means, most importantly, that the machine does not make the decision about the conflict of interest: this is the responsibility of the human editor. We call the process “machine-aided intelligence”. It is similar to your car satnav reminding you of the speed limit for the road you are on: the machine advises, and the human makes an informed decision.

“In this way”, explains Tat Okada, Senior Manager of Kyorinsha, “we have the best of both worlds. The machine makes sure we have identified all the potential conflicts of interest, and our expert team of reviewers then use their judgement to decide if these should be reported as COIs. By using the UNSILO automated tools, we are able to carry out over 20 checks on every manuscript, with all decisions taken by our expert editors. It is a perfect combination of human and machine.”

Receive an email every time we publish a new blog post