Japan-based Kyorinsha is one of the largest agencies in Japan providing editorial and publishing services. Their editors have been using the UNSILO AI-based technical checks in production for several months. Now the tool has been assimilated into their operational workflow, via the Scholar One integration, Kyorinsha carried out an evaluation to compare manual checks with checks aided by AI. What were the results? 

Background 

It’s important to remember that the process of creating an academic article involves several hands. The author writes the manuscript, but then an editor at a publishing house will appraise it, usually more than once, and will carry out a number of checks on it. Those checks may include, for example: 

• Is this article well-enough prepared to be sent out for peer review? 
• Does the article meet the house style for the journal in which it will be published? 

• Does the writing meet accepted criteria for accurate, ethical, and replicable research?  

Any action from the above checks may involve the author being contacted again after the initial submission. An activity of this kind will, comprise both elapsed time and actual time spent evaluating the submission workflow. It is important to be clear what we are measuring and to keep the two measures distinct.  

Methodology 

Commented Kyorinsha: “The duration of time we measured was between when we first opened the paper’s pdf file and when our checklist was completed. We did not count the time to prepare a notifying email to the author about the parts where needed to be corrected. We focused solely on the actual checking process.  
“Our checklist” means that we have a checklist paper that our staff use. The list varies from journal to journal, and different staff may use different checklists.  

Results 
We found that using the UNSILO Technical Checks across ten articles resulted in reducing the time required to check the manuscript in all cases ranging from 2% to 51.9%, with an average (mean) of 26.7% reduction in time required to check the manuscript compared with a human-only check. The participant in the test had only about one month of experience in checking manuscripts. She did not see the machine-based checks in advance of carrying out her human edit (hence avoiding any bias from seeing what the machine identified.  
 
Of course, measuring the time taken is only one metric; quality is another, equally important metric. After running the time checks, we compared the accuracy of the results which is shown in the attached file also. “Good” for each manuscript and section means that the particular part of the submitted manuscript was in accordance with the instructions for authors. “Bad” did not meet the instructions for authors. 

Based on the definitions above, we compared the results of our results. We found that the machine had errors on two checks: structured abstract and conflict of interest. The structured abstract check failed because the specific requirements for the journal under consideration were different to the standard structured abstract checks provided by the tool.  

Conclusion 

Following the trial, UNSILO now enables journal editors to configure the values for these fields on a journal-specific basis, so if, for example, a journal uses standard headings “methodology”, rather than the default heading “methodology”, the check can be adjusted for this.  

Currently, editors find they have to double check the manuscript manually to recheck results shown by the Technical Checks, to see if they are really accurate, and that is what may be causing the users the extra time, resulting in longer time. By configuring the checks on a journal-by-journal basis, the checks can become more automatic, and as a result, publishers will start to see both a reduction in the time taken to check and at the same time a better quality of submission.