Japan-based Kyorinsha is one of the largest agencies in Japan providing editorial and publishing services. Their editors have been using the UNSILO AI-based technical checks in production for several months. Now the tool has been assimilated into their operational workflow, via their integrated manuscript tracking system software, Kyorinsha carried out an evaluation to compare manual checks with checks aided by AI. What were the results? 

Background 

The process of creating an academic article involves several hands. The author writes the manuscript, but then an editor at a publishing house will appraise it, usually more than once, and will carry out a number of checks on it. Those checks may include, for example: 

• Is this article well-enough prepared to be sent out for peer review? 
• Does the article meet the house style for the journal in which it will be published? 

• Does the writing meet accepted criteria for accurate, ethical, and replicable research?  

Any action from the above checks may involve the author being contacted again after the initial submission, and any delay of this kind impacts significantly on the time to publication, so it important that authors are only contacted with a clear statement of what needs to be fixed. The in-house appraisal time comprises both elapsed time and actual time spent evaluating the submission workflow. It is important to be clear what we are measuring and to keep the two measures distinct.  

Methodology 

Commented Kyorinsha: “The duration of time we measured was between when we first opened the paper’s pdf file and when our checklist was completed. We did not count the time to prepare a notifying email to the author about the parts where needed to be corrected. We focused solely on the actual checking process.  “Our checklist” means that we have a checklist paper that our staff use. The list varies from journal to journal, and different staff may use different checklists.  

Results 
We found that using the UNSILO Technical Checks across ten articles resulted in reducing the time required to check the manuscript in all cases ranging from 2% to 51.9%, with an average (mean) of 26.7% reduction in time required to check the manuscript compared with a human-only check. The participant in the test had only about one month of experience in checking manuscripts. She did not see the machine-based checks in advance of carrying out her human edit (hence avoiding any bias from seeing what the machine identified).  
 
Of course, measuring the time taken is only one metric; quality is another, equally important metric. After running the time checks, we compared the accuracy of the results which is shown in the attached file also. “Good” for each manuscript and section means that the particular part of the submitted manuscript was in accordance with the instructions for authors. “Bad” did not meet the instructions for authors. 

Based on the definitions above, we compared the results of our results. We found that the machine had errors on two checks: structured abstract and conflict of interest. The structured abstract check failed because the specific requirements for the journal under consideration were different to the standard structured abstract checks provided by the tool.  

Configuration for specific journals 

Following the trial, UNSILO now enables journal editors to configure the values for these fields on a journal-specific basis, so if, for example, a journal uses standard headings “methodology”, rather than the default heading “methodology”, the check can be adjusted for this.  

Currently, editors find they have to double check the manuscript manually to recheck results shown by the Technical Checks, to see if they are really accurate, and that is what may be causing the users the extra time, resulting in longer time. By configuring the checks on a journal-by-journal basis, the checks can become more automatic, and as a result, publishers will start to see both a reduction in the time taken to check and at the same time a better quality of submission.  

Conclusion

There is no perfect methodology for checking an academic paper. Both humans and machines fail to identify possible problems, and of course both will make errors in judgement. However, the trial showed that machines are much more reliable at finding errors, while humans are (currently) better at deciding on the best action from those errors.

Clearly, using the UNSILO AI-based tools alongside the human journal editor shows a dramatic reduction in the submission time; but it is important to note that the machine does not replace the human editor. The machine simply reduces the time to identify the problem areas of the submission, and leaves the human editor more time to focus on what they are good at, which is making judgements. It is the combination of machine and human capabilities that brings about the time reduction, without reducing quality.