While ERL is taking its time type-setting my paper, Brandon Shollenberger has uncovered Cook's draft (?) response. It is an interesting read. Just like the journal did not want me to talk about Cook's paper, Cook's responses to the questions raised are hidden in an appendix.

I raised five points.

1. "Cook et al. (2013) do not show tests for systematic differences between raters. Abstract rater IDs may or may not be confidential (Queensland, 2012, 2014), but the authors could have reported test results without revealing identities."

I could not find a response. Raters systematically deviate from each other, as shown here.

2. "The paper argues that the raters were independent. Yet, the raters were drawn from the same group. Cook et al. (2013) are unfortunately silent on the procedures that were put in place to prevent communication between raters."

Cook replies that "[r]aters had access to a private discussion forum" and notes that they "are able to identify potential cross-discussion of 0.26% of the sample" but admit that "some discussion may have been missed in this manual search".

Recall that the original paper had that "[e]ach abstract was categorized by two independent [...] raters."

3. "The paper states that “information such as author names and affiliations, journal and publishing date were hidden” from the abstract raters. Yet, such information can easily be looked up. Unfortunately, Cook et al. (2013) omit the steps taken to prevent raters from gathering additional information, and for disqualifying ratings based on such information."

Cook replies that "raters conducted further investigation by perusing the full paper on only a few occasions". A few is unquantified.

Recall that the original paper had that "[a]ll other information such as author names and affiliations, journal and publishing date were hidden."

4. "Cook et al. (2013) state that 12,465 abstracts were downloaded from the Web of Science, yet their supporting data show that there were 12,876 abstracts. A later query returned 13,458, only 27 of which were added after Cook ran his query (Tol, 2014a). The paper is silent on these discrepancies."

To the first point, Cook replies that "[d]uring the process of importing entries into the database, some papers were accidentally added twice and subsequently duplicate entries were deleted."

The original paper has that "[i]n March 2012, [they] searched the ISI Web of Science [...] [t]he search was updated in May 2012". Between March and May, most papers from the first download had been rated, and there are significant differences between the first and second period of rating. See here and here. It is not clear from Cook's response whether the duplicate entries are from the same download or from different downloads.

To the second point, Cook replies that "these databases and search algorithms are dynamic". That is, of course, true. At the same time, the Web of Science includes a field that has the date of data entry. This enables the reconstruction of historical queries.

5. "The date stamps, which may or may not have been collected (Cook, 2013; Cook et al., 2014b), reveal that the abstracts were originally rated in two disjoint periods (mid-February to mid-April; second half of May). There was a third period of data collection, in which neutral abstracts were reclassified. Unfortunately, Cook et al. (2013) do not make clear what steps were taken to ensure that those who rated abstracts in the second and third periods did not have access to the results of the first and second periods."

Cook replies that "the only thing that distinguished the first and second rating periods was that one was before and the other after the hacking event." He does not explicitly say that no data were analysed, and he does not provide evidence that the ratings are the same before and after. See here and here for the tests that show the contrary.