Abstract
While earlier research had exposed severe problems with the data quality and analysis of the 97% consensus paper (Cook et al, 2013, Environmental Research Letters), this note finds the authors have contradicted themselves and that the data gathering invalidates all results.


The 97% consensus paper (Cook et al., 2013) was hailed as the best ERL paper of 2013 (Cook, 2014). Downloaded more than 228,000 times (ERL, 2013) and with an Altmetric score of almost 1500 (Altmetric, 2013), it definitely was the most visible.

The core result of Cook et al. is unremarkable (Montford, 2013): They find that the academic literature says that human activity is one of causes of the observed global climate change.

Some have claimed that Cook et al. found a consensus on the dangers of climate change (Kammen, 2013) or on the need for climate policy (Davey, 2013). They investigated neither. Even some of the authors of the paper misrepresent its findings (Nuccitelli, 2014, Friedman, 2014, Henderson, 2014).

Cook et al. took a sample of the academic literature and rated its contents. The raters were recruited through a partisan website (Cook et al., 2013) and frequently communicated with each other (Duarte, 2014). Their sample is not representative of the literature (Tol, 2014a). The sample was padded with large numbers of irrelevant papers (Tol, 2014a). For example, a paper on photovoltaics in Kenya (Acker and Kammen, 1996) was taken as evidence that climate change is caused by humans as was a paper on the coverage of climate change on US TV (Boykoff, 2008). Three-quarters of the "endorsing" abstracts offer no evidence either way (Tol, 2014a). Their attempt to validate the data failed (Tol, 2014a). An attempt to replicate part of the data failed too (Legates et al., 2013). The data show inexplicable patterns (Tol, 2014a) while the consensus rate suffers from confirmation bias (Cook et al., 2014a, Tol, 2014b).

The problems do not stop there. It appears - no survey protocol was released - that the research team (1) gathered data (19 February to 15 April 2014), (2) studied the results, (3) gathered more data (11 May to 1 June 2012), (4) studied the results again, (5) changed the classification system, and (6) gathered more data and reinterpreted the rest. The results from step (1) and (3) are different (raw sample chi-sq(df=6)=255, p<0.001; matched sample chi-sq(df=6)=393, p<0.001). The results from step (3) and (6) are different too: The dissensus rate changes by one-half (Tol, 2014a).

Alterations of the sampling strategy or survey design during the course of data gathering, and revealing preliminary results to the data gatherers are typically frowned upon because of the risks of (inadvertently) skewing the results. In this case, the authors had a sharp prior on the results before the first data were collected (Andrew, 2013, Montford, 2014).

Cook et al. were slow to release part of their raw data, hindering attempts to check their analysis in contravention of journal policy (IOP, 2013). Part of their data were never released.

Time stamps were held back because they "were not collected" (Cook et al., 2014b) although they were referred to in an earlier exchange (Cook, 2013). Time stamps were part of an unofficial data release (Shollenberger, 2014). This allowed the above reconstruction of the data gathering process.
Time stamps corroborate my earlier hypothesis (Tol, 2014a) that some of the strange patterns in the data are due to rater fatigue. This was contradicted by (Cook et al., 2014b) even though John Cook had earlier noted that "[e]veryone's suffering rater fatigue" (Cook, 2012). Indeed, one rater read and classified 765 abstracts in the course of 72 hours. Rater fatigue implies unreliable data.

Rater IDs were also held back, referring to a confidentiality clause in an ethics approval (UQ, 2014) that does not cover this part of the data gathering (UQ, 2012). Rater IDs were unofficially released later. The officially released data show that different people rated the same paper differently in one-third of all cases (Cook et al., 2013). The unofficial data reveal more. The survey started with a team of 24 raters, but numbers fell quickly. There is a difference between the raters that stayed and those that left (t=-6.51, p<0.001). The group of raters also changed its composition after the data were inspected (chi-sq(df=23)=7265, p<0.001).

In sum, one of the most visible climate papers of recent years is not sound. Whereas previous critique could be interpreted as a lack of competence (Tol, 2014a), the later data release suggests that Cook et al., perhaps inadvertently, worked towards a given answer. This reflects badly on the authors, referees, editors and publisher. It also weakens the activists and politicians who cite Cook et al. in support of their position.

The data and statistical tests underlying this note can be found at http://www.sussex.ac.uk/Users/rt220/CookERL.xlsx Jose Duarte and Andrew Montford had excellent comments on an earlier version.


REFERENCES
ACKER, R. H. & KAMMEN, D. M. 1996. The quiet (energy) revolution: Analysing the dissemination of photovoltaic power systems in Kenya. Energy Policy, 24, 81-111.
ALTMETRIC. 2013. Score in context [Online]. Available: http://www.altmetric.com/details.php?citation_id=1478869&src=bookmarklet [Accessed 5/9/2014.
ANDREW. 2013. Cook's 97% consensus study game plan revealed. Popular Technology [Online]. Available from: http://www.populartechnology.net/2013/06/cooks-97-consensus-study-game-plan.html.
BOYKOFF, M. 2008. Lost in translation? United States television news coverage of anthropogenic climate change, 1995-2004. Climatic Change, 86, 1-11.
COOK, J. 2012. Fatigue. Available from: http://rankexploits.com/musings/2014/sks-tcp-front/#comment-130926.
COOK, J. 2013. Query re request for Cook et al. data [Online]. University of Queensland. Available: http://www.sussex.ac.uk/Users/rt220/Cook31July.png
COOK, J. 2014. Skeptical Science consensus paper voted ERL's best article of 2013. SkepticalScience [Online]. Available from: http://skepticalscience.com/SkS-consensus-paper-ERL-best-article-2013.html.
COOK, J., NUCCITELLI, D., GREEN, S. A., RICHARDSON, M., WINKLER, B., PAINTING, R., WAY, R., JACOBS, P. & SKUCE, A. 2013. Quantifying the consensus on anthropogenic global warming in the scientific literature. Environmental Research Letters, 8.
COOK, J., NUCCITELLI, D., SKUCE, A., JACOBS, P., PAINTING, R., HONEYCUTT, R., GREEN, S. A., LEWANDOWSKY, S., RICHARDSON, M. & WAY, R. G. 2014a. Reply to 'Quantifying the consensus on anthropogenic global warming in the scientific literature: A re-analysis'. Energy Policy, 73, 706-708.
COOK, J., NUCCITELLI, D., SKUCE, A., WAY, R., JACOBS, P., PAINTING, R., HONEYCUTT, R., GREEN, S. A., LEWANDOWSKY, S. & COULTER, A. 2014b. 24 Critical Errors in Tol (2014) - Reaffirming the 97% consensus on anthropogenic global warming. Brisbane: SkepticalScience, University of Queensland.
DAVEY, E. 2013. Climate change, acting on the science. Gov.uk [Online]. Available from: https://www.gov.uk/government/speeches/edward-davey-speech-climate-change-acting-on-the-science
DUARTE, J. 2014. Cooking stove use, housing associations, white males, and the 97%. José Duarte [Online]. Available from: http://www.joseduarte.com/blog/cooking-stove-use-housing-associations-white-males-and-the-97
ERL. 2013. Metrics [Online]. Institute of Physics. Available: http://iopscience.iop.org/1748-9326/8/2/024024/metrics [Accessed 5/9/2014.
FRIEDMAN, D. 2014. A climate falsehood you can check for yourself. Ideas [Online]. Available from: http://daviddfriedman.blogspot.co.uk/2014/02/a-climate-falsehood-you-can-check-for.html.
HENDERSON, D. R. 2014. David Friedman on the 97% Consensus on Global Warming. Library of Economics and Liberty [Online]. Available from: http://econlog.econlib.org/archives/2014/02/david_friedman_14.html.
IOP. 2013. IOP Ethical Policy for Journals [Online]. Institute of Physics. Available: http://authors.iop.org/atom/help.nsf/0/F18C019D6808524380256F630037B3C2?OpenDocument
KAMMEN, D. M. 2013. The story of a presidential tweet. The Berkeley Blog [Online]. Available from: http://blogs.berkeley.edu/2013/05/29/the-story-of-a-presidential-tweet/
LEGATES, D. R., SOON, W., BRIGGS, W. M. & MONCKTON OF BRENCHLEY, C. 2013. Climate Consensus and 'Misinformation': A Rejoinder to Agnotology, Scientific Consensus, and the Teaching and Learning of Climate Change. Science and Education.
MONTFORD, A. W. 2013. Consensus? What Consensus? London: Global Warming Policy Foundation.
MONTFORD, A. W. 2014. Fraud, bias and public relations - The 97% consensus and its critics. London: Global Warming Policy Foundation.
NUCCITELLI, D. 2014. Twitter profile [Online]. Available: https://twitter.com/dana1981 [Accessed 5/9/2014.
SHOLLENBERGER, B. 2014. TCP Results! Izuru [Online]. Available from: http://www.hi-izuru.org/mirror/
TOL, R. S. J. 2014a. Quantifying the consensus on anthropogenic global warming in the literature: A re-analysis. Energy Policy, 73, 701-705.
TOL, R. S. J. 2014b. Quantifying the consensus on anthropogenic global warming in the literature: Rejoinder. Energy Policy, 73, 709.
UQ. 2012. Notification of approval [Online]. Available: http://www.climateaudit.info/correspondence/foi/queensland/cook%20consensus%20Documents%20released%20under%20RTI.pdf.
UQ. 2014. UG and climate change research. UQ News [Online]. Available from: http://www.uq.edu.au/news/article/2014/05/uq-and-climate-change-research.

UPDATE: The comments by an editorial board member are below. I appealed this three times, but to an avail.

This is an unsolicited Perspective on the Cook et al. paper from 2013 [1] by an author who wishes to reflect on the original study published in ERL.
This submission follows a solicited Perspective [2] on the Cook et al. paper, which reflected on the study and the broader implications of the study internationally at the time of publication.

This Perspective also follows another submission to ERL, by the same author, of comments on the Cook et al. study, which were rejected by ERL after peer review. These comments were subsequently published [3, 4] with a response from Cook et al. [5].

ERL welcomes debate and, despite the fact that we have already published a Perspective on this study, the original paper continues to be highly popular and debated. In theory then, ERL could publish another Perspective, provided it contributes to healthy scientific debate, is original, timely, and advances the literature on this important theme.

Overall, this current submission shows a different standpoint from the commissioned Perspective published in 2013, and thus could have had the potential to contribute. Unfortunately, in its current form, this piece is not of satisfactory scope or breadth for a Perspective, and is not sufficiently original. It also has problems of unsubstantiated assertion, lack of disinterested reflection, and polemic expression.

I have the following detailed comments and suggestions for improvement:

1. Scope: ERL guidelines suggest that a Perspective is a commentary “highlighting the impact and wider environmental implications of research appearing in ERL”. This submission does, briefly, look at the impact of the Cook et al paper internationally (particularly lines 29 to 38 on page 1 and lines 40 to 45 on page 2). Unfortunately, at present, this discussion of impacts forms a minor part of the submission, and the bulk of the discussion is a repeat assertive critique of the methodology of the original study, which is not the scope or purpose of a Perspective in ERL. Generally, there is also a missed opportunity here to discuss the one-year-on implications of the original research (particularly given its profile internationally). There is not even a reference to the previous ERL Perspective [2], which did look at implications of the work in 2013, and which I think could, very usefully, have been the basis for this follow up Perspective.

2. Breadth: This might have been an interesting perspective had the author put this study in the context of the fairly prolific academic literature on the theme of scientific consensus around anthropogenic climate change (including the ERL commissioned Perspective[2]). In the context of this submission, it is odd that the author does not reflect on other previously published studies, including the original study, (on which Cook et al. base part of their design[6]), and other studies of climate scientists’ viewpoints on anthropogenic climate change. In fact it is remarkable, and would have been an interesting point for reflection, that the results of other studies published in peer reviewed journals in this field, conducted over different periods, and using a variety of methods (ranging from direct interviews of climate scientists to literature reviews of climate papers) all find the same narrow range of 94-98% scientific consensus on anthropogenic climate change [6-9]. In this context, it is not clear why the author focuses so much time and energy on this one study within the overall literature, as it was, essentially, simply a timely, interesting, original study, published by ERL as such, which corroborates results of other studies within this field. A good Perspective would look at the wider implications of this latest study, including how it reflects overall state of knowledge of the theme, and would discuss how this study fitted into the overall policy debate - this is a missed opportunity to do this.

2. Originality: The author himself points us to three other published papers (two of which are his) discussing the Cook et al. paper[3-5]. The bulk of this submission goes over the same ground in less depth or justification, but with stronger and less considered assertion (Lines 42 to 61 on page 1 and lines 1 to 38 on page 2). The majority of this submission does not add any new information, and it is not clear why the author wishes to re-publish the same ideas contained in his already published methodological analysis of the Cook et al. study. At present then, the bulk of this submission is unoriginal. I would suggest that all this material is removed, as it adds nothing new, and the author would then have space to consider points more relevant to a Perspective (as discussed above).

4. Tone of Piece: This is the most troubling aspect of this submission. I will begin by stating the obvious: there is an important space in science for debate – openness to doubt and debates about “truth”, are an essential part of science. This debate can be very lively, but is at its most healthy when it is conducted with respect, and with rigorous presentation of evidence to back a position. In the spirit of this, ERL hosts Perspectives to encourage reflection and discussion of the scientific papers that it publishes.

Overall, the commentary could be made substantially more balanced and contemplative – for example, as proof of “truth” the author cites himself and a series of mostly social media sources, with little reference to the academic literature and with little evidence of neutrality in his selection of “evidence”.

There is a more unfortunate and confrontational aspect to the tone of this submission when the author makes his final unsubstantiated reflections on the soundness of the original research and assertions about Cook et al.’s scientific conduct (as he does in line 42 on page 2 “Whereas previous critique could be interpreted as a lack of competence (Tol, 2014a), the later data release suggests that Cook et al., perhaps inadvertently, worked towards a given answer”). At it’s most basic, there is simply not sufficient evidence of this assertion of unsoundness: the authors of the original paper made their hypothesis clear, (and most studies would have had the same hypothesis, particularly in a context where previous studies have found overwhelmingly significant positive results looking at the same issue). The original paper also had a detailed section on study design and methodology and a discussion of interpretation issues and other study design possibilities. Cook et al have also been open to release raw data, and to discuss their study methodology in subsequent correspondence. There is nothing to suggest this is “unsound” – on the contrary, most scientists would think it suggests exemplary scientific conduct.

Finally, it is not quite clear what the author of this submission aims to achieve when, much more problematically, he goes on to suggest that somehow the act of publication of the Cook et al paper by ERL “reflects badly” on the reviewers of the original paper, on ERL editors and on the publishers. What does the author imply with this statement and on what basis? The author has no evidence, and there is no evidence, that ERL and the IOP employed anything but normal state-of-the-art peer-review processes and publishing procedures when dealing with Cook et al.

Overall, I welcomed this unsolicited Perspective in the interests of a constructive debate around an interesting study published in ERL. However, in its current form, I do not find it to fit ERL guidelines for a Perspective, nor to be original, nor to be of sufficient breadth or disinterested reflection to contribute to the literature, or to knowledge.

If the author were able to do this, it would be interesting to see a substantially revised version of this perspective with these concerns addressed.

1. Cook, J., et al., Quantifying the consensus on anthropogenic global warming in the scientific literature. Environmental Research Letters, 2013. 8(2): p. 024024.
2. Reusswig, F., History and future of the scientific consensus on anthropogenic global warming. Environmental Research Letters, 2013. 8(3): p. 031003.
3. Tol, R.S.J., Quantifying the consensus on anthropogenic global warming in the literature: A re-analysis. Energy Policy, 2014. 73(0): p. 701-705.
4. Tol, R.S.J., Quantifying the consensus on anthropogenic global warming in the literature: Rejoinder. Energy Policy, 2014. 73(0): p. 709.
5. Cook, J., et al., Reply to ‘Quantifying the consensus on anthropogenic global warming in the scientific literature: A re-analysis’. Energy Policy, 2014. 73(0): p. 706-708.
6. Oreskes, N., Beyond the ivory tower. The scientific consensus on climate change. Science, 2004. 306(5702): p. 1686.
7. Anderegg, W.R.L., et al., Expert credibility in climate change. Proceedings of the National Academy of Sciences, 2010. 107(27): p. 12107-12109.
8. Doran, P.T. and M.K. Zimmerman, Examining the Scientific Consensus on Climate Change. Eos, Transactions American Geophysical Union, 2009. 90(3): p. 22-23.
9. Farnsworth, S.J. and S.R. Lichter, The Structure of Scientific Opinion on Climate Change. International Journal of Public Opinion Research, 2011.