During peer-review, should I comment on the authors' messy code?
I'm reviewing a paper in pure mathematics. A lot of results in the paper depend heavily on computer computations, and the authors have provided in the article a link to the Magma code they used for most of these computations. However, this code is almost impossible to understand due to the messy way it is written. For example, it does not use any indentation, and all variables are given names like 'aaa' or 'X' that do not give any information about their purpose in the program.
On the one hand, the mathematics underlying these computations is explained sufficiently well that it is possible to reproduce the results without using the authors' code (this is what I ended up doing). Also, the paper contains only a link to the code and not the actual code itself, so I'm not sure if the code is really in scope for the review. Moreover, hard-to-read code seems to not be uncommon in academia, and most people don't seem to mind. On the other hand, I think a small amount of work from the authors (who presumably do understand the code) would make this code a lot more usable for others, just by replacing some of the variable names with names that actually convey some meaning.
My question is, is it reasonable for me to tell the authors that their code is needlessly difficult to understand and should be improved?
3 Answers
If the authors have provided a link to their code as a reference, then it is appropriate to offer commentary, particularly if the article is based on the code.
However, I would recommend making the critique constructive: offer concrete suggestions for how to improve it rather than just saying it’s “messy” or “sloppy” and needs to be “cleaned up.”
The code is within the scope of the review, and it is appropriate to review this and offer constructive suggestions in relation to its deficiencies. Now, bear in mind that the onus is on the author to satisfy reviewers of their argument, and if the argument depends on computer code that is so messy as to be unreadable, it is not incumbent on you to fix this for them. In this case, constructive advice might be limited to explaining why it is presently too hard to read (i.e., lack of indentation, unclear variable names, etc.), and this could reasonably lead to a recommendation to revise and resubmit. Try to be clear and comprehensive in describing why the code is presently difficult to read, so subsequent re-submissions can be expected to be up to scratch.
The best thing to do in these cases is to treat the computer code just like the prose in the paper. Just as with prose, the computer code needs to be clear and intelligible, relative to the standards for coding. If it is messy and unintelligible then it needs to be revised until it is clear. Reviewers do not shy away from rejecting papers when the prose is unintelligible, so it is perfectly reasonable to request that computer code be made intelligible.
Yes, you should comment and possibly more.
You've said it yourself:
A lot of results in the paper depend heavily on computer computations.
Well, the program code for computations is therefore part of the work you are reviewing. If the text of the paper was difficult to read, would you not consider that a weakness? Logically, therefore, the same is true for the code (even if it's to a slightly lesser extent).
Also, if the code is unreadable to you - maybe there are errors in it, despite the sound math underneath. And finally, if you can tell what the results should be without the code, then why even have the code?
So, if you feel the messiness does not preclude "parsing" the paper, then comment on it (and perhaps, if relevant, downgrade it from Strong Accept to Weak Accept, although perhaps that's too harsh - depends on the specifics.)
If you need to read the code to verify the results, and you simply cannot, then that's a more serious problem. But before saying something like "Requires revision", consult with the journal editor / the program committee chair / etc.
Note: I'm a Computer Scientist, so my answer might be somewhat biased. On the other hand, I have written pure-theory papers with no code.
Have a similar question?
Ask the community →