I have noticed several instances (and there have been a few flags) where the question says quite clearly something like:

I am looking for references about [some topic]. I'm not really interested in anecdotes or opinions; I am looking for data or studies.

And yet the top answer (often quite upvoted) is:

Well, I don't have any references, but I've observed this too. One time, there was an [anecdote].... It seems to me that [opinion and speculation]....

To me, this is particularly annoying when the asker states that they are an "expert" (e.g., the reference request is based on an observation from many years of teaching) and the reply is "common sense" (e.g., a student's opinion or reasoning).

Relying on downvotes to handle these answers does not seem to be working (particularly on popular questions where far more people have the upvote privilege than the downvote privilege). And to be fair, often there are no studies that anyone is aware of, and so if we delete all answers that don't contain solid references, the question will go unanswered.

So: I am just trying to get a sense for how people feel about this. Is this a problem/annoyance/sub-optimal behavior that we should move towards discouraging or disallowing? Or is it fine and there is nothing to see here? I'll add some voting options to make this slightly more concrete, but feel free to add your thoughts in an answer.

  • Let me suggest that providing both a positive and a negative answer to such meta questions is confusing. It is harder to interpret voting on the pair. One answer, with a positive slant is easier to evaluate and interpret.
    – Buffy
    Commented Jan 16, 2022 at 14:17
  • 5
    Voting only up or down on a single answer creates a false dichotomy, since there is only one score on one axis and zero nuance available, and false appearances of support, since anybody can upvote but only those with a constructive main post can downvote. Using multiple answers and ignoring the negative votes is a far more straightforward way of polling (since it's how almost every election, survey, and poll everywhere works...).
    – Nij
    Commented Jan 16, 2022 at 21:43
  • 1
    Do you want to rule out frame challenging answers? Are we a research service only?
    – Jon Custer
    Commented Jan 17, 2022 at 19:57
  • 1
    Can you give an example of such a question? It doesn't need to have one of the answers you're asking about, but I'm having issues picturing an on-topic question of the sort you're talking about. Commented Jan 17, 2022 at 20:31
  • only those with a constructive main post can downvote What do you mean? The reputation threshold? If that's it, I don't think upvotes from low-rep users appear in the count. Commented Jan 18, 2022 at 9:21
  • 1
    How many questio/ns do we actually have where an answer provided peer-reviewed studies? Like, two Commented Jan 18, 2022 at 17:16
  • 2
    Jon: well, first, this is just a discussion, I don't "want" any particular outcome. But of course frame challenges should be allowed....what I'm asking about is, for example, if someone asks "Is there any published research, or other solid evidence, showing that holding exams on Friday is helpful or harmful?" and the response is "Speaking as a student, I hated having exams on Friday, since I'd rather have the weekend to study." This answer providing one data point and a common-sense argument is not really helpful to the OP. Perhaps this (made up) example also addresses Scott's question.
    – cag51 Mod
    Commented Jan 19, 2022 at 5:41
  • Azor: I think two is a bit pessimistic, but indeed people rarely give sources here. Sometimes this is the nature of our site; much "academic lore" is unwritten. But in other cases, this seems to be a problem.
    – cag51 Mod
    Commented Jan 19, 2022 at 5:43
  • 1
    @cag51 I've always thought of that as the reason why we exist - there's not citations for a lot of these things, and so we do supply that by having experienced academics answer, and we can see if other experienced academics disagree. That commenter seems to have missed the point. On the other hand, is finding pedagogical references part of our mission? Commented Jan 19, 2022 at 22:46
  • There's a difference between supporting your own answer and not supply a reference when the questions asks for one. Commented Jan 19, 2022 at 22:46
  • @JonCuster This site is emphatically an opinion service, not a research service. That's why we vote instead of doing experiments. Commented Jan 20, 2022 at 1:16
  • The association bonus allows anybody with 200 rep on any other site to upvote anything here, without ever making any contribution at all. Such users are also more likely to vote by what sounds good or what they agree with, because that's how they got here in the first place ("a HNQ, ooh shiny, I have opinions about that!"). Being able to downvote however, requires having made a more than token effort. As a result, any question that gets to HNQ will almost invariably attract a wave of upvotes, but nowhere near the downvotes necessary to balance them appropriately. @FedericoPoloni
    – Nij
    Commented Jan 30, 2022 at 21:06

7 Answers 7


Downvote and add a comment to make it clear what the problem with that answer is.

  • Hmm...this is currently the top answer, but another answer stating that things "should be decided by upvotes/downvotes, as now" got heavily downvoted (-4 as I write this). Difficult to know how to interpret that.
    – cag51 Mod
    Commented Jan 19, 2022 at 5:46
  • 2
    @cag51: My interpretation is that this answer reflects the minimum that almost everybody can agree on. People who vote for it do not necessarily consider it to advocate that downvoting and commenting is the only thing we should do (because otherwise it would have said so and that’s what makes this answer different. Thus I do not see this answer in conflict with the current second ranking answer.
    – Wrzlprmft Mod
    Commented Jan 19, 2022 at 6:44
  • 1
    @cag51 I voted as you described because usually upvoting is the wrong answer and often commenting is the correct answer. Commented Jan 20, 2022 at 1:14

These answers should not be allowed. We already have flags for "not an answer", and a response to a reference request that doesn't provide a reference is certainly "not an answer" to the actual question. Thus, under existing policy, such answers should be deleted. There is no need for "zero-tolerance" -- flaggers and mods can handle cases individually -- but in most cases, such answers should be deleted.


These answers should not be allowed, but we should not delete them retroactively and for new questions we should first notify the post author with a post notice.

In case we decide to no longer allow these kind of answers, I think it would be unfair to the answerers of the already existing questions to delete their answers after the policy has become effective.

For newer questions, I think that before deleting an answer, moderators should invite the author to add references by adding the following post notice and wait a few days before deleting:

Post notice for reference requests

  • 2
    My only concern about the second part is that most questions get most of their attention in the first few days; waiting "a couple of days" and then deleting it means deleting a lot of highly-upvoted answers after the discussion has died down.
    – cag51 Mod
    Commented Jan 17, 2022 at 1:12
  • 3
    For post notices: what I'd really like to do is to have a post notice similar to the "controversial question" post notice, where we explicitly say "this question is a reference request; answers that do not provide references may be deleted without notice." But AFAIK, we're stuck with the canned post notices, no way to change them (let me know if I'm wrong on this).
    – cag51 Mod
    Commented Jan 17, 2022 at 1:14
  • 3
    @cag51 Apparently, if we have a strong enough case, we can ask for a custom post notice. Commented Jan 17, 2022 at 12:21

The appropriate response depends on the "hardness" of the reference request, which depends on how the tag is interpreted.

  • "Hard" reference requests give clear reasons why an experience-based answer is not acceptable, as in this well-received example. In the case of a "hard" reference request, an answer without a reference is not an answer and should be downvoted and/or deleted.
  • "Soft" reference requests bundle "have there been any studies?" in with a larger question that can often be answered without references, such as this example where my answer was accepted with no references. In this case, references are optional and reference-free answers are entirely reasonable to consider.

The distinction between "hard" and "soft", however, is a matter of interpretation and may change as the question is edited.

  • I did not downvote, but I am not 100% on board with this. We often get low-effort answers; I fear that this concept of "soft questions" may encourage them. I don't want to give people the message "go ahead, tell us quickly your opinion on this topic without thinking about it too much, in the end this is only a soft question". (This does not apply to the answer you linked, which in my opinion is a very good one.) Commented Jan 24, 2022 at 12:02
  • @FedericoPoloni For myself, I think that low-effort answers like that are well dealt with by down-votes already.
    – jakebeal
    Commented Jan 24, 2022 at 18:32

Questions which request references should be checked to see if they question is about the "content of research." If it is, then the long-standing practice is to close the question.


Note that many questions on the site have partial answers and even answers that don't, precisely, follow the instructions of the OP. Too many questions, IMO, are of the Yes/No variety, so, technically, a one word answer would follow the OPs instructions, and be useless.

But the site isn't intended as a resource pool like a library is. It is a site that offers advice to academics on their questions and often enough the question they ask are subtle enough that explanation is needed, even redirection. Academics have misconceptions like anyone else.

I'll admit that I'm one of the "offenders" here. I try to label my answers as partial or advisory when they aren't technically answers. For such resource requests I might answer when I think that the existence of the resource is very unlikely but that orthogonal thinking might resolve the OPs need for the resource.

I think we are fine without disadvantaging such answers. The OP can ignore them, and others with similar concerns might benefit from them.

  • Agreed. It feels like overmoderation to me. Further, the type of question occurs with low enough frequency that a policy simply isn't necessary. Commented Jan 18, 2022 at 1:58

These answers should be decided by upvotes/downvotes, as now. Upvotes and downvotes will reward and punish the best answers. Often the answer to a reference request is "there are no references," and so empirical answers are the best we can do. Of course, flaggers and mods can decide on cases individually -- but in most cases, we should not delete such answers.

  • I don't understand downvotes on this answer. Normally downvotes on an answer imply disagreement with it. I agree with this. Should I vote it up, or down? Note that other meta answers have similar characteristics
    – Buffy
    Commented Jan 16, 2022 at 13:51
  • In other words, is a down vote here to be construed as an upvote on the OPs other answer? And should we be able to vote on both, doubling the impact.
    – Buffy
    Commented Jan 16, 2022 at 14:04
  • 3
    @Buffy I don't understand what confuses you. If you disagree, downvote. If you agree, upvote. Commented Jan 16, 2022 at 15:06
  • @FedericoPoloni, I don't think everyone voting sees it like that. Nor do I think everyone understands they get two bites of the apple with these two early answers. So the vote totals are less meaningful.
    – Buffy
    Commented Jan 16, 2022 at 15:09
  • 9
    Up/downvotes on Meta usually have a different meaning than those in the main site, and it's not uncommon to use them as a "popularity poll" on how the main forum should be run. Commented Jan 16, 2022 at 15:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .