28

This is to share a case study about the "fastest gun in the west" issue discussed on Academia.SE here and more broadly here.

On this question, the sequence of events was:

  • Buffy, Guest, and I all submitted answers on literally the same minute.
  • Buffy's answer was immediately upvoted, and from there people piled on. Within a ~day, he was ahead by 40 votes; I think the score was something like 100-60. Our answers are very similar, so it's a bit strange that there was such a difference (I blame the dog picture...).
  • Then, OP accepted my answer, meaning that my answer is displayed first, despite having fewer votes. The gap quickly began to shrink, and a few days later, I am ahead by 10, 216-205.

Since neither of us over edited our answers, this is a clear illustration that the top-rated answer gets a huge bias. [For that matter, I've seen similar behavior in close voting -- hard to stop the close train once it gets rolling -- but that's a separate topic.]

Of course, I don't particularly care about my imaginary points, but it does seem like an obvious weakness:

  • Better answers submitted after ~10 votes are in are unlikely to be read
  • This affects new users who can't even comment yet
  • As Buffy points out, the same is true for downvotes too (though it seems like we downvote less on this SE than others), and arbitrary downvotes are a sure way to discourage new users.

We've discussed this before, but I guess my questions are:

  • What could we do, mechanically? Does SE support solutions like hiding the vote totals for the first few hours, or is there simply nothing that can be done without getting the powers that be to write entirely new code?
  • If we do have the power to make such changes, has the community already decided not to? The questions I linked seemed open to such changes, but nothing happened.
13
  • For this particular case, you can blame voters from Hot Network Questions ;) . Anyway, what is the purpose of hiding the vote totals for the first few hours? Do you mean that voters are biased to already highly-upvoted answers? Does that also include hiding it from the OP? Will that also affect the answer order, since one of the sort is by "votes"?
    – Andrew T.
    Jan 29, 2019 at 8:19
  • 3
    The point is: does it really matter?
    – Massimo Ortolano
    Jan 29, 2019 at 9:11
  • 9
    Actually, "piling on" by voters also happens in the negative direction. I think that the more likely reason for this behavior is the inclusion of some simple phrase within the answer that people either especially like or dislike for some reason. I've seen both quick up and down voting on some of my recent answers. Some of them are hard to explain.
    – Buffy
    Jan 29, 2019 at 12:20
  • 3
    @MassimoOrtolano - I would advocate the perspective that it does, particularly for newbies on SE, as many/most SE privileges are rep based. Post 3k rep points, perhaps not so much. The psychological biases that people carry in voting (this post being an example) can potentially become a barrier to contributions. e.g. X saw a question that he could contribute to, something beyond the existing 5 answers, with the top answer at +40. We have a problem if X thinks "Why bother posting an answer, no one will bother reading past the top 2 answers. It will collect say 2-3 votes and will stay buried".
    – 299792458
    Jan 29, 2019 at 15:36
  • 1
    @MassimoOrtolano (contd.) While SE has inserted badges to denounce this reasoning, we perhaps can't deny that it does indeed happen. Even more so on the relatively softer Academia.SE, as opposed to other more technical stacks, where the validity of an answer is more objective (yes or no). In my early days on SE, I personally found this to be some sort of rich-gets-richer system, and it was pretty discouraging. What this post does is, it very interestingly brings out is a case study perspective on this issue, which totally supports the observation. I think it warrants a discussion.
    – 299792458
    Jan 29, 2019 at 15:45
  • Andrew - Yes, I think it's clear voters are biased to upvoted answers, and hiding vote totals for the first few hours would force people to think critically before voting. This may introduce additional complications, so I'm not necessarily advocating that particular solution just yet, but offhand it seems like a good idea. Massimo - I think it does matter, though admittedly more for new users and downvotes than for upvotes on high-rep users, Buffy - agree, updated my post, The_Dark_Side - agree, well said, thank you.
    – cag51 Mod
    Jan 29, 2019 at 15:51
  • I think hiding vote totals for a bit isn't a bad idea, but it has the downside of someone finding the question and not knowing what the community thinks the best answer is for a while. Maybe not important on this site as much, but I could see SE being reluctant to implement it on a technical site, lest someone follow really bad advice that was actually downvoted to hell but no one could see
    – Azor Ahai -him-
    Jan 29, 2019 at 22:14
  • 1
    @TheDarkSide, for the record, I will seldom post to a question with five answers if I think that, collectively, they provide a sufficient answer to the OP. For me, having a big rep only means that I'm being more useful to people than otherwise. But it also gives me the courage to occasionally say things that I'm pretty sure will be controversial. Not everyone wants to read some things that they need to read - especially on ethical issues (and copyright).
    – Buffy
    Jan 30, 2019 at 13:12
  • This probably belongs more on the main meta, though it's been discussed there already as well.
    – Bryan Krause Mod
    Jan 30, 2019 at 18:10
  • 2
    Here is an example of negative piling on: academia.stackexchange.com/a/124038/75368. I say some things that (IMO) are a bit misinterpreted, but that people really don't want to hear, whether accurate or not.
    – Buffy
    Jan 30, 2019 at 19:08
  • @Buffy - Haha, I was thinking the same thing when I saw that thread :-) Though in that case, I was rather hoping to see some insight about why this would or would not be illegal/unethical/messy, rather than just advising to take the safe course and avoid risking any blowback (which is what I would do, since I don't want to find out "what could go wrong" the hard way)
    – cag51 Mod
    Jan 30, 2019 at 20:45
  • 4
    If you attempt to solve this by hiding vote totals, you also need to randomize the order of posts seen by different viewers. Otherwise, a reasonably good answer at the top of the list will also collect "that's OK, upvote it, move on to the next thread" bias. The basic issue, IMO, is that people in general like expressing their opinion (i.e. voting) more than they like mental effort (i.e. choosing carefully what to vote for).
    – alephzero
    Feb 9, 2019 at 17:24
  • I think the problem might come because people only vote on the posts they read. Top posts get read, bottom posts get neglected. Especially on Academia where the answers tend to be lengthy.
    – LN6595
    Feb 11, 2019 at 3:19

2 Answers 2

Reset to default
13

It's not something we can solve as academia.SE.

As you noted, it's been a known thing for many years over at meta.stackexchange.com (Fastest Gun in the West Problem). They even have a likely solution noted there, which is what Reddit adopted some years back. I imagine there must be counterarguments as to why that isn't a great idea. But regardless, so far as I know it isn't something that Stackexchange is planning to deal with :-(

2

Some of this seems not to be "first answer bias".

You say that when your question became "accepted", then your answer started quickly accumulating more upvotes at a faster rate than buffy's, even though it was the other way around before. This seems to be like a "top answer bias".

I disagree with the other answer, which says that "it's not something we can solve as academia.SE". I believe this has been already fixed on Meta.SE where instead of sorting the questions by the number of votes (which I think is the default here):

enter image description here

the questions can be sorted by most recently active:

enter image description here

It's possible that MSE also has "votes" set as default and I just changed it some time long ago to "active" without noticing, but either way we could easily make "active" the default instead of "# of votes" so that the first answer (or most voted answer) doesn't get an unfairly disproportional number of upvotes. We could even make a new one called "random".

I do think these things would improve the site, because I find the "chain-reaction voting" to be a much bigger issue here than on any other site I'm active on (I have 1000+ rep on 7 sites and 150+ rep on 26 sites, and I created one site currently in Beta from scratch in Area51: so I've been on SE almost 24/7 for a while, as anyone who lead the launch of a site would be able to appreciate). I recently wrote a Meta post here about "chain-reaction voting" (on Academia.SE) too: How do people here feel about chain-reaction downvoting of posts?.

Let me conclude with my opinion about why "top answer bias" (not necessarily "first answer bias") happens:

I can at least speak from my own experience on Academia.SE: Recently I read an answer by BryanKrause for which I commented "this might be one of the best answers I've ever seen on SE, and I've saved it somewhere for me to re-read over and over again later", and I upvoted. However it was a long answer with a lot of substance, which I read slowly and nodded my head to the entire time. By the end of reading the post and reflecting so much over it, I remember being too exhausted to read all the other answers with the same amount of attention. I just double checked and it's true that I didn't upvote the 2nd or 3rd listed questions either!

Often people read the first answer they see, and they might put a decent amount of energy into processing it or commenting on it, then they move back to whatever they were working on or whatever SE site they were on before being the HNQ list drew them here. If the answers were set by default to "random" or "active" instead of "votes" or "oldest", then the "first answer bias" or "top answer bias" could be eliminated, without too much disadvantage.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .