What to do with questions asking to evaluate commercial online services?

Question

We have at least one old unclosed question asking to evaluate some commercial online services: Is Academia.edu useful? This was cited in favour of questions going into the same direction: Is Peer.us useful? The latter question in turn has attracted a controversial answer originating from the operators of that platform. Both questions have received close votes in the past.

I think it’s time for a general rule how to deal with such questions and thus I ask:

• Shall we allow all such questions, only some of them, or close them altogether? If only some, where do we draw the line?

• If we allow all or some of such questions, how shall we deal with opinionated answers and comments, i.e., answers not focussing on objective observations but mainly on bashing or praising the platform? In my experience, it is almost inevitable that such posts will happen, no matter how objective answers the question seeks.

(Note that I explicitly exclude services without commercial goals, such as the Arxiv.)

Answer 1

Historically, this community has had a strong policy of staying away from assessments of (most) specific organizations, commercial or otherwise. I believe that this is a good policy for several reasons:

• Assessments are often highly relative and based on perspective
• There is a temptation for advocacy, whether for personal ("Go Tech! Beat State!") or financial ("Buy our widgets!") reasons.
• Complementarily, people are likely to become upset if others place harsh judgement on an organization that is important to them.
• Many organizations (especially new entrants to a field) will change quickly in their nature, impact, and significance, and answers will tend to become rapidly obsolete.
• Allowing any assessments of organizations opens the door to a potentially unbounded flood of requests to assess other organizations.

I think that this should apply no matter how large the organization, when the question is about assessing the organization.

Instead, I notice that most well-regarded questions about organizations seem to fall into two classes:

1. Questions about established places like LinkedIn, ResearchGate, Facebook, Google Scholar, IEEE, etc., which already assume that organization is notable and legitimate but which are asking advice about how to manage some aspect of one's interactions with it with respect to some aspect of academia. Thus, we take the organization for granted and ask for experts in it to share their experiences.

2. Questions about whether to trust a possibly sketchy organization. These are often something that can be generalized to a class of organizations, like how to assess whether a conference or journal is predatory.

In neither of these cases do we need to assess an individual organization, and thus we avoid the tar-pit of associated problems.

Thus, if a question cannot be edited into one of these two classes, I think that it should be closed. I think the Academia.edu question might be able to be turned into a question in the first class, but I don't think the Peer.us question can be.

TL;DR: Evaluating an organization is not OK. Evaluating a class of organizations is OK.

Answer 2

Note that we have allowed questions about a specific possibly predatory journal, e.g. How is publishing in JoVe (a "video journal") perceived?.

What I see in this particular question that seems to me to make it well posed is

How do they compare with similar services; and how they want to monetize their product

Furthermore, I think the deleted controversial answer can easily be edited and restored, as it answers the business plan aspect.

Where to draw the line? This will be easier to figure out if and when we start getting more of this type of question. (Right now we have a total of two, if I understood right.)

How to deal with opinionated answers and comments, i.e., answers not focusing on objective observations but mainly on bashing or praising the platform? In the same way we deal with other opinionated answers that are presented without documentation or specific support for the position being presented.

The business side of academia is starting to change, as business models in the modern world are changing. Let's not put our heads in the sand and refuse to question and understand these changes.

---

Edit: Responding to the question in the comment (To what answer are you referring?) -- in the question post, I read

This was cited in favour of questions going into the same direction: Is Peer.us useful? The latter question in turn has attracted a controversial answer originating from the operators of that platform.

When I go to the link, there is a notice at the bottom of the page about a deleted answer.