What is a good way to gather feedback?

[LiamLime]

Gathering feedback is notoriously difficult. Most game developers gather feedback after a game's launch through completely automated systems. Devs then test their decisions through AB tests. AB tests however sort of rely on users being unaware of the tests happening, which is impossible here.

The game does have a feedback system, however it's not really made in a way which allows people to review how a change affected the various gathered parameters, additionally the parameters which are gathered mostly fall into the "fun things to read about" category, not the actually useful, but way less fun, parameters such as user retention, focus, engagement, session length, activity during sessions, action ordering, etc. + mapping how all these parameters change over time, over player age (how long they've been with TGS), over the course of an individual play session, etc. Lastly there is the extra question of whether players are actually willing to be tracked for feedback reasons, knowing that the gathered information would help coders see if the changes they do actually had their desired effect.

I can already hear your cynicism about how coders would not look at data. It's not necessary that they do, instead this could fall into the domain of an analyst, or anonymized data could be available to the public for scrutiny. If code leads impose expectations on coders that content which fails to produce the desired effect will get reverted, then this might lead to a gradual change in coder culture. It won't be instant though.

The system on github works fine. The 24 hour review and comment process is generally a good thing, that's how most development (game or otherwise) companies operate. It would be nice if "I don't like this, close" happened less frequently, but whatever, it's volunteer work, imperfection will happen. I however don't think the 24 hour review period is a good enough place for player input on changes. The period is too short for enough players to see the change ahead of its addition to the servers. I also don't think player reviews before the change goes live are necessarily a good thing. Not because the players reviews are bad, just because it would slow down updates a lot.

We also have the forums. Coders can post feedback requests and players can post topics giving feedback on various things. There are however two issues here: Many players seem to jump to sensationalism and absurdity in many of their replies. Sensationalism is when players write an over the top negative reply about some minor, inconsequential change, stating about how it is the worst thing ever, usually in more vivid language. Absurdity is when they cite counter-examples to prove their reasoning, which are extremely unlikely, rare or inconsequential. On the coder side, reading and replying to forum feedback is difficult, because it requires a lot of time to get through it all, it requires manual collation of how many people like it and how many don't, and it requires the coder to make assumptions about how many people are in the silent majority for and how many against the change. In other words, the feedback forums are completely useless to coders, except when they provide very detailed and thought out reasoning for the poster's opinion. There are ways of encouraging that, however it requires much closer forum moderation and many people would perceive it as censorship (or it could actually devolve into censorship if not implemented correctly). Examples include mandatory post forms, minimum or maximum word counts, minimum quality standards for posts to not get deleted, etc.

Ingame polls are also a thing. Unfortunately, however, these are difficult to use and would need to be used much more. These are much faster to respond to for players, allowing much more of them to state their opinion. They however do not allow players to post more detail about their opinion, they can only choose from one of the pre-defined answers. This does however allow coders to get a very quick tally of how a change is perceived.

One more thing which has already been tried, and didn't really work, is a dedicated beta server. This server would run the latest code and have daily code updates, with the other servers only getting updated once per fortnight or month (unless critical bugfixes require an out of cycle update). While this is common with indie studios, it may not work well here, simply because there aren't enough players to sustain it, meaning it would either get all the population or none of it.

tl;dr:

So with all that in mind, what would you say is the best way for coders to gather feedback? Think of how well each person can express their opinion's nuances, whether they can suggest improvements, how easy it is to respond to, and also how quickly a coder can aggregate the feedback.

Suggestions explained above are: AB tests, per-player feedback gathering (aka tracking), github reviews before changes go live, feedback forums, ingame polls and a beta server. More suggestions are always welcome.

[Miauw]

polls are, imo, not really very good.
they tell you if players like or dislike something, but they don't tell you what people like or dislike about that thing, which is very, very important.
sure if nobody likes a thing after a few weeks it should be removed, but learning why it should be removed is still useful, and in many cases players can be divided on a feature and we need more detailed feedback.

[Aranclanos]

why do you make a new account to post this
to don't derail too much, I like stats that I never have. All the time people talk about their feelings on balance issues, which is fine too, but the lack of real stats is exhausting. Things like percentages of victories across the different types of antags or even the reasons of why the shuttle is called, round durations (that's a bit harder to measure though). Stuff like that.

[invisty]

Where's the option for "NAAAAAAH I CANT HEAR YOU", *merges pr* ?

[LiamLime]

why do you make a new account to post this
to don't derail too much, I like stats that I never have. All the time people talk about their feelings on balance issues, which is fine too, but the lack of real stats is exhausting. Things like percentages of victories across the different types of antags or even the reasons of why the shuttle is called, round durations (that's a bit harder to measure though). Stuff like that.

Some of that does exist though.

For per round stuff you have (example for Traitor): http://ss13.eu/tgdb/tg/latest_stats.html#mode_traitor
That has the per-antagonist type success rate as well as how well or poorly each antagonist did on individual objectives. Round durations are also below

For global antag success rates (irrespective of round type or objectives) you have this: http://ss13.eu/tgdb/tg/latest_stats.html#antag_success

For global antag objective stats (irrespective of round type) you have this: http://ss13.eu/tgdb/tg/latest_stats.html#antagobjetive

For survival and escape rates, you have this and the graphs after it: http://ss13.eu/tgdb/tg/latest_stats.html#survivalrate

And for round duration without respect to round type, there's this: http://ss13.eu/tgdb/tg/latest_stats.html#rounddur

I however wouldn't call any of that extremely important data for decision making. Antagonists are not the majority of the player base during a round and focusing exclusively on them would be wrong. I also don't think data about shuttle call reasons is gathered.

[Not-Dorsidarf]

A reminder that it's very, very rare for features to be merged after 24 hours. Almost all, even minor content ones, are made to wait something like a week.