Critique Of Current Beta Test Team Process & Suggestions For How To Enhance It Moving Forwards

The community's desire to be more involved in beta testing is well documented here on the forums and elsewhere.

Having reviewed the Q&A between Sam and Matt on the "state of play" stream on 19th October, below is a summary of my understanding of the current beta team process, followed by a critique of the process and a few starter suggestions for how DTG could successfully involve the community to enhance the beta testing process moving forwards.

Link to conversation relating to beta testing from the stream:


Summary of current beta test team / process:

• The beta team is around 20-30 people - a mix of rail experts and devs - all avid TSW players
• The group is that size because:
  • DTG are unsure how to effectively manage beta testing with a larger number
  • There is a trust issue, with people breaking trust in the past, so Matt has kept the group small.
  • Beta team feedback isn't always consistent, which Matt considers to be an issue
  • The beta team haven't always played nicely together in the past. This is inferred from Matt's response - see full quote below
• Matt personally has final say on who is in and who is not.
• Matt either knows members directly, or they are recommended to him.

Concerns with current process:

In Matt's own words,

"There is so much myth and uncertainty about what is right in the world of rail, it's ridiculous. The number of times that even the group that we've got on there at the moment can't agree on how something works.

One of the reasons that the Boston signalling is not in the greatest place is because it was changed 2 or 3 times following a number of rounds of feedback from the beta testers - "we've found some new information, it works differently" - so we re-wrote a bunch of stuff and the end result just wasn't where we wanted it to be.

Now I'm not at all blaming the beta testers for that. We very much value what they do. But you can imagine, that's with 20 or 30 people. You grow that to 500 and all of a sudden I've got all of the truths being thrown at me in the world that don't match up with each other, and so I need to grow that team in a way which means I can see consistency. I can see people that will actually work well together in the beta team and not rub each other up the wrong way, and create a bad environment".

1. Uncertainty & consistency

• It is unclear where the "uncertainty" is and how this relates to beta testing. Using the example above, was the Boston signalling system classified information, or would that information have been available to the dev team prior to development?
• Instead of developing (in this case the signalling system on Boston) to a clear specification, and validating that specification against user feedback, the example above suggests the dev team were relying on feedback from the beta team to "advise" them on how it should work. If accurate, the current setup is more akin to a "think tank" than a beta testing group.

• It is also unclear why there is an expectation for beta testing feedback to be "consistent". In any group of human beings, there will always be differing opinions, to differing degrees.

2. The beta team must "not rub each other up the wrong way".

• Unclear how / why this is relevant.
• Even if beta testers are having direct conversations with one another, why is "personality fit" a factor in driving this process? This is professional software development - set clear, fair rules and enforce them fairly and consistently. If testers behave in an unprofessional manner, remove them from the team.

Recommendations:

• Beta testing conducted remotely through a standardised communication / feedback process directly with DTG, who evaluate collective feedback against their own specifications for how things are supposed to work.
• If people don't play nice, they don't play at all. Extending the beta testing programmes will make it easier to find people who act in a professional manner. Based on sustained interest expressed by the community, you won't have difficulty finding those people!

4. Does everything go through Matt?

• It sounds like just about everything relating to TSW development goes through Matt. Is this accurate?
• Matt appears to holds the keys to entry, to the point he knows everyone, at least to some degree.
• Based on the quote above, Matt appears to be the (sole?) recipient of all beta testing feedback.
  
• If the above statements are accurate, this simply isn't an effective (or manageable) way of working and certainly not conducive to growing the beta testing team..

Recommendation:

• Matt delegates responsibility for co-ordinating beta testing.
• A sensible place to start would be to ask the head of QA, or at least for their input on who would be ideal candidate(s) with the organisation. Organising such a group would likely be bread & butter to a QA specialist, or at least an enticing challenge to overcome.
• Beta testing feedback tracked through a system which makes evaluation of feedback manageable

Game developers use beta testing effectively, so extending the beta testing team in a way that benefits the team and the quality of the game is 100% achievable.

Extending the beta testing programme would enhance the quality of your products, support the dev and QA teams, and enhance engagement with the community.

It's a win / win.

I welcome your feedback, and especially examples of how beta testing has been successfully implemented by other game developers, with ideas for how DTG could enhance the process moving forwards.

Thank you.

 

Matt most likely does know everyone on there, I think possibly in part because he was absolutely chastised on here for not knowing someone who had previously been in the beta team.

There's also no suggestion anywhere that the disagreement on Boston was like children having a slapfight in the playground (which is what you seem to be implying), it was most likely a civil disagreement, or at worst a heated debate - and who said anything about it being driven by personality?

For such a small group, why is Matt taking in the feedback and then relaying it to the devs a problem?

 

There is another game I've player for far too much time (WoT), which as far as I can tell has beta testers and Supertester.

The Supertester program works as follows: you download another completely different version of the game, which basically is one update ahead, before X date you have to try all the new features and vehicles, and then you have to answer a survey about it. I know all of this because a friend of mine recently got into the program.

I think I would really like if TSW had a similar system, I wouldn't mind having to pay myself for the DLCs (although you shouldn't be forced to get them all), or only having them on lease or in a special beta version of the game. I'm guessing too many people would try to join just to get free DLCs, and wouldn't bother to try things properly.

 

Isn't that just the QA team, which are different to the beta team? Only difference to that Supertester system I can see is the QA team are actual paid professional employees, not volunteers from the community.

 

As far as I can tell both QA and Beta have strict and specific "things" they have to test, the community-done supertester system is much more broad, you basically play the game as you would normally and give feedback.

 

I was under the impression the Beta team just test what they want, when they can (they are simply volunteers, remember)

Also, you say the supertesters "basically play the game as you would normally", but earlier said they "have to try all the new features and vehicles" - idk if it's just me but those two statements seem to conflict each other.

 

They are given an account with the new features and vehicles, they play normally using these.

The difference would be that for example (in the case of World of Tanks), the beta team would be assigned specific tasks, for example:

Try to do X, see if there are bugs, now go that Y spot on the map, and see if you can climb up somewhere you shouldn't be able to climb, try this input combination, see if there are consequences, etc...

Supertesters instead are given just a "beta version" of the game, with the new features and tanks, and they are free to do whatever they want, as long as they form a meaningful opinion to answer the survey at the end.

It would be like the difference of giving you a free TSW DLC and then asking your opinion, and having tasks assigned, which might for example be:

Try to climb up every platform, see if all the level crossing work, see if you can get into every train at every platform and not fall down like in HHL, etc..

From what Matt said I got the impression both Beta and QA have a "list" of things they have to make sure are right, I might be mistaken though.