Poll: Should Dovetail Test New Train Sim World Content Before Releasing It For Sale?

Is it really worth spending a few hours testing content for Train Sim World to see if it works, before it is put on sale? Or do you prefer the way things are right now?

 

I 100% say that a product should be tested entirely to ensure all major bugs are found and fixed. Someone should actually walk the entire route themselves yo find the holes in the world, and to find the invisible walls that stop you from getting around. I do not mind small bugs in the game that is to be expected, but the amount of bugs that has been in ever release should not have happened. I pay quite a lot to buy this DLC and do not like being hampered by bugs that stop me from enjoying the sim.

Hope DTG is watching this thread as we need better quality testing before the product goes out to us.

 

I'm really torn on that one. The kneejerk reaction would be, of course I want stuff that works. But then, Dovetail is driving the trainsim hobby into the ground, or at very least hamper its progress into the future. The sooner they get purged from the scene for someone else to take over, the better. Further disasters a la Oakville can only accelerate that process.

So maybe don't force yourself too much. Get good or go broke, both is fine with me.

 

I 100% agree with both of you!!!!!!!

 

In all honesty it's not feasible to test it properly without external beta testers working for free.

A lot of the testing issues come down to the service/timetable mode. It probably seemed like a good idea at the time, but they clearly didn't consider the load it would present to QA. If you have 60 one hour long scenarios, which need to be play tested on three different gaming platforms, that's 180 hours testing, with a 40 hour work week, that's a full time job for someone over a month. The tester can't just be anyone, they need to be someone who understands US, British and European signalling, as well as all the associated safety systems, therefore you can't pay minimum wage, so let's say £12 an hour; That's £2160 and 4 weeks work, just to have someone play each service on Xbox, PC, and PS4. This is before we consider that every service needs to be run early, on-time and late, to ensure that this doesn't break the AI. This also assumes that everything works first time, if it doesn't, it needs to be diagnosed, fixed and retested; More time, more money. Then they need to test the locomotives/units and ensure everything works correctly there... It's a massive task and just not realistic to undertake for a small company, so they don't do it. They copy/paste parts of the services, change the times and AI around, test a few bits of it and hope for the best.

There's a reason TS1 DLC only has a handful of scenarios, not because it's not possible to create 20, 30 or 40 scenarios, but because it just isn't realistic in terms of resources.

If you want a timetable/service mode you have to accept it's going to buggy as hell, because it can't realistically be tested. They can't scrap it either because there is no scenario Editor or Steam Workshop, and even if there was, what about consoles? The game is a result of poor planning and poor understanding, so there's not a lot DTG can do at this stage to correct things. There's more bugs than they have time to fix and even when they fix some bugs, the fix ends up creating new bugs. It's a bit of a mess.

 

I personally think that one main reason why DTG is like this is because they have a chokehold on the train sim market. Back in the old days when TS was still called Railworks, they had a bit of competition with Trainz, but that stopped. Nowadays they don't really have any competition. If somebody else came out with a really good train sim that had good graphics, good gameplay and physics, properly tested content with minimal bugs, and gave people what they want, then people would start spending money on that better product rather than DTG's trash.

Therefore, DTG would have to step up their game and bring both customer feedback and quality control into play in order to compete. I know MS just came out with a flight sim that looks absolutely incredible. If they could make a new version of MSTS like that, it would be amazing. Or maybe if Trainz focused more on driving trains while keeping their route editor. The Trainz editor is in my opinion the best on the market, but the driving and physics are absolute garbage. Plus the game runs on an ANCIENT engine. Maybe step up the driving and make it resemble that of TS or TSW (not a blatant ripoff mind you) and update the game's engine. Combine that with the superb editor and you have an awesome sim on your hands that could possibly compete with DTG's sims (or I should say sim and arcade title).

Either way, it would be nice if somebody else could make a really nice sim and give DTG a run for their money. Scare them into shape with competition.

 

Well then, where do I sign up? We have a whole forum full of players here, that is on the edge of declaring war on DTG for the LOVE they keep pulling off. That must be good for something!

I don't know British signalling in its entirety, and even less American, but as long as I stick to betatesting German content, that shouldn't be an issue!

 

I have offered to be a beta tester for TSW specifically for PS4 (for free) as it’s usually me that is first to find bugs once it’s released. No answer yet

 

if trainz came to console then i would go with them rather than dtg at least Trainz has an editor

 

Of course it should be tested before being released. And I mean thoroughly. Moreover, every time a new feature is introduced, all existing content should be revised to confirm that no regression bugs have come out. It may not be obvious to everyone but it's kind of a standard in software development. A company that doesn't bother testing and supporting its digital products is a bad company.

 

Thanks for this - it was nice to see you write a slightly more generous post about DTG.... ;-)

Although.... I disagree with you.... LOL. This is because I do not think that £2160 is a great deal of money to spend on testing - for a major product release. My reaction to this is... "Is that all....?" This is surely a very small fraction of the overall development costs (?), and it would be easily recouped through:

- more sales, earlier (at premium price).... driven by higher trust in the likely quality;
- earlier learning on efficiencies gained for use on later route developments;
- less support time (although the Support Team don't actually provide any real "help", they do carry out a lot of pointless admin "processing" support tickets by shutting them down as soon as they arrive); and,
- less time spent in meetings and at water-coolers complaining about customers having unrealistic expectations about things working properly, and potentially complaining about or blaming each other for yet another fail....

More seriously - these testing "costs" could be managed and distributed - for example.... you could ask everyone in the company (140 people) to take an extra (paid) hour at lunch to play-test every working day for a week - so a broad mix of users (including the receptionist and the cleaner, through to the engineers checking other people's work) - and that would be 140 x 5 = 700 hours of testing, available right there. You wouldn't just find bugs - you would find all those bits of "game-play" that makes TSW inaccessible to the vast majority of human-kind.

It would also be an excellent whole company team building exercise - and very well worth doing... twice... for each release (at say, alpha and beta stages)....., so... 1400 hours of testing..... but, obviously, it's for DTG to steer the culture in their own organisation.... This would be in addition to a scheduled programme of testing and evaluation.

So.... this 1400 hours would be the absolute minimum that I would expect - and I am being quite serious about this.... I found it to be utterly shocking that a few early adopters of the Oakville DLC turned up so many major faults and bugs, including finding that the new "SUPER COOL FEATURE" (tank loading) was broken - and that they managed to do this in the few hours after the product was released.

 

Of course...! You would think that this is all obvious.... but I don't think that DTG is a "normal" software company.... And I don't think that they have any sort of "normal" development process (or, at least, there is no evidence that they do....)...

I don't know (for sure) where the problem comes from, but I suspect that it is historical, and built-in to the culture - and it is something that they might not ever change.... because maybe whatever they do has sort of worked and helped them to scrape along for over ten years.. with Railworks, Train Simulator 20xx, etc...

But I think TSW should be treated differently - you can't get away with this on PS4 and XBox, unless you want to permanently trash your reputation in both of these markets, and alienate yourself from millions of potential customers... Just saying....

 

You can't just have anyone testing; How's the cleaner or the accountant going to know if the PZB isn't working correctly or if a yellow aspect should be flashing not steady? The testers need to have a complete understanding of all the signalling and safety systems, or else you'll miss bugs as well as have false positives muddying the waters. Also, when a bug is found, it has to be the correct department to investigate and fix the problem; During software development a lot of issues can take many hours to replicate, identify the cause and correct.

You may get away with a carpet bombing approach for testing a simple mobile game like Candy Crush Saga, but for a game like TSW, which in many ways is quite technical, you need people who know what they're doing.

What DTG do wrong is that they don't do any testing, at least that's what many of the bugs suggest. As you note, the tank car loading is broken; I would at the very least expect the handful of scenarios to be thoroughly tested across all platforms, so that bug should have been discovered. The same goes for the banner repeaters on ECW; The banner repeater approaching Eastbourne showed the incorrect aspect 100% of the time, so even if they only tested the handful of scenarios, again they would have found this bug.

 

Testing does take place, obviously. The testers report their findings to the development team.

However, it's not the testers who decide if the product is in a shippable state - that's up to the project manager. If there are no game breaking bugs (which prohibit you from even playing the game) then the product will be shipped on the planned date and a bugfix release will come later. It's as simple as that.

The banner repeater at Eastbourne showing an incorrect aspect, as annoying as it may be, is not game breaking. You don't get a penalty when you go past it and it doesn't cause AI trains to misbehave.

 

...such as half the route missing...

Badum-ts.

 

It's not pretty, but it's not what I'd call a bug. A bug is something not working as you'd expect it to, half the yard not being there is done on purpose.

East Coastway got some bugfixes a few days ago. The banner repeaters should now work as expected, for example.

 

Not the TPWS at Lewes, as you've noted elsewhere, or certain flashing yellow signals. Still lots of bugs. The flashing yellow issue is down to DTG not understanding signalling, which highlights why having testers (and devs) with knowlege of signalling systems is so important.

 

Bugs are a fact of life, and since they have a finite amount of resources they will have to give some stuff priority. That's just how it works.

The TPWS issue at Lewes requires a rather specific set of circumstances to be met in order to appear. I can imagine that a bug like that goes way down the list of priorities.

I have to agree with this, it does feel like DTG have a bit of a knowledge gap when it comes to signalling.

 

So half the route missing is not a bug, but a feature?

I mean, I see where you're coming from, and you're right to be fair. r/AssholeDesign, not CrappyDesign. But from the cosumer perspective, that is most definitely not "working as expected"

 

Either way, it's not up to the QA department or the external testers to say "you're missing half the route". They judge against what they're given and the outlined test framework. Of course they can let their opinion be heard, but if the project group decides that it's "by design" then it will be shipped.

QA and external testers simply don't have the authority to criticise a design.

 

"QA and external testers simply don't have the authority to criticise a design."

Good thing that us (ex-)consumers have though, mwahaharr!

 

Anyone can be a tester. You wouldn't necessarily ask the cleaner to test PZB though.... You might ask him/her to switch the game on and play a scenario on NTP, involving a Class 47 and some newspaper vans - and then see if/how.... he/she does it. And if they can't do it, you can ask them what they tried to do...

And they might ask you what all the icons mean - because, quite frankly, I still don't know what all the collectible icons mean, for example.....

And you might then get a perspective of what it is like to be a parent that knows nothing at all about trains, that is helping their eight-year-old child start to play Train Sim World, a child that when they get interested in the game, then wants the DLC for their birthday...

You'd then also get a cleaner that has some idea of what the company does, feels valued by the company, and can talk to friends about it, and encourage other people to buy the game.

But if all eight-year-olds, and their cleaner-parents, are not welcome to play the game, then fair enough - just make that clear on the packaging/marketing, perhaps using the two fingers icon, which should be clear enough for all to understand....;-O

 

As a consumer, you definitely do. Your wallet is quite powerful.

This is true, but you will need to provide guidance. How does the thing you want them to test work, and what should they look for?

I was once asked to help out with testing some reports. While I knew how to run the reports, I knew nothing about the data in them so all I could say is "yep I see a report" but I had no idea of telling if the data was accurate. The numbers I saw meant nothing to me

 

That would be a different type of testing, more of a focus group, they're not looking for bugs in your example, but how the user interacts with the product. If some users can't work out how to start the locomotive, perhaps the manual needs to be clearer, but it's not a bug per se.

If you let the janitor, who knows nothing about AWS, play a scenario and they say it's fine, that doesn't mean a lot. For example, perhaps the AWS didn't sound for a red signal, but the janitor was using the HUD, so he just stopped the train at the red signal and didn't know that a horn was supposed to sound, so he reported nothing wrong with the scenario. Or perhaps he reports the train was really slow, but when asked whether he left the handbrake on, he doesn't know, so now the scenario needs to be played again, checking the handbrake is off to ensure that was in fact the issue.

My experience within most industries is that most people who are not directly involved in the product are not specifically interested in that industry. If they're an accountant at a pie factory, they probably don't care about pies any more than the average person does; They're interested in the accounts, whether it's pies, cars or software the company produces.

I maintain that DTG can't thoroughly test every timetable mode service, it's just not practical to do so. If I was DTG I'd probably test every scenario and then maybe a 5-10% sample of the timetable mode services, running each one early, late and on time along with the full accompaniment of safety systems.

 

Yes, precisely. I am aware of the likely qualifications of a cleaner, and whether they can possibly carry out a systematic array of performance tests on beta software... although that said... you might be very surprised.... I worked in a... let's say... fairly important... office, where the lowest ranking admin guy (filing assistant) was a fairly well respected civil engineer in his country of birth - like - he had built bridges - major ones.

Anyway - my point is that everything about the game needs to be tested, and a heck of a lot of it needs to be looked at through the eyes of people that know nothing about trains.... It depends what you want your market to be, whether you aim to interest a large chunk of 100,000 dedicated railway enthusiasts, or whether you are going for the same group, plus a smaller slice of 100 million more average people. I think you can hit both markets, but only if the software is accessible, which, indeed, OSD was specifically marketed as being, quite ironically..... ;-)

Yes, I think that you are right. And if DTG tested every "Scenario", and a sample of the timetable (or just the first two minutes of each timetable service), that would be a massive improvement from what appears to be... well.... no testing at all.

 

Scenarios get tested end to end, timetabled trains get tested occasionally. As indicated before: it's impossible to test them all under all circumstances, and since the pattern basically repeats every hour you should be good.

The development tools themselves have a way to "simulate" if the planned timetable is even possible. By simulating a timetable, the game performs a full end-to-end run of the timetable using AI trains and checks for conflicts. If none are found, you have a timetable which is workable in theory.

As we have seen in practice, this isn't completely flawless. AI trains behave differently to player trains, which sometimes leads to unexpected results. Therefor you still need Mk 1 eyeballs to look at the product, although it will sometimes be very hard to come across broken services

Some issues I recall:
- RRO: an AI driven freight trains SPADs near Hagen, and ends the session (not sure if fixed?)
- ECW: when driving a semifast from Eastbourne to Brighton, you could get stuck just before Brighton on an eternal red if you left Moulsecoomb on time (has been fixed)

 

Thanks - I think this is a very helpful post. This is how I guessed they tested the timetable, without physically testing the services individually - using the "Scheduler" (which I assumed was basically a table of data... or even a relational database).

Anyway.... do you have any views on why there is a seemingly random success rate with achievements/trophies.....? I think that there is a play-testing issue here - and that there are many routes for players to complete the actions that are seemingly required for an achievement, but that not all of these routes are recognised as the right process in the code. So, if you launch a service in the wrong way, for example, then the relevant data doesn't get tracked, and you don't get your achievement/trophy.....?

Do you have any views? Is it easy enough to step back through the code that leads to an achievement being awarded...? I had assumed it would be really very easy, but I am baffled as to why DTG don't do the checks - and fix everything - it would make a lot of people happy (including children of very annoyed parents....).... ;-)

 

There are the usual bugs in the Oakville DLC as there are in the Baby Bullet DLC (and every other) and more testing would certainly help with some of that but it would not help in the case of the Hamilton Area Industrial Rail Yards Balls-up (or HAIRY BALLS for short). That goes beyond merely a lack of testing and is much more worrying and disappointing. But enough about that for now...

I think people are generally more forgiving of a few bugs that can be ironed out after release than they are of major flaws and long-standing unresolved bugs like the 47 on consoles. That was also probably known about prior to release and as yet no fix has been possible. Some things are going to get released that have errors that seriously annoy some people and others won't care as long as the game is playable, enjoyable and close to being as advertised.

One thing that annoys me (apart from the rubber band powered Class 47) is when a dozen timetable services have been selected for Journey mode and nobody has actually played them to ensure they work. The same goes for the actual scenarios. That's not a lot of testing to be fair and should be done.

The odd sound effect not playing correctly can be annoying but that's not going to get my blood boiling, nor is the wrong type of bell sound, or less than perfect distant scenery (missing water in San Francisco). I don't think any of those kind of bugs would delay a release even if they were detected.

Everyone has different expectations and some people do expect too much. How many bugs is too many bugs? Lots do get fixed after release and others not even then. We can and will vote with our wallets if things get too bad for us.

 

I don't know about that I'm afraid

My guess is that the game expects steps to be performed in a certain order, and exactly as specified.

 

I agree with everyone's comments, sadly a lot it could boil down to the fact of paying someone to do it.
Maybe UE4 isn't all that it should be

 

I'd happily add my name to that list should it become a reality!
Having beta tested for World of Tanks (for free) for a couple of years this would be by far the easiest and cost effective way for DTG to have products checked.
Whenever there's been new tanks or maps we've been given them with the instructions to go and break them in any way we can. There's often been, as a result of such sessions major changes made as well as simple bugs solved.
It does mean I have two versions installed on the PS4 but it's a small price to pay to at least try and prevent rather than cure issues that arise.
I don't necessarily think there has to be a level of 'knowledge' of railways to do so either. What matters is numbers, as the more that take part the more chance of bugs etc being found.
It would I think need some form of discussion platform though in order that cross referencing could be carried out though.

I don't think there's anything I've not read in the thread that doesn't warrant some merit as ultimately the majority of us just want a half decent product rather than the three legged, one eyed donkey we currently have!

 

Exactly how would I drive 450 miles in the Class 45 in a certain order other than doing timetable services? It's pretty much exactly as is specified!!
As synthetic states its sheer bloody luck if any trophies pop, personally I'm not bothered by 'bling' but totally appreciate why some are.

 

As far as I know, signaling works fine in Peninsula Corridor, not sure about others, but I wonder what content was the OP playing that made them create this thread?

If it some old route, then hey, they didn't know, but yet they didn't fix it. Good thing my next planned route is Long Island Railroad and perhaps some British railways afterwards, and not the Canadian thing, if they really did screw it up.But there is competition, but none of them have gone as far as being able to walk around

 

Things are tested before release.

 

And by what facts are you basing this wild assumption?

 

Yes, I am sure that "Things" get tested. Those "Things" just happen to be:

- the local pizzaria's meal deal
- the coffee machine in the office
- whether Dave is going to be upset if he finds that his stapler has been cast into a bowl of jelly

And, although all of the above testing is laudable, I would like someone to test the game itself, starting with the DB BR 155, going through tunnels on RSN, on the PS4.

 

This is really interesting.... So anyway - I have bandied about this idea that maybe it would be nice if people do some testing, before they sell stuff, and I have had some really interesting responses.... some of which I would class as "empassioned resistance" to the idea. And can you guess who are the most passionate about not testing stuff....?

Go on... guess.... Make a stab in the dark of a guess.....

Well - I don't know what cards you've picked up in this magic trick.... but I think the three cards you've got are:

1. 2nd party loco developers
2. 2nd/3rd party loco/route partnership programme developers
3. you (a 4th party loco developer...?)

It's like..."no no no no no no no".... Don't make us test stuff. We might try to make it... and it might not work.... but if we have to take the time to actually test it properly.... then we won't have the time to make more largely untested content.

This isn't good enough for PS4 and XBox. It's absolutely fine for community development (which might involve sales, and "free beta testing" by paying customers) - but not good enough for a finished product on PlayStation or XBox.