Reduce Ue4 Stutter By Introducing Pso Pre-cache

Note: Content in this post has been sent to DTG via support ticket #71423 (not publically visible) and is pending an official response. I'm posting here to get second opinions from the community and alert me of any mistakes I might have made in the post. Also, share some knowledge I got from the trip

I recently ran into this Steam review that got a dev repose (See attached screenshot or ThisLink)

in which dev "dtg_dj" states that "...UE4 suffer from shader compilation stutter......a second run of the same content will be smoother......because shader is cached when it first appears and allows for reuse......thus much faster than compiling on-the-fly at first appears...cache size is limited will be overwritten by newer data".

This response is mostly correct but it seems that no further technical measures will be taken or planned from the DTG side to reduce or eliminate the stutter by reading this dev response. (Maybe DTG have plans but didn't state them, I'll give DTG the benefit of the doubt)

BUT, there are some things that DTG technical-wise can and definitely should do (also apply to any other UE game devs):

• PSO Pre-Caching!

PSO is the abbreviation of Pipeline State Object which encapsulates all the states required for the rendering pipeline in DirectX12 and Vulkan. This includes shaders, vertex format input layout, blend states, rasterizer states, depth-stencil states, pipeline configuration and other GPU settings.

PSO Pre-Caching involves gathering and storing a PSO list during the development phase and compiling them on user machines, allowing the game to reference these objects at runtime without the need for on-the-fly compilation. This can significantly reduce latency and stutter caused by "cold/unwarmed shaders", as the necessary rendering states are already available when needed.

• "I seem to have heard about 'shader pre-compile' or something?"

Shader pre-compile has been used on multiple DirectX12 games to improve performance in recent years.

PSO Pre-Caching encompasses shader pre-compilation but also includes the entire state configuration necessary for rendering. By caching the PSOs, developers can avoid both shader compilation and the overhead associated with setting up the rendering pipeline dynamically.

• Technical Feasibility Research

PSO gathering and pre-caching were formally introduced in UE5.0, first automated in UE5.1 (Automated PSO Gathering) and further improved in 5.2 (DX12 PSO Precaching) and 5.4 (Programmatic PSO Precaching).

It seems that UE4 doesn't have that technology introduced by default back in the day, but it's possible to backport this feature into UE4 as proved by a recently released UE4 game.

Outcast - A New Beginning is a UE4 game released on 15 Mar 2024, with a massive open world and jetpack fly gameplay (high-speed scene loading), it has backported PSO-related features proved by having a PSO Pre-Caching loading screen on startup. This game is visually excellent without noticeable stuttering, unlike in TSW, by gathers and pre-caches about 32k PSOs on startup.

This proved that UE4 games like TSW can do the same and would effectively reduce stutters caused by generating a new PSO on-the-fly. The PSO cache generated in this way is not subjected to limited size and thus won't "be overwritten by new data" as stated in the "dtg_dj" response. The PSO cache will regenerated when the game has an update that changes the PSO gathered list (material BP creating or editing) or GPU driver updates.

• Conclusion

Incorporating PSO Pre-Caching in UE4 can significantly alleviate micro-stutter issues, providing a smoother gameplay experience. By understanding the root causes of stutter and leveraging the capabilities of PSO cache, developers can enhance performance, allowing players to enjoy Train Sim World without interruptions.

 

I couldn't agree more. There are options for DTG to vastly improve this situation but they don’t seem inclined to take them sadly. You would think a large annual release would be the perfect time, but no. Saying, ‘the first time you play any of our products the experience is terrible but it might get better the more you play’ is an odd marketing ploy, however you look at it!

I really really hope that next years annual release sees some movement in this area.

 

This + DLSS 2 or 3 + DX 12 AS native would be a good start

 

100%, why are DTG so reluctant to actually improve the core of the game, does it help that it’s a pc/console game, would things of been different if the pc version remained totally separate from the console versions or should that make no difference?

 

I don't mind the "compilation stutter" during the first run, I'm convinced the "ride as passenger" feature was created with this in mind
The micro stutters kill the immersion, though.

 

@George C. Barnes Do you know if the .ini setting D3D12.PSO.DiskCache=1 and D3D12.PSO.DriverOptimizedDiskCache=1 allow the PSO cache to be enabled and work well in TSW?

 

Sadly DTG don't develop their builds in DX12 natively yet, apparently it doesn't work properly and hence why they enforce DX11 as standard and do not include an option for the user to change it on PC outside of the Steam start line change.

 

That's disappointing but not very surprising tbh.

Does your source have any info on why there's no DLSS/FSR implementation in TSW as of yet?

 

Create a forum post about it and see if you get a response.

 

If I'd have to guess, it's because they don't have enough competent game programmers, if any. And I don't mean that in a bad way, I don't want to bash the skills of anyone from the team. Clearly they have content creators, 3D artists, route builders, etc, but it quite often feels like the whole team is just that - content creators, but no software developers.

Surely there must be someone, but they are either too few in headcount or don't have enough experience. I mean the core software is full of issues and they are getting fixed in an incredibly slow rate. Quite often improvements are only introduced by community effort, like TOD2 or when people kept spamming them with Unreal Engine volumetric clouds tutorials.

I don't know the internal structure of DTG, but it would be nice if they spent a bit more resources on the core software. Train your existing devs more and/or get more experienced devs into the team. Just like with the routes themselves, there are so many things that could be easily improved with a bit more time and experience.

 

The consoles have their own core development issues too, I think in general they just lack a real interest in the core.

 

I think this could be a big part of it tbh.

I don't want to misquote anyone but i swear i remember Matt referring to "the programmer", or "the software engineer" in response to a forum question recently.

As in a singular person.

IF that's the case, it could explain why core development takes so long.

I don't know what industry standards are on staffing ratios of programmers vs artists are, and a sim dev is probably a little different enterprise anyway, but i'd love to hear about it from someone with real experience in game development.

 

Going by previous public comments on this forum, I'd confidently say the core development team is smaller than the Community team.

I also believe a lot of the team that knew how to code UE4 were released or reallocated to other projects with the Focus group.

 

I don't think there's a standard. It simply depends on the project.

The team my brother works with at the moment consists of over 400 people, I think, split between programmers, artists, etc. No idea what the ratios are.

Edit: forgot to say, it changes depending on the stage of the project, as people drop out and move onto other IPs.

 

It is quite common that an issue is given to just 1 person, or at least just 1 person is responsible for fixing it.

They probably would benefit from having more people in the coding department, but also don't think there is just one programmer because Matt said that phrase.

 

You're right there. There is definitely more than one programmer at DTG. I don't think they'd get anything done with one programmer.

 

Fair enough

 

Sadly their priorities are for compatibility with existing platforms to sell dlc's, not the few with high end machines.

 

Thing is, with the mass adoption of the newer 30 series and above Nvidia cards, DLSS should really be a priority of the development team to implement, as it allows them to do more with less. With both the PS5 and XSS/XSX using FSR 3 from AMD, they really should be looking at frame generation, however yet again they need to make sure the game runs on the older generation of hardware, instead of growing a pair and saying that these are no longer supported.

The fact we are still waiting for DX12 support whether it be BETA or "use at your own risk" within the game menu's is a sorry state of affairs. I think TSW on PC is one of only a handful of titles running UE4 that doesn't have the option to toggle DX12 in it's menus.

I wouldn't be surprised if TSW 5 is the last one you see support Xbox One and Playstation 4, with TSW 6 being for current generation and the mid life upgrades to them.

If you announce now, 12 months out from the next big build of TSW that you aren't supporting previous generation hardware, people can't really complain about not being told, but as we know DTG won't do this, as it's all about profit over anything else.

 

I can't understand why DTG doesn't consider upgrading the engine to 4.27 to add DLSS3, even if they stay on 4.26.1, they could still choose to introduce DLSS2 and make it available for players to use.

 

I'd go as far to say they don't want the headaches that upgrading the UE4 engine brings with it. I think it would break too much stuff that they don't have the resources to fix. I personally believe that every TSW version, should be updating the UE engine build to make sure that it's the most optimised version with the newest features, but again its cost vs output arguement.

 

I also saw the reply to Stream review and it lacks a lot of information. Even if PSO would help the main problem we still have is that game uses real-time shader compilation, not compilation before loading game, as other Unreal games do. Those other games are not suffering this issue. So this is not a problem of Unreal Engine but a the result of how you implement your game. Real-time is easier to implement as user system will compile shaders on demand when they are required and you don´t need to care about that. You just pass the efforts to user system, in plain words. Instead of that developers should list the needed shaders to compile (the shaders included in each DLC), so that systems could compile them in advance. They can do it and process is documented. So there´s no technical limitation that prevents this to happen.

On the other hand the proposed mitigation of running a route once to generate shaders is only working while GPU drivers are not updated as new drivers version will make old shaders invalid, so you will need to run all routes again with all rolling stock (as rolling stock also includes shaders, not only routes) to generate shaders again. Basically the problem will last forever until devs acknowledge the only way to solve it needs efforts on their side and they follow the path other studios followed to prevent this problem. Indeed this is not a technical problem but an engine feature that needs to be used correctly to prevent that it results in a problem for users.

Cheers

 

Hate to mention this but, Bus Simulator 18 & Bus simulator 21 on the same UE4 engine does not perform this badly unlike TSW does. There must be a way DTG can solve this as the devs for BS18 & BS21 have done so brilliantly.

 

BS18 & BS21 have pretty terrible texturing, light sources, physics & render distance. Everything that Astrogon aimed for with those titles are completely different to what DTG has aimed for. TSW isn’t the most realistic environment but it’s leagues above the cartoon world of Bus Simulator.

Never mind the fact that you have to contend with much more intensive assets (trains loaded with simugraph) running at much higher speeds than the rather basic busses in BS.

 

The games don't work in the same way. The Bus Simulator games are likely using pre-compiled shaders as the maps are relatively small square-shaped worlds where it would be easy to do so. While pre-compiled shaders may be possible within TSW, I think they'd take a long time to generate to the point where it isn't worth it.

I'm not sure what Bus Simulator game you've been playing, but the one I have isn't like that at all. The lighting is a damn sight better than TSW's ever has been. In most cases the textures are better as well, and the night lighting is too.

Granted the physics and simulation don't make it an OMSI 2 competitor, but that has graphics that were ten years out of date when the game came out.

The only major things I'd do to improve Bus Simulator would be to tidy up some of the worse areas of the map and to add extra antialiasing options... That goes for TSW as well, but there's plenty more wrong with TSW beyond antialiasing.

 

I was talking about BS18 & BS21 by StillAlive Studios which uses the UE4 engine which TSW uses today. Let me rephrase my question, I was talking specifically about the stuttering in almost if not all routes at the beginning of a run, not to mention that some suggest to pre-run the route end to end so that the shaders are then cached only for the stuttering to ease off. Please do correct me if I’m wrong, since I own both BS18 and TSW respectively and considering that they use the same engine UE4 albeit two complete different games and that UE in-general is quite demanding, both games should run near perfect? Unless the TSW devs are beefing up LoDs, graphics, etc which is why this is only happening with TSW games. All I’m trying to say is, why I play BS18, the frametimes are very solid whilst TSW, constantly changes

 

The games work in different ways and since the lighting is compiled on the fly you should expect slightly worse performance in TSW. However, TSW runs a lot worse than it really should regardless of situation which is a large concern with the game. To be fair, Bus Simulator 21 runs worse for me than it should, which does leave the possibility that the lighting is also not precompiled. I doubt this though.

That's unless I'm misreading your post and you mean the games could work together to improve performance somehow. If this is what you mean, then no, it can't work like that.

 

Since there is a way to further address the lag issue, why doesn't DTG adopt this approach?

 

Sorry! I personally think that the inconsistency of everyone's computer hardware makes DTG unable to apply this solution. If everyone's computer has high-end CPU+GPU hardware, I think there should be no problem. In the final analysis, it is the computer hardware that makes the DTG game unable to run, hahahaha

 

PC settings are scalable so you only cache what your setting are set at .
The inconsistent driver performance for your individual GPU is more of an issue.
Tried six different drivers before I felt satisfied,I was getting the best from my hardware.

 

The level of stutter is down to the coding of the UE4 engine to handle many different factors, yes hardware configurations will play a part in these stutters, however if the stutters were a problem in every UE4 engine game, then you could blame the engine, but with some absolutely stunning UE4 games, that use pre-compiling shaders you see hardly any issues with stutters, so to say that TSW wouldn't benefit from this and that the issues are down to hardware I find that hard to accept, PERSONALLY!

To help mitigate stutters and performance issues, DTG would be better off investing some time in learning FSR and DLSS and implementing that into their UE4 code, so that the GPU can help process the load better by offloading it. PS5 and XSS/XSX would massively benefit from adding FSR as they are natively supported via the chipsets, and with 30 series GFX cards in PC's becoming the normal now, DLSS would create many benefits for the PC market, it's a win win.

The loseless scaling app isn't a good replacement for natively supported DLSS by the way.

 

Since years we spoke about DLSS etc.
But they don't give a f... Because it's not a source of revenue for them...

 

It was a year or two ago when they mentioned the "stutter fix" by playing the route for a second time, and back then, I thought, "Great... that's a job done for them, now they'll ditch searching for an actual fix." I'm sad I was right. I am not giving them the benefit of the doubt if they use this workaround as an official review reply with no mention of improvement being worked on. They've simply cut yet another corner.

 

Yup, this is pretty much the crux of every issue in TSW.

Yes, the majority of games on the market only chase revenue, but at least the devs/publishers on most of them have the sense to see that they need to implement changes that don’t directly increase revenue.

The industry is in a shambles with the release now, fix later behaviour, but DTG can’t even abide by the latter in most cases.

Surely at some point TSW is going to get to a point where it’s entirely borked & the issues can’t be ignored, but are DTG really waiting until the death knell to start addressing core issues?

 

I think that's an understandable explanation.

And honestly I think time is better spent on other optimalizations that this. This is mostly an issue on the first run, while other performance issues will be present all the time.

The game can sometimes halve the framerate depending on what you have on screen.

Other than working on core software optimalization, I think it would be also worth checking to see what assets cause performance issues. A single unoptimized asset can completely ruing performance when used in great quantities.

Like those boxcars on Clinchfield that result in big framerate drops (which has been ackowledged, but never fixed), or the old performance killer grass on Sherman Hill.

 

Now this might be a very stupid question from someone outside game creation, but shouldn't you be able to exploit the fact that the game is literally "on the rails"? Unlike with many other games, you know precisely (at least in scenarios and timetables) what needs to be loaded next, what needs to be compiled, or where the origin needs to jump (why does it need to jump?). Couldn't it smoothly prepare the next tile/shaders, knowing it will need them in 5 minutes? In most other games, systems need to be ready to load in any direction at any point in time; here, it is just one direction, and speed limits/red lights/schedule dictate when you get there.

 

Remember that you can walk around outside the train, and also that there are multiple external cameras.

 

For me the old routes works as always before. It doesn't looks like the game problem, it's more likely the new routes problem.

I'm willing to put up with micro-stutters if they stop causing sound stuttering.

 

Of course, there are exceptions (although walking is so slow that it shouldn't cause much trouble), probably free roam would also be unable to benefit from it. But most of the time, we're in the cab following instructions. So yes, you could still end up with stutter situations, but 95 % of the time, it could be a solution (that is, IF it's possible).

 

Ok I did have to toy around with this (in UE5) as it seems very interesting and the concept is pretty straight-forward. First, there's the PSO Precaching that happens at runtime. Basically when you run the game, PSOs get compiled when a particular asset is loaded for the first time and then stored in the cache so the game doesn't have to stutter next time this specific asset is loaded.

It's easy enough to gather every single asset at startup and load them so that the cache gets generated and this makes a good difference in Lyra (the Epic sample project) but you do have to consider the number of assets when doing so. Lyra only has around 16000, which is pretty manageable but if you look at TSW (Get Asset Registry > Get All Assets > Array Length), even my slimmed-down copy with just 20 DLCs comes up to 234861, which is a lot more and I wouldn't be surprised if someone people could get over a milion assets.

Second available option is generating PSO cache files as a developer (by running the game with debug options), processing them and bundling them with the game however this isn't great if you have user-generated content as it obviously wouldn't be included. The cache files would also probably be pretty sizable (haven't tested anything in this regard) due to the amount of content.

So yeah, the second option seems pretty viable compared to loading everything on startup.

 

Ups. Didn't see Matt's reply and I think I do agree with what he says. Origin-shifting is necessary due to floating point precision decreasing with distance and it has to happen within a single frame in order to not impact other parts of the game. And that's the main thing here

There are engines out there that work with Double precision instead of Float and (in theory) don't need to do any origin-shifting but that's a very very custom thing that shouldn't be done on an existing game project. Notable examples would be Edge engine for DCS and the custom version of Cry Engine made for Star Citizen.