[tutorial] How To Speed Up Loading Times For Old/3rd Party Routes

Warning: For experienced users only. Filenames need to be checked for umlauts and hash symbols before doing this! -> https://forums.dovetailgames.com/th...s-for-old-3rd-party-routes.51181/#post-443977

This is a quick to tutorial to generally make use of the new fast loading routine for older non-DTG routes or 3rd party content.

The thing I'm going to explain is how to create your own .ap storage files which will significantly speed up loading time, reduce disk fragmentation and also save storage space.

Your harddisk is made up like a shelf, containing little drawers where files are stored. The Windows NTFS filesystem has a so-called cluster size of 4kb (4096 bytes) for drives up to 16TB, which you can think of as the size of the drawers.

Train Simulator is made up of thousands of files, small ones and bigger ones. Due to the filesystem, even a 500 byte sized file will take up one drawer, meaning 4096 bytes. There can only be a fragment of one single file in each cluster. So having a lot of small files will result in taking up more space than adding the filesizes together.

I'll explain how to save space and by keeping everything in one (or more) big file, the loading speed is greatly improved.

For this example I took the GN Cascadian route, which comes with assets the size of roughly 1.2GB.

 

Step 1 - Get 7zip (https://www.7-zip.org/) - a free fast versatile archiver, and install it.

Open it and navigate to the route's Assets folder.


In there, if you have run the route at least once, you'll find the file blueprints.pak. It is a cache for the game's 3d model geometry files.
Delete it, it will be rebuilt once you start TS again.

 

Step 2

Now select all of the folders.


Now press the "+ Add" button on the toolbar.

IMPORTANT: If you have "_.sbn" and "_.sbn.MD5" files in here like in this example, be sure to select them too otherwise you'll get an effect I call Polygon Madness. They must be included in the archive!

 

Step 3

A window will pop up. Select the highlighted parameters, make sure to select "zip" and compression "0 - Store" as we will not compress the files, only putting them in one big file.



Press "OK". Don't worry about the filename.

 

Step 4

Now rename the Cascadian.zip to CascadianAssets.ap

Although you can choose the filename as you wish as long as the ending is ".ap", I'll stick to DTG's method appending "...Assets.ap"

 

Step 5

Delete all folders (or move them to a backup folder) you have now archived, leaving only CascadianAssets.ap.

 

Comparison:

Starting a scenario before packing on that route took 59 seconds on a normal hard disk.
Using the newly created .ap file the same scenario only takes 17 seconds to load.

The Assets folder took up 1,275,125,760 bytes of HD space.
New size: 1,261,580,288 Bytes, about 14 megs saved.

Use at own risk. You can do the same for the Content/Routes/ folders, but the assets loading is the most time consuming process.

Another advantage of having .ap files is that if you want to use modified files, you do not have to overwrite anything, you just recreate the folder structure outside of the .ap and put your modified files in there. TS will merge them in memory.

Enjoy,

torfmeister

 

And you've just broken all third party freeware routes that depend on the original file structure, I'll wait the extra couple of seconds thanks.

The thing is though since the fastload feature was introduced it sped up ALL of my routes loading times not just the .ap packaged ones, yes ok the .ap packaged routes are a little faster than the other ones but to be fair that's not really comparable either because most of the .ap routes have either partial extractions* from their .ap parent or are third party freeware which has to collate assets from many different source files, as far as I can make out the fastload implementation also (and most importantly imo) made better use of reading data from Sata & M.2 nvme drives, so if you're struggling with loading times still then think about upgrading your storage devices.

* Anyone who uses Armstrong Powerhouse enhancements (including track and weather), third party reskins or community made patches/fixes, and even RailDriver or other controllers will have partially extracted .ap files, if the route was originally not in an .ap file then you will not know what you need to keep and what you can delete, thus rendering all additions I just mentioned broken.

 

This is simply wrong. The file structure is internally the same, the .ap is NOT treated as a folder, it is just a transparent container.
Nothing will be broken. That would mean breaking by unpacking vice versa. Reading your post shows you clearly do not know how TS works.

So if you criticise do it with knowledge please.

You're free to choose what you want to pack or not. It will never break anything if you do it correctly.

 

You know what just comes to my mind. Why the heck am I wasting my time on this forum, time I could spend enjoying the game and keeping my knowledge to myself. I've helped some people, shared some of my experience and then too often have to justify myself because people don't understand what I'm saying.

 

Without wading into the will it/won't it discussion. This is something that is pretty easy to test.
1) Make a backup of your files
2) Convert an asset tree that is known to be used by a 3rd party freeware route to an .ap file
3) Temporarily delete the original files (you can copy them from the backup it you don't like the results)
4) Run the route...if it loads (or won't load), you have answered the question.

I would be surprised if this was the case, otherwise I would assume that you couldn't use any .ap based route as an assert for route building. However, I am not yet a route builder, so have no 1st hand knowledge of these things.

After I've had a chance to do another Railworks folder backup, I want to try doing this more for the "container" aspect of things. By storing the original assets in an .ap file I think it will be easier to keep track of the patches/mods that I have made, as they will all be outside of the container.

I feel like I have had this same discussion no long ago on these forums in a different thread. It is great that we are having discussions about how we can make TS20xx better...
...even if we don't always agree.

 

If that's the case then fair enough, I'm not too proud to admit when/if I'm wrong, this is the first I've ever heard (in over a decade of modding) of an .ap/.zip being an empty container (thus ignored by the file structure) I never claimed to be an expert and if you cared to see it from the other perspective for a moment you'd easily see and recognise why I came to that assumption.

However the rest of my post is based on my own experience and I stand by my comments (correctly or incorrectly).

I never claimed to know all there is to know about TS or zip archives, does that mean that I cannot question any aspect of advice or a tutorial? Ok maybe I could have worded my initial paragraph better.

I think you've missed my point, if you pick an asset whether it's .ap zipped or not then that's the path the route you are building will look for, I never said you can't use an .ap in route building, I was merely going under the assumption that archiving an existing route would add to the file path thus breaking it's link to the route.

I assumed I was conversing (on a discussion forum no less) with an Adult, you are clearly struggling with the concept that people may not necessarily share the same opinion or even knowledge as yourself, as I said if I'm wrong then I'm wrong, no biggie, I'm not upset if that's the case, however this discussion is no reason to throw your toys out of the pram!

 

For TS is does not matter if the files are outside or inside an .ap - the file structure is transparent meaning the paths are still the same.

And like sphexi said, it brings a lot of advantages. You can clearly separate original from modded content.

Have you ever gone into World/Scenario Editor just looking at things without changing something and when you want to exit a pop-up appears, saying "TS needs to save changes to the route."? There is no way to say "No" other than closing down TS in the task manager to prevent it from changing the route.

But if the route files are in an .ap, you don't have to care. TS can't write to that. Press ok, and when you go into the route's folder you'll see that TS places the changed tile outside of the .ap, there's suddenly a folder "Networks\<Loft/Road/Track> Tiles" or "Scenery" which you'll just need to remove to keep the route in its original state.

And the loading speed improvement is impressive, I also ap'd my EU assets (Kuju\RailSimulator, GermanRailroadsRW, EisenbahnwerkRW, Aerosoft\AS_Common), 3GB and if you are heavily into editing and have to switch between game and editors often you'll appreciate that.

It's an advantage also for making backups, copying a single 2GB file takes much less time than hundreds of files the size of 2GB, simply because the allocation table/Journal has to be updated only once instead of some hundred times during copying.

Did not want to offend you Reef. Was just because you accused this method of breaking stuff. And it's a tutorial. You can do it if you want to or be happy without it.

 

So a question
What do you do where you have a Third Party Route that uses assets from multiple providers?
The Canadian route you gave as an example has everything in the one folder - what if they are not?

 

It makes absolutely no difference. You can still partly extract files from the .ap. TS doesn't care if the files are inside or outside an .ap.
And if you have a copy of a file outside, like TS Tools does when extracting RouteProperties.xml from the MainContent.ap, it uses the most recent one.

Think of the .ap as a transparent box. The loading routine that works an a lower level than the game engine loads all of the files it finds in the asset path, unboxes the ap and presents them to the game engine for use.

As for now I have packed all my assets using the method Assets\Developer\Addon\AddonAssets.ap, the same way the official content is distributed. And it's very fast and convenient using the 7zip file manager.

 

As of now I found only two problematic routes: Simtrain.ch's SBB Route 1+Roter Pfeil and Heidi Express.

When putting these in an archive there's missing textures in the game. Extracting again, fine. Looked into the assets and found they broke a golden rule (apart from missing localisation, when TS' language is set to english you won't see any messages in some scenarios).

The golden rule they broke is: Do not put umlauts in TS' filenames (e.g. I_Mühlehorn.GeoPcDx). Inside blueprints it's ok as they are UTF-8.

So I say again: Use at own risk and be careful with Swiss/Austrian/German content.

 

Good news. I contacted my cousin who's an IT pro, and he sent me this powershell script. It checks the current folder including subfolders for filenames with umlauts (ASCII > 127). If it does not output any files you can safely put them in an .ap.
If it does like in this example (SBB Route 1) then leave the files as they are.

Just enter the short script in PowerShell ISE, cd to your Railworks\Assets and Railworks\Content\Routes folder and run it.

I will post a list of DLC that should be left as is.


For CopyPaste, here's the script

Code:

clear
$filenames = Get-ChildItem -Force -Recurse -Name
foreach ($item in $filenames)
{
    $enc = [system.Text.Encoding]::UTF8
    $byteme = $enc.GetBytes($item)
 
    foreach ($singlebyte in $byteme){
 
        $checkbyte = [int]$singlebyte
        if ($checkbyte -gt 127 -or checkbyte -eq 35) {
            $item
            break
         }
    }
}

 

If you get a message telling you that executing scripts is not allowed on your system do the following:

1. Start PowerShell as Administrator
2. Enter set-executionpolicy unrestricted -force
3. Enter get-executionpolicy
4. Start PowerShell ISE normally and run the script

So far all Kuju, RSC and DTG (except DTG\KarlsruheStrasbourgSignals\RailNetwork\Signals\German HV\Indicatoren\DEs_HV_Zs2_2Lß_Oben.bin, so leave KarlsruheStrasbourgSignals unpacked - interesting that DTG left those outside the ap ) assets are fine for packing (most of them are already ap'd anyway). I found no problems in any route folder, as they usually (not always!) do not contain any assets and have strict naming conventions.

And shame on me, with Cascadian I used the wrong example:
G-TraX\Cascadian\Scenery\Buildings\gn_sign-îndex.bin
G-TraX\Cascadian\Scenery\Buildings\gn_sign-îndex.GeoPcDx

(Do not rename them if that's your idea, because the file names are referenced inside those files and the route tiles, you'd run into trouble soon.)

I've attached a list of assets the script found. So the addons containing these should be left unpacked.
Of course you may have a different set of addons. I think Peter has all DLC that's available . I've only got 365 GB...

 

And by the way I'm testing on a separate Railworks copy which has only Kuju\RailSimulatorCore, adding assets one after another to see if I can pinpoint the Consist Editor crash and if it is somehow related to this...

 

Torf

Nice analogy - how does it work when TS starts and it checks the scenario db with many scenario files inside the .ap file structure? IMO this is where fast loading could be very useful. Any new DTG .ap route is installed and the .ap file will have to be read to check the scenario db.
I struggle to understand "-fastload" as once the scenario db is checked (the longest procedure that I see - over 2000 scenarios) - TS loads the main menus etc - but then loads ONLY1 route, 1 scenario + several/few Assets (some .ap some not). Is the reduction in load time significant if you use - fastload? What does it actually speed up?
pH

 

I'm just packing all my Routes to MainContent.ap's, sorting out my own scenarios which should stay outside for further editing. I'm expecting a faster startup procedure.

The -fastload option now is changed to -DisableFastload as it is on by default since an update. It's not only for the startup, and after DTG fixed the Union Workshop assets there is no reason to disable it any more. How it technically works, I don't know. Maybe by loading larger chunks into memory, and the ap format is very suitable for that because the filesystem doesn't have to check the FAT/Journal hundreds of times for the physical address of each asset, which requires a lot of head movements in the drive. Like posted before, I reduced loading times playing a route which loads many assets from 1 minute down to 17 seconds. Perfect for editing and testing.

All this may not make a big difference if you have an SSD. I'm still hesitating to buy one, because for me the better reliability of an HDD is important, and as I do many file operations (which SSD's don't like, especially not fragmentation) I'll stick with HDDs at the moment.

Edit: I'm not sure how the compiling process of the SDBCache works. It seems it's not scanning all scenarios, only for differences since the last run. That would explain why after some time and OOM crashes (the Cache is updated again on normal game exit) some scenarios are missing, which is often reported, especially Quick Drives. A full scan takes very long, delete the SDBCache and see for yourself. Last time I did it it took about 15 minutes. (~4000 scenarios)

 

Torf

This is not the case with the latest SSDs. SSD's used not to like too many writes to the disk as it supposedly shortened the life of the SSD - but nowadays with an inbuilt TRIM command, this would not be a significant problem.
My oldest SSD which is the system drive is 4 years old and because it's on the system drive has the most writes to it - is still going strong.
A SSD with TS installed does not see many writes to the disk - even using the editor so the longevity of the SSD due to writes is not an issue. Reads from a SSD are NOT important wrt longevity.
SSD's do not suffer from fragmentation per se, and the TRIM command frees up unusable blocks very efficiently.
IMO SSD's are far more efficient than a spinner HDD and their reliability/longevity is no longer in question.[/quote]

 

Technology is moving fast. Thanks for the update. My system HDD is 11 years old and still doing well (Western Digital).

Anyway it should be an improvement. And keeping files together, I like that. And with 7zip File Manager seamlessly accessible. Look how Unreal Engine games stuff all their data into one big file. Nice and clean

 

Just found that scenarios containing movie .ogv files like Story of Forest Rail crash when packed. The internal movie player which seems to be a separate exe/dll apparently has no access to the .ap.

Confirmed by the fact that the two tutorial missions containing videos for Canadian Mountain Passes are installed outside the MainContent.ap by default.

So, videos or filenames with special characters must not be packed.
You can check for videos by opening a command prompt, cd'ing to Railworks, then typing
dir /s *.ogv

https://steamdb.info/depot/222582/

 

The hash (#) in a filename (which should never be used as some file systems treat this as a numeric wildcard!) gets also treated differently when inside an archive. I've updated the power shell script to report them too.
Found those in UKTrainSim's QiLian Mountain Route.

 

Update: Oovee's Class 57 Freightliner and Class 57 Rail Tour must also not be packed, as the script needs write access to store settings.ini in the corresponding asset folders. Otherwise the script won't work and the locos are inoperable.

 

Kuju\RailSimulator\RailVehicles\Diesel\Class166\Default\Destinations\destination_#.TgPcDx must be present outside the .ap, too.

Edit: Texture is not used, so this is not important as the destination board is not implemented anyway on the default 166 (Oxford-Paddington)

 

I know it's an old conversation now but I will try. I was intrested in the amlauts and hashes you wrote about. Here in Czech Republic, we frequently use letters with diacritics (like š or á) and sometimes these appear in file names. Is that also a problem? Shall I not pack them into .ap? And what about for example &?

Is there any better option than just leaving these folders alone? For example put the folder with the problematic file outside of the .ap and pack the rest of the folder?

 

Many file readers cannot handle & so in the USA where many names are xxxx & yyyy Railroad they tend to put the letter n instead of the & example the Boston & Albany would become the BnA.

 

Okay, thank you. And what do you think about symbols like -, + or =?

 

I would avoid in file titles

 

And if I just put the problematic files outside the .ap, will they work? (of course I would recreate the path of the file next to the .ap)

 

Yes it's only when the "problematic files" are archived in .ap's that they then become problematic.

What do you mean? The file path will remain the same, think of the game engine looking at the files in a top down view as though they are layered.

Bottom layer = the core game files
Middle layer = .ap archived content
Top layer = raw files and folders in the same location and file structure as the .ap archives

The game will see the core files unless there is a layer of .ap's hiding that core file then it will see them instead, if however there are raw files and folders then they would layer over the .ap's and the game would see that content over the .ap's and or the core.

Edit: Re-read your post again after I posted, you mean duplicating the file path i.e. having the same file structure that's within the .ap archive? .... Yes that's correct. Sorry I was thinking at first you meant altering file paths in .bins or .xmls to direct them to your files, my bad..

 

Thank you for your insightful answer. Yes, the final paragraph is what I needed to know.

So if I understand correctly, I don't even have to delete the problematic files from the .ap if I recreate their path next to the .ap because the game will prefer the files that are outside the .ap? Is it enough to just put the two or three problematic files into the path next to the .ap or shall I recreate the whole folder those are in?

And (I'm sorry for my stupid question) how do I practically recognise, in which layer I should I put the .ap? Because sometimes there are more complicated paths like: Assets - CS Addon - JTovacovsky - Vozidla - Rail Vehicles - passenger - CZ_CD_Apmz -... for example. In here, Rail Vehicles is the layer I should zip right?

 

That's correct, but I'd personally remove them anyway as it's just cleaner and good housekeeping.

You only need the files (in their correct folder path so you need to create all containing folders as they are set out but only the ones that ultimately lead to your problematic files, there should be no empty folders), duplicating the whole file would just be consuming valuable HD space unnecessarily.

RailVehicles should be the first level of the file structure within the zip/ap, consider the zip file an invisible container that the game doesn't see so it ignores that as being part of the actual file path.

Using your example your archive will contain everything within the Vozidla file.
for example:
raw format = \Vozidla\RailVehicles\passenger\CZ_CD_Apmz
archived format = Zip named Vozidla.ap and inside that will be - RailVehicles\passenger\CZ_CD_Apmz

So usually and as your example shows after the Assets folder the DLC name will be the first folder (the root folder for the DLC/Mod if you like) - CS Addon, within that will then be the mod authors name (this is usually the common and expected way across the TS DLC/Mod world but there are exceptions) - JTovacovsky, then within that is the rolling stock/assets name - Vozidla, everything inside this can be archived (assuming it obeys the rules Torfmeister laid out in the original post) by right clicking the Vozidla folder and clicking on 7Zip > Add to archive, make sure the zip settings are then set to zip and normal deflation and then click Ok, you can then rename the zips filetype from .zip to .ap (you can do this within the settings where it shows the file path in the first box but you will then have an archive named Vozidla.ap.zip and still need to remove the .zip part from the end once created anyway, so it's easier just to leave the file path as is and rename the .zip to .ap once the archive has been created).

 

Okay, thank you so much for your time and patience. That finally is all I need to know I hope it helps other people as well.

 

I followed the steps here to the letter in order to try this out but when the quick drive scenario loaded (The Falmouth Branch in this case) much of the scenery and all of the trackwork was missing. Any thoughts anyone?

 

It does state (for experienced users). I tried this before on a purchased DTG and it too caused problems. I would just resync from Steam and let things happen through existing code. Sync with Steam usually brings in the optimized .ap file. Problem is 3rd party reskins etc which wish to leverage this code and can leave behind artifacts which cause problems. If you then delete the wrong stuff - guess what - your links to existing assets have also been affected!

 

I was able to restore things as It was all backed up.

 

An *.ap file is a fixed size - if you Delete or change it Steam sees it as a Faulty File and replaces it.