Just Trains Common Library Now An Ap File

Discussion in 'General Discussion' started by Cyclone, Jun 11, 2022.

  1. Cyclone

    Cyclone Well-Known Member

    Joined:
    Feb 29, 2020
    Messages:
    3,853
    Likes Received:
    1,324
    I just want to throw out a quick note on something that appears to have been quietly made public, but I've noticed in a couple of SteamDB listings.

    The Just Trains Common Library files used to install with a number of routes independent of each other, meaning that it was possible some routes would have different Common Library files installed. The Common Library app only bundled with one or two routes, and every time a route installed without that bundle treatment, you would see a new CommonLibrary folder with a new batch of files in it. From what I can tell so far, this is no longer the case. It is now possible to have the entire Common Library - which combines files from every Just Trains route on Steam - installed on its own without any route installed. The beauty of this is that you can now buy almost any Just Trains route on Steam and, provided the Common Library of tracks, scenery, signals, and other elements is used, you will get the entire library. Additionally, now that it's an AP file, elements will load faster; it might be recommended, in fact, for people buying routes from Just Trains directly to even keep this version of the Common Library installed and take advantage of the benefits of faster loading of assets within AP files, thereby speeding up the game load on a slower system.

    How do you do this? Buy a Just Trains route on Steam and enjoy it as you normally would (note that ECML North, while bundling the library, still installs separate files to the folder that I believe can be deleted), then when finished with it, go to your DLC manager and uninstall the route. Do not, however, uninstall the Common Library. Otherwise, if you later buy another Just Trains route that uses it, the Common Library will NOT auto-install itself a second time upon being removed; you will have to check to have it reinstalled separately (it only auto-installs the first time with the first route using it as it's a new app in your library that one time). So I would recommend uninstalling the route and not the library itself. Pictoral explanation:

    upload_2022-6-11_9-48-23.png

    Notice that Southwestern Expressways to Reading is uninstalled in the above DLC management image. However, the Common Library stays installed. This way, if I later decide I want to acquire the Metropolitan Line, I will not have to dig through and recheck the Common Library as I've kept it checked. This also means those assets remain available even with no other Just Trains routes installed, meaning the common assets can be utilized on custom routes in the Workshop with a requirement of pretty much any single Just Trains route. Note that assets from specific routes, like the ChilternMainLine folder (London-Birmingham, London-Aylesbury, Metropolitan Line) or MMLAssets (Sheffield-Derby, Derby-Nottingham/Leicester, and JT's direct Nottingham-Lincoln which may come over at some point given they've stated themselves it should be, as well as their yet to release Hope Valley), would still require those routes, however, and the Common Library will not provide those specific assets (though it still bundles with the routes in question and is needed for track, signals, etc.).

    The Common Library presently cannot be acquired for free, but I would actually recommend at this point that Just Trains themselves perhaps make this version of the Common Library available directly on their site (if possible) as a separate install included with each relevant route to make life a lot easier for anyone trying to acquire custom route assets and provide a consistent front for all customers. The benefit of this is that each route install would itself be much lighter and much cleaner for the user as those common files can be scrubbed from their own route installs as well to avoid having file duplication. It is well known that Just Trains routes on Steam can often have issues, and this is an excellent first step to resolving duplication issues, otherwise, issues can still remain if you have purchases directly from JT bundled with some from Steam as the directly installed files will still take precedence over the ones in the AP file I reference. This can cause graphical issues at stations and other places where files do not match.

    If you have any direct installed routes, I still suggest wiping the Common Library and using only the AP file. However, you can do an alternative; you can unpack the AP file into the Common Library folder every time you purchase something directly from Just Trains. This can be done by opening the AP file in a package manager.

    upload_2022-6-11_10-9-15.png

    You can then select everything and drag it from the archive to the CommonLibrary folder where the file is located, at Assets/JustTrains/CommonLibrary. Overwrite if prompted. This way, your Common Library files will continue to work with all routes seamlessly as all files will be unpacked for use. You should still get the AP file loading benefits, as well, for any routes installed from Steam, though items installed from JT itself may still be a bit slower since the route files themselves (in the Content folder) are not packed directly like they are on Steam, and literally can't be since many provide bonus scenarios using third party stock that DTG cannot independently support in their AP files.

    If you see any graphical issues while playing any Just Trains route, make a note of where the issues occur and ask here. I've become good at figuring out what assets are involved and may be able to give you directions on what files to move where to make it work. It may not be a Common Library file, but perhaps something else that needs fixing. I'm sure folders like the ones I mentioned above will be fixed over time, but for now, we still need to figure some of these out until repairs occur. :D
     
    • Helpful Helpful x 4
  2. Reef

    Reef Well-Known Member

    Joined:
    Jul 27, 2018
    Messages:
    2,806
    Likes Received:
    1,817
    I can see this causing problems down the line.
     
  3. torfmeister

    torfmeister Guest

    I've mentioned this in a few threads, Cyclone. The CommonLibrary.ap should contain everything for each depending route.

    To avoid trouble, move all other loose folders next to the .ap to a backup location - the CommonLibrary.ap should contain all files and be the only file in its folder, having other leftovers from earlier installations might explain the crashes people report (Remember UnionWorkshop's GEEastAsiaSceneryAssets.ap?);)

    Same goes for the GreatWestern.ap

    So at the moment the only safe method for installing JT stuff is to

    a) Select a dummy installation path for the JT installer, then check the files and copy them
    b) For Steam, before downloading, I rename the RailWorks folder to RailWorks1 and create an empty RailWorks folder where the installation goes to. Then check again. This saved me from issues (I have a lot of JT stuff, mixed Steam and JT downloads.

    Still JT has a lot of shared files which are not unified - but I think they're working on it - think of the Kyle Line Coaches in SWEX distribution (3kb .ap) - a sign they've lost oversight over their own files.

    In the end the user should not have to worry about all this...

    But I can see how it came to this. They have this huge rail network, cut out areas to sell as DLC and just distribute the assets this area needs. With the new CommonLibrary.ap they're going the right way - it's now the same file for everyone. But as it is not like the DTG file hierarchy (bundle>package>app) they will ship this file with each of their routes - which is ok if it is always the same, and download bandwidth is not important anymore.

    The bad thing is and that happens, if you install an older JT release and it overwrites some of your newer assets. (Happened with the HST folder, I couldn't be bothered to compare each file but I encountered the same files with different filesizes)

    DTG have a clever way of file bundling and distribution, you mostly have one route folder and one asset folder per DLC. Shared files like the ELAP are linked to the package as an app and can therefore be checked for presence. Very clean.
     
    Last edited by a moderator: Jun 12, 2022
    • Helpful Helpful x 2
  4. Cyclone

    Cyclone Well-Known Member

    Joined:
    Feb 29, 2020
    Messages:
    3,853
    Likes Received:
    1,324
    One comment. Some routes are not sold on Steam. I have not verified any files in exclusive routes yet.

    Also, I would suggest backing up what is already installed. I rename JustTrains to something like JustTrains1. That gets the new install to the right folder instead of a dummy. I am sure, based on the SWER situation, that they are going to work to get these figured out now. So in the end, GreatWestern and HST will be smoothed out. Separate apps? No idea. Mark1Coaches was the SWER situation.
     
  5. torfmeister

    torfmeister Guest

    I suggested moving them to a backup location ;)
     
  6. Cyclone

    Cyclone Well-Known Member

    Joined:
    Feb 29, 2020
    Messages:
    3,853
    Likes Received:
    1,324
    I thought you suggested a dummy location for the JT installer. Running the installer again would basically uninstall the dummy, and having to run the installer thrice instead of once with some folder renaming of what is already there.

    But no biggie, same idea in mind: have a backup folder in place.
     
  7. torfmeister

    torfmeister Guest

    Same idea, but I think you misread my post.

    - A backup folder to move additional folders and files from the CommonLibrary folder
    - Dummy folders for installation, well that's just my method to see exactly what files are being installed. From there I copy them manually to the RailWorks folder and check what is being overwritten, so I can decide to keep a newer file that would be overwritten by the installer. Also to check for duplicate loose files that are already in one of the .ap's, therefore not installing them.

    This takes some time, but saves me from crashes. All my JT stuff is working.
     
    Last edited by a moderator: Jun 12, 2022
    • Helpful Helpful x 1
  8. Cyclone

    Cyclone Well-Known Member

    Joined:
    Feb 29, 2020
    Messages:
    3,853
    Likes Received:
    1,324
    We have the exact same idea. I am researching some file information right now doing exactly this and making a copy of the results externally for comparison. I can then uninstall it for a later test in another folder.
     
  9. torfmeister

    torfmeister Guest

    The CommonLibrary.ap just got updated again.

    Might be a good idea to clear the cache now (blueprints.pak was not deleted by update).
     
  10. Cyclone

    Cyclone Well-Known Member

    Joined:
    Feb 29, 2020
    Messages:
    3,853
    Likes Received:
    1,324
    Oh, hmm. Did some other files get added? Though it's down to 3.86 GB, so more likely files got removed.
     
  11. USRailFan

    USRailFan Active Member

    Joined:
    Jul 2, 2018
    Messages:
    556
    Likes Received:
    152
    Isn't it just compressed?
     
  12. USRailFan

    USRailFan Active Member

    Joined:
    Jul 2, 2018
    Messages:
    556
    Likes Received:
    152
  13. Cyclone

    Cyclone Well-Known Member

    Joined:
    Feb 29, 2020
    Messages:
    3,853
    Likes Received:
    1,324
  14. torfmeister

    torfmeister Guest

    No. It is bigger. Remember a GB is 1024MB, a MB is 1024 KB. A KB is 1024 bytes... the .ap is bigger now.
     
  15. torfmeister

    torfmeister Guest

    No. The ap file is a zip with zero compression, just a bundle of files like a UNIX tarball. It increases loading speed instead of allocating hundreds of loose files for read access which is slower. Compression would increase loading times.
     
  16. Peter Hayes

    Peter Hayes Well-Known Member

    Joined:
    Dec 9, 2016
    Messages:
    814
    Likes Received:
    328
    I thought that the .ap (NOT AP) file structure was so that the Checksum algorithm needed to check only one file for corruption/changes etc rather than the thousands of files it had to check previously and that obviously increases loading speed as Torf says above.
    Once the checking process is complete then TSC will still have to access the individual files in the .ap file structure (and/or modified files outside the .ap fs - first).
    AFAIK - Zero compression means that TSC is reading, loading, and utilising the files "faster" by NOT having to uncompress them first.
     
  17. torfmeister

    torfmeister Guest

    Of course - only needing one checksum to verify a DLC is also an advantage.
     
  18. torfmeister

    torfmeister Guest

    So I've bought ECML Newcastle - Edinburgh in the sale, and what happened? It installed an unpacked version of the CommonLibrary - with slightly different files, I found a street crossing proxybin to have different content than what is in the CommonLibrary.ap, also bogus Mk1Coach xmls without bins, a MK1_BSO GeoPcdx with different filesize than in the Mk1CoachesAssets.ap - instantly deleted all that to keep Mk1CoachesAssets.ap and CommonLibrary.ap to be the only files in their folders.

    JustTrains should seriously source the CommonLibrary out to a seperate app which only gets installed once and be the same for every route, like the ELAP for example.

    How is an inexperienced user supposed to cope with that, even DTG support can't sort this? Explains all the mostly negative Steam reviews for JT routes.

    Might check MML Sheffield - Derby - which has only negative reviews... makes me curious :). Checking SteamDB again a loose assortment of CommonLibrary files next to a proper MMLAssets.ap, so again to avoid trouble delete all of the loose files and folders and you're good to go if you've got the latest CommonLibrary.ap.
     
    Last edited by a moderator: Jun 24, 2022
    • Like Like x 1
  19. torfmeister

    torfmeister Guest

    Another one.

    Installing (Steam) MML - Leicester Derby Nottingham BEFORE installing MML - Sheffield Derby will lead to missing assets as a smaller MMLAssets.ap will be installed. (missing 78 MB of assets).

    Probably no problems when buying directly from JT (no .ap, so only new files are added)

    In detail installing in this order will remove:

    Everything in MML\PreLoads
    MML\Scenery\Clutter\BikeRack and TapinMachine

    some files in Scenery\Structures\Textures
    Scenery\Stations\Textures\
    everything related to Alfreton.
     
    Last edited by a moderator: Jun 24, 2022
    • Helpful Helpful x 1
  20. Cyclone

    Cyclone Well-Known Member

    Joined:
    Feb 29, 2020
    Messages:
    3,853
    Likes Received:
    1,324
    I'm going to add on to this. Check the route markers file (Sheffield-Derby.dcsv) with Nottingham to Lincoln. I do not believe it's updated at the moment. The one available directly from JT is complete, at 7.63 KB. If you find it's not yet up-to-date, then the one directly from JT can be installed over it in the interim.
     
    Last edited: Jun 25, 2022
  21. torfmeister

    torfmeister Guest

    I was on a shopping spree - also got the new Nottingham - Lincoln line. MMLAssets.ap grown to 1.477.685.276 bytes. Up to date now.

    So if you're out to purchase JT stuff from Steam - be sure to install in order of release dates (visible on the Store page). Until they've unified all their assets.

    Btw. RouteMarkers\Sheffield-Derby.dcsv is 5.711 bytes and has a corrupt xml structure, it is missing some closing tags. It contains 34 items (numbered 2-35).
    Before the last line two closing tags are missing!

    </cCSVItem>
    </CSVItem>


    Also after "Spondon" there's a </CSVItem> tag instead of </cCSVItem>... which closes the array prematurely thus not showing all route markers.

    Holy moly. Everywhere I find bugs....
     
    Last edited by a moderator: Jun 25, 2022
  22. Cyclone

    Cyclone Well-Known Member

    Joined:
    Feb 29, 2020
    Messages:
    3,853
    Likes Received:
    1,324
    The numbering ends at 35? Correct; the list stops at 35 in the included one, at Radford Junction.

    This comes from Just Trains, and goes all the way to Lincoln. 41-47 are pictured.

    upload_2022-6-24_21-52-35.png

    Regarding ECML North, I don't think it's been converted to use the AP file yet. Once it is, that problem with different files will probably vanish, but testing has to be done on the changeover first I would imagine. Try it yourself by deleting all of the individual files that get installed, but check with me because I have a list of all assets included comparing it against the Common Library. I can tell you what to not touch if necessary.

    I can confirm the array error in the JT version of the file:

    upload_2022-6-24_21-54-48.png
     
    Last edited: Jun 25, 2022
  23. torfmeister

    torfmeister Guest

    The Steam version of the N-L file ends at Radford Junction.

    But I can spot a mistake in the JT file you've posted too - before the last tag there is a </CSVItem> closing tag missing which was opened in line 3... also note the misalignment due to missing tabs (not important but an indicator of things gone wrong)?

    Have not tested but this could eventually lead to a crash in the editor.

    Can you post the file here I will rectify it for everyone. Notepad++ shows a line from opening to ending tags so you can easily spot errors there.
     
    Last edited by a moderator: Jun 25, 2022
  24. torfmeister

    torfmeister Guest

    I have created a ticket - both Steam and JT RouteMarker files are corrupt - wrong closing tags and same id for Swinderby and Hikeham.

    2022-06-25 04_37_51-[#47729] Corrupt file _ Dovetail Support.png

    For those who can't wait, I've provided an updated and fixed version of the file, containing 47 stations to choose from in the editor. Use at own risk ;)
     

    Attached Files:

    Last edited by a moderator: Jun 25, 2022
  25. torfmeister

    torfmeister Guest

    ECML North is not ap'd - i did not install the loose CommonLibrary files as all of them are included in the updated CommonLibrary.ap. Runs well and is beautiful, just drove the HST first service into sunrise...
     
    • Like Like x 1
  26. Cyclone

    Cyclone Well-Known Member

    Joined:
    Feb 29, 2020
    Messages:
    3,853
    Likes Received:
    1,324
    That's what I thought. I did a spreadsheet looking through everything using CommonLibrary files and believe every single file is accounted for, so it should be possible to delete pretty much any Common Library files in other installs. What I have not scanned and tested yet: the SECML, Bishop Lydeard to Darlington, Bristol to Avonmouth, Western Mainlines, a couple extensions that are not standalone routes, and anything else sold exclusively on their site and not on Steam. There could be a few missing files I have not accounted for yet.

    BTW, I attached my own copy (should be identical) of the edited file referenced above to a JT ticket I conveniently had going. I also proposed a version for the standalone routes that doesn't point to things not included with the route it applies to. Not posting this one as yet because you still get Sheffield-Derby in there anyway, but if you don't wish to use the fixed version torf posted and want to test a standalone Lincoln one (I literally took the full one and removed irrelevant entries, renumbered, and saved), I can post it if anyone wants it.
     
    Last edited: Jun 25, 2022
  27. torfmeister

    torfmeister Guest

    One thought - I guess the fastload advantage in conjunction with .ap files comes from the fact that (and this is just my observation and conclusion) after the .ap is loaded into memory by the operating system, TS uses kind of its own file handler to access the data. I don't even know if Windows allocates the memory, might also be TS uses its own routine and filsystem for accessing specific data inside the .ap directly from disk. That could be an explanation of OOMs.

    To find that out I'd need a good system monitoring program... back in my Commodore Amiga days I used a tool called "SnoopDOS" which logged every file access, very easy to use. Showed read access, write access, file locks and so on.

    One fact leads me to this - a scenario that includes a movie (Canadian Mountain Passes tutorials or the Forest Rail timetable intros) will not show the video when inside an .ap - because the movies are handled by a dll that requires standard file access to stream them.
     
    Last edited by a moderator: Jun 26, 2022
  28. Cyclone

    Cyclone Well-Known Member

    Joined:
    Feb 29, 2020
    Messages:
    3,853
    Likes Received:
    1,324
    Sherman Hill uses introductory movies. So does Riviera Line. Do these work? I've seen them in both cases, and I believe they are still packed.
     
  29. torfmeister

    torfmeister Guest

    I don't think so, can't remember SH or Riviera movies. Sherman Hill uses introductory camera scripts but no video files.

    If you check Canadian Mountain Passes, it comes with an additional Scenarios folder next to the MainContent.ap, for the tutorial missions which feature introductory narrated .ogv videos - those will not play from inside an .ap, I've double checked that. Black screen - freeze.

    There's really just CP and Forest Rail I can remember. Apart from that I packed all route folders into MainContent.aps. No risk of accidentally saving unwanted changes to the route (TS still has no option to quit the World Editor without saving changes). So changed tiles are saved in external folders which you only need to delete to undo your changes.
     
    Last edited by a moderator: Jun 26, 2022
  30. Reef

    Reef Well-Known Member

    Joined:
    Jul 27, 2018
    Messages:
    2,806
    Likes Received:
    1,817
    Not teaching you how to suck eggs (old English saying "teaching your grandmother how to suck eggs" i.e. telling you something you already well know, if you're unfamiliar with it, and no, I don't know what egg sucking has to do with grandmothers either... anyway.)

    But for those that may not know, if you bork something up in the editor simply press Alt + Tab and press the red x in the top right of the preview tab to exit TSC without saving.
     
  31. torfmeister

    torfmeister Guest

    I know. Dirty exit. Not necessary when packed. Simply press yes and delete new folders.
     
  32. Cyclone

    Cyclone Well-Known Member

    Joined:
    Feb 29, 2020
    Messages:
    3,853
    Likes Received:
    1,324
    That might be it, then. I did not realize there was a difference.
     
  33. Peter Hayes

    Peter Hayes Well-Known Member

    Joined:
    Dec 9, 2016
    Messages:
    814
    Likes Received:
    328
    Torf
    The OOMs are mainly caused by INVALID PAGE FAULTS - simply windows loads TSC into the VAS and that, in turn, is loaded via the MMU into the RAM. Any mismatch between the VAS address and the RAM address - OOM.
    Some sound files can load directly into the RAM and an issue there can result in an OOM - not sure this happens in TSC.
    The Windows tools that I use ( wrt SnoopDOS - I remember it well)
    VMMAP ( showed some fragmentation and loss of contiguous space)
    ProcExp
    ProcMon and
    Xperf.
    Nothing conclusive was found with any of them.
     
  34. torfmeister

    torfmeister Guest

    Sounds - there's one thing I noticed years ago that is still present. If you play a Ruhr Sieg scenario and quit whilst being near a station, there's an endless ambient sound looping in the background that only stops a few seconds after you shut down TS. This, as not being a sound expert, reminds me of my Amiga coding days, something like forgetting to free allocated memory at program end or break - this must not happen.

    So there might be something with the proxyxml file, I think it's the Hagen station ambient sound. I found this to cause infrequent OOMs when playing another route after that.

    Somewhere I heard of a bug with sound "InstanceGroups"... I think it was some patch on RWA.
     
    Last edited by a moderator: Jun 27, 2022
  35. Cyclone

    Cyclone Well-Known Member

    Joined:
    Feb 29, 2020
    Messages:
    3,853
    Likes Received:
    1,324
    Have you tried the updated version of the route? I'm wondering if that causes the same issue. It's on the German Railsim site.
     
  36. torfmeister

    torfmeister Guest

    It's not only specific to that route, I've noticed this too on some other routes, can't remember which though.
     

Share This Page