Tutorial Lidar Tutorial

Discussion in 'PC Editor Discussion' started by Lulon, Sep 29, 2023.

Tags:
  1. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    I'm waiting for matts tutorial every time i try get the area I want and import it I get a flat surface
     
    • Like Like x 1
    • Helpful Helpful x 1
  2. KrisKol

    KrisKol Active Member

    Joined:
    Apr 2, 2023
    Messages:
    140
    Likes Received:
    80
    Matt also said that he unfortunately found a bug in the LiDAR import, while prepping his tutorial.
    he is going to get it fixed.
    Its also a good thing in a way, because it means He and Us, will be able to get a better understanding as to how it all works properly.
     
    • Like Like x 2
    • Helpful Helpful x 1
  3. VictoryWorks

    VictoryWorks Well-Known Member

    Joined:
    Sep 22, 2017
    Messages:
    256
    Likes Received:
    1,319
    Struggling like many of you. Got the LIDAR data, merge it, Pseudo M it, tile it, export it, add it to the UE project. hgt data works fine, apply the LIDAR asc's and it just modifies the tile to be flat and about 1km up in the air.
    Tried it every way I can think of and just can't get the LIDAR to work correctly. Wondering if I fouled up an import setting when I was initially messing around with no clue.
    Is there any way to set the entire Editor back to default settings?
     
    • Like Like x 1
    • Helpful Helpful x 1
  4. Daunfr59

    Daunfr59 Active Member

    Joined:
    Dec 8, 2016
    Messages:
    86
    Likes Received:
    114
    I have imported UK LIDAR data and got it to work using Lulon's method although I had to reinstall the editor at one point as it kept crashing.
    I am trying to transfer my Train Simulator (Classic) route based around Chippenham Wilts and the LIDAR data has loaded in exactly on the spot I chose as the route origin. Track bed, embankments, roads etc are all visible and there are even flat "pads" where large buildings are in real life.
     
    • Like Like x 1
    • Helpful Helpful x 1
  5. blogtraindriver

    blogtraindriver New Member

    Joined:
    Dec 8, 2018
    Messages:
    6
    Likes Received:
    6
    I had the same problem.
    But then I re-read the original explanation at the top of this thread. And there was one point that I had overlooked:
    point 7: It is very important to get the data lined up with the correct coordinates. i.e. that the merged tif files must be adapted to the coordinate system used in TWS4 in general because it is: EPSG:3857 - WGS 84 / Pseudo-Mercator.
    Then I think you will be successful too! Good luck.
    By the way: There is a batch button (at the bottom) which works perfectly when tiles need to be converted from .tif to .asc - and then it's no problem with the many tiles
     
    • Like Like x 2
  6. fakenham

    fakenham Well-Known Member

    Joined:
    Dec 9, 2016
    Messages:
    755
    Likes Received:
    1,089
    So, I've had a crack at this using a combination of the tutorial above and the video. It goes kind of well in QGIS, but I having trouble understanding the correct coordinate systems to use, because I'm using a UK route area and the tutorials are working from Australia/German areas I think.
    Could anyone who's had success with the UK lidar just highlight what coordinates systems they used - becuase when I remimported my TIFF files to check, they spawned on the equator! Obviously they won't show correctly in game then!
     
    • Like Like x 1
  7. fceschmidt

    fceschmidt Well-Known Member

    Joined:
    Oct 22, 2020
    Messages:
    312
    Likes Received:
    405
    Try EPSG:25830 (UTM Zone 30 North) :)
     
    Last edited: Oct 5, 2023
    • Helpful Helpful x 1
  8. fakenham

    fakenham Well-Known Member

    Joined:
    Dec 9, 2016
    Messages:
    755
    Likes Received:
    1,089
    For all the guys that were getting flat terrain upon importing LIDAR (and in the air) , I think I know the reason - Tick the Use Pseudo Mercator Box in the snap to landscape tab (see below)!
    I was getting the flat in the air terrain, this solved it. Hurrah.
    It was quite a tedious process, but a combination of this post and the need for the correct coordinates (and the video linked) worked well. Took long enough though. Thanks to all that helped.

    lidar.png
     
    • Like Like x 2
    • Helpful Helpful x 2
  9. VictoryWorks

    VictoryWorks Well-Known Member

    Joined:
    Sep 22, 2017
    Messages:
    256
    Likes Received:
    1,319
    Got it sorted today, thanks to everyone for the detailed comments and instructions
     
    • Like Like x 1
  10. DTG Lukas

    DTG Lukas Developer Staff Member

    Joined:
    Dec 18, 2017
    Messages:
    65
    Likes Received:
    353
    The rough edges happen when the file has no „no data value“ in it
    I am changing the code for that, also so that it’s not flattening areas where there is no data.

    But I am not sure when change will be released
     
    • Like Like x 4
  11. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    Will give this a g
    you have to use the tif ones. in the dropdown menus choose LIDAR Composite DTM, year 2022 and resolution DTM 1M
     
    • Like Like x 1
  12. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    problem i'm having when i select apply lidar data to selected lanscape proxys i get a solid block pop up (see image). I also uninstalled and reinstalled the route editor then uninstalled installed qgis and installed the 3.28."firnze" version this seems to have sorted some issues and my editor doesn't crash, well it hasn't up to now.

    EDIT: Just a quick thought. when i went to created a new route the area I chose is at a sea level of approximately 19m but I left this at 0 could this be the reason I'm seeing blocks when I apply lidar data to selected landscape proxys? And when I go into network edit mode my track laying circle is up in the air
     

    Attached Files:

    Last edited: Oct 6, 2023
    • Like Like x 1
  13. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    LIDAR composite DTM is the one i downloaded select the latest date in next dropdown and then 1m in the next dropdown
     
  14. fakenham

    fakenham Well-Known Member

    Joined:
    Dec 9, 2016
    Messages:
    755
    Likes Received:
    1,089
    Frustrating.
    A few things to try maybe.....
    I take it you've ticked that pseudo mercator box in the editor, mine was unticked by default.
    When I applied my lidar, I right clicked whilst in the tile to bring up the menu, then clicked lidar from there, may help?
    I used Qgis 3.32.3, worked OK.
    Are you seeing all the lidar entries in the menu in editor on the left when you import them?
    I left the sea level box as 0 when creating my route in the UK, don't think this makes a difference.
    With the reprojected tifs, do they appear back in the right geographical location when you place them back in QGIS? - I made a mistake with the coordinate system and they spawned near the equator.
    For the coordinate system, I think in the end I used the one highlighted in the shot below (for East UK). However, I might have used the one just below it! I can't remember, sorry about that, but it was definitley one or the other.....was getting frustrated at that point.
    coord.png
    The problem with this is that there's quite a few places to get slightly incorrect, as I found out - it's quite annoying.
    So I used a combination of this tutorial, the video in this tutorial and some of DTG's video for the later stages.
    Hope it helps.
     
    Last edited: Oct 6, 2023
    • Like Like x 3
  15. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    thanks will take a look when I get home been dragged out to meadowhall:(
     
  16. VictoryWorks

    VictoryWorks Well-Known Member

    Joined:
    Sep 22, 2017
    Messages:
    256
    Likes Received:
    1,319
    I was getting that solid block when the LIDAR data wasn't exactly how it needed to be. Make sure you follow the instructions in the first post to the letter, every single bit. If you find a different way to do something, don't! Follow it EXACTLY. And keep any eye on the bottom right in QGIS. It seems to like swapping the CRS data type without telling you.. or it did for me anyway.
     
    • Like Like x 3
  17. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    what puzzles me most is why, for the editor, we have to use the .asc format when unreal engine uses .tif and .raw for importing height files. would've saved a lot of time and issues.
     
    • Like Like x 2
  18. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    I think you're onto something here in the video tutorial i'm following he says to search for 28356 and the map in the corner of qgis highlights the place hes working on in australia so gonna look for the one that covers the uk
     
    • Like Like x 1
  19. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    one thing i noticed in the tutorial i watched the guy only imported the .asc files but in Matt and Wills video he imported them into a separate folders from the .hgt files so i'm gonna do the same as Matt and Will see if it makes any difference I also chose, in qgis, to select a crs that covered the whole of the UK and not Australia :), fingers crossed
    EDIT: YES YES YES. thanks for your help Matt, Will, Fakenham, VictoryWorks and everybody here.
     

    Attached Files:

    Last edited: Oct 10, 2023
    • Like Like x 2
  20. fakenham

    fakenham Well-Known Member

    Joined:
    Dec 9, 2016
    Messages:
    755
    Likes Received:
    1,089
    Great to hear you've got it sorted, it's cool when it works and shows all the cuttings and embankments. It's a long hard slog after this though, there's so much to learn. Just trying to find all the useful assets is a task in itself!
     
    • Like Like x 1
  21. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    I was thinking about assets yesterday and wondering if you can create your own in blender or something similar to import or use ones from unreal engine marketplace. If possible i'm not sure how to use blender (so something i'm willing to sit and learn) and not sure how to get assets from epic games marketplace into our editor this i'm going to look up too. I don't want to be using just any assets, from what we have, I'm determined to make my route (although small) to be as close to the real thing as possible. One thing I have is determination. In 2001 I had a traumatic brain injury and doctors said after my operation I'd be in a wheelchair for up to 2 years maybe longer, I was so determined I was only in it for 6 month. Once I set my sights on something I have to get it done. EDIT: just another thought can we create assets in unreal engine 4.26 and then import them into our editor?
     
    Last edited: Oct 7, 2023
    • Like Like x 1
  22. MattD6R

    MattD6R New Member

    Joined:
    Oct 3, 2023
    Messages:
    2
    Likes Received:
    0
    After several attempts I had success with the lidar. For me the issues was the flat raised areas mostly due to that I was importing lidar data too close to the edge of my source lidar data. So make sure you extend the source lidar a bit further out than the area being imported. Also with the CRS type as well you need to make sure you are setting that explicitly at every step that asks for this even if it shows the correct value at the lower right. Also before importing within the editor you need to set the "Use Pseudo Mercator" to "yes. When I missed the CRS was wrong the lidar worked but the terrain was all wrong.

    But when I finally got it right the level of detail is worth the effort, and aligned very well with the real contours of the land.
     
  23. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    UPDATED 10/10/23:: This is information for people creating a route that's in the UK. I followed along with (1) TSW4 PC Editor | How to import terrain - YouTube This video is a great start and to help others along. @ 4.00 to get UK lidar data go to https://environment.data.gov.uk/DefraDataDownload/?Mode=survey in the download options select LIDAR Composite DTM, Year 2022 or whichever the latest year is, resolution DTM 1M . @4.36 set the number to 27700 It should be something like OSGB36/BRITISHNATIONALGRID EPSG:27700 @11.36 go to this tutorial posted by Matt https://forums.dovetailgames.com/threads/plugin-creation.74410/#post-727523 and create your plugin for your route and then follow this tutorial .@1839 and @1908 Tick the Use Pseudo Mercator Box in both snap to landscape and Tile placement tabs Lidar settings dropdown. Make sure you setup your data, route and folders exactly the same. This is what worked for me I hope it helps.
     
    Last edited: Oct 10, 2023
    • Like Like x 2
  24. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    I think I took on a bigger challenge as far as assets go the area i created is built up with housing and industrial buildings. My other route I was going to do is more flat and less built up , Skegness to Grantham, so until I learn more I'm going to save my Consbourgh to Swinton Route and go to skeggy haha
     
  25. london#6712

    london#6712 Active Member

    Joined:
    Mar 8, 2021
    Messages:
    103
    Likes Received:
    118
    i need help with mine, ive done everything but it doesnt go to lidar it just says converted.
     
  26. Zeo2113

    Zeo2113 Member

    Joined:
    Dec 10, 2016
    Messages:
    52
    Likes Received:
    17
    I tried following DonnyDave's guide for UK routes, I've got as far as getting HGT and ASC files in and working with Google Maps Overlay but UE4 doesn't seem to understand AUX or PRG files, am I doing something wrong? Also, as soon as I put the LIDAR data in, the map rises super high and all flat with no option to undo it.

    I think after spending days at attempting to do this, that I might just wave the flag and surrender until Matt's guide comes out.
     
    • Like Like x 2
  27. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    i've looked around and the aux and prg files aren't recognised in our editor so i'm deleting ones i'm doing atm will edit this post if it still works EDIT: It's OK to delete the .aux and .prg files as these are not needed in the editor
     
    Last edited: Oct 8, 2023
  28. Lesethard

    Lesethard New Member

    Joined:
    Sep 21, 2023
    Messages:
    10
    Likes Received:
    1
    Haha my Lidar was totally fine all the time. I just forgot to set the checkbox... ;) Thank you for your friendly reminder. I finally managed to import Lidar into TSW Editor.

    Good point, but i don´t know how to add all my layers at once to the batch process. I can only add one by one and it still takes so much time... Isn´t there any possibility to batch all your TIF files in a folder at once to ASC files in another folder?
    I did some research on the internet, but was not able to find any useful code for the batch process :/
    Do you have an excample code which i can adapt to my file path?
     
    • Like Like x 1
  29. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
     
  30. KrisKol

    KrisKol Active Member

    Joined:
    Apr 2, 2023
    Messages:
    140
    Likes Received:
    80
    I have already surrendered. I dont want to have a plug in folder full of incorrect data.
     
  31. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    you don't need to import AUX or prg files
     
  32. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    when your lidar raises like a block and flat its because there is no lidar data for that area, or it could be you've got the lidar but not the right co ordinates In your route creation. Best way to get the correct location lon/lat is opening Google maps and right click on so.ething, in my case I chose skegness railway station as it was at the area I wanted to start my route.
     
  33. Richie

    Richie Member

    Joined:
    Apr 22, 2020
    Messages:
    39
    Likes Received:
    49
    When you open the translate tool, there is a "run as batch process" button at the bottom left of the tool. Click it, then for files, click autofill, then select open layers and select all the layers to convert. Autofill the rest of the required fields then run.
     
  34. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    that's really helpful thank you :)
     
    • Like Like x 1
  35. Zeo2113

    Zeo2113 Member

    Joined:
    Dec 10, 2016
    Messages:
    52
    Likes Received:
    17
    I did all of that, I used Southampton Central as my point of reference (50.907257629569436, -1.4133854210332524), all the HGT files are importing fine, I can see river beds, the hills and the flatlands of the nearby docks. I used SU 41SW from the Defra Survey Data Download - LIDAR Composite DTM 2022 DTM 1M and followed the guides above but still flat lands on importing. Maybe I did miss a step but not sure which one :(
     
  36. fakenham

    fakenham Well-Known Member

    Joined:
    Dec 9, 2016
    Messages:
    755
    Likes Received:
    1,089
    I know, it can be a painful experience. Have you used the following (highlighted in blue) UK based coordinate system when following the tutorials?
    upload_2023-10-12_11-57-58.png
     
  37. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    When you have got your hgt files make sure when getting your lidar you cover an area just bigger than the area you selected for your .hgt. also when creating your new route in the lon/lat for Southampton Central station just enter 50.90724 and -1.41335 there's no need to enter all of the numbers you've posted above
     
  38. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    not sure why but i'm getting an error saying unable to save package because the file is read only has anyone else had and solved this ?
     
  39. jh#9091

    jh#9091 New Member

    Joined:
    Oct 3, 2023
    Messages:
    4
    Likes Received:
    0
    hi how do i change trains and times on pc editor?
     
  40. Boingleboeing

    Boingleboeing Member

    Joined:
    Jul 31, 2021
    Messages:
    85
    Likes Received:
    45
    I do just want to point out guys, I don't want to be that guy who gets annoyed because of this, but just to try to keep them sort of clean, this is a thread on LIDAR data if you have any other questions find the appropriate thread or make a new one. This is just so that we can separate all the threads and not get mixed up ;)
     
    • Like Like x 2
  41. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    A good point. its hard enough for some of us to find the help we're searching for without having to scroll through unrelated topics. As a suggestion can we have any tutorials put into a section that no one can comment on so they're easy to find as and when we need them?
     
    • Like Like x 2
  42. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    A great tip from OlaHaldor that saves a lot of time in QGis when you have a lot of files. Right, so instead of importing all my geotif files, and then merge, I just merge the files off the bat. No reason to spend time waiting for the import, then wait for the merge.

    Just import from folder instead.
    [​IMG]

    After you've "re tiled" the merged heightmap and stored to a separate folder, you can batch convert the exported geotiff tiles like this. Click the "Run as batch process" at the bottom of the Translate window. Then click the "Autofill" under "Input Layer". Load the files by "add all files from a directory".
    [​IMG]

    You will need to make sure that the files are stored as .asc as well.
    Click the button with the three dots to pick a folder and a name. In my example I named it "dem_"
    [​IMG]

    Once you're happy with that, a new box will ask you what you'd like to do:
    Select "Fill with parameter values", and it will autofill all the output fields with something like "dem_<numbers and stuff that makes sense in a sequence>.asc"
    [​IMG]

    Once that's done, click Run, and watch the magic happen.
     
    • Helpful Helpful x 3
  43. JT-Nor

    JT-Nor New Member

    Joined:
    Sep 14, 2023
    Messages:
    17
    Likes Received:
    13
    I had issues with spikes in the lidar terrain, along the lidar tile borders.

    Picked up from Lucas on the stream yesterday about the "-9999" and that solved my issues.

    Step 7: Add -9999 to No data value for output bands
    [​IMG]
    Before:
    [​IMG]
    [​IMG]
    After:
    [​IMG]
    [​IMG]

    Regards,
    JT
     
    • Like Like x 1
    • Helpful Helpful x 1
  44. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    Just to let everyone know I'm in the process of typing up a route editor route creation guide. I'm gathering all my information from these forums, YouTube and other sources and compiling it all into easy to follow step by step instructions (with pics) and once checked will work out a way I can get it available for everyone to download and follow after I get the green light from DTG to do so. I've got tomorrow off work so with a bit of luck and plenty of coffee I should get it finished.
     
    • Helpful Helpful x 1
  45. willbraddock#4738

    willbraddock#4738 New Member

    Joined:
    Sep 30, 2023
    Messages:
    21
    Likes Received:
    16
    Hi All,

    I have followed the Lidar/SRTM part of the styream from the other night. FOr some reason when I apply my liodar data to the dem it just make sthese really big boxes, any ideas? Any help would be greatly appreciated. :)

    upload_2023-10-14_9-22-37.png
     
    • Like Like x 1
  46. DonnyDave

    DonnyDave Active Member

    Joined:
    Jun 29, 2020
    Messages:
    220
    Likes Received:
    106
    You get those when there is no lidar data available for that area for the editor to find. It maybe that its just not available or the area you've selected to download wasn't selected that far
     
  47. JT-Nor

    JT-Nor New Member

    Joined:
    Sep 14, 2023
    Messages:
    17
    Likes Received:
    13
    I got similar boxshape in my early tries, I believe it was in the reprojection step that I did wrong.

    During reprojecting setup I follow this and had success:
    Selected the merged layer.
    Force Source CRS according to the data source
    Target CRS to the EPSG:3857 Psuedo Mercator one
    Nodata value for output bands: -9999 (fixing spikes in Lidar tile borders)
    Output file resolution in target georeferenced units: 1.0

    Start with a few tiles to check and expand. I wasted a lot of time on doing huge areas before actually doing it correctly...
     
    • Helpful Helpful x 2
    • Like Like x 1
  48. willbraddock#4738

    willbraddock#4738 New Member

    Joined:
    Sep 30, 2023
    Messages:
    21
    Likes Received:
    16





    This was the issue, thanks for the help guys!

    Although I've now lost my map file in the editor so I can't reload it - can see it in WIndows Explorer though?
     
    • Like Like x 2
  49. JT-Nor

    JT-Nor New Member

    Joined:
    Sep 14, 2023
    Messages:
    17
    Likes Received:
    13
    Not able to load in "file -> recent levels" either?

    Try dragging the file from windows explorer window to content browser folder for the route in editor.

    Its a know bug that files are not loaded so hopefully will be sorted next editor update.
     
    • Like Like x 2
  50. willbraddock#4738

    willbraddock#4738 New Member

    Joined:
    Sep 30, 2023
    Messages:
    21
    Likes Received:
    16
    Hi All, thanks for all your help and responses today - Have managed to make some great progress. Any ideas how to texture the terrain?

    Thanks again!!
     

Share This Page