Tutorial Lidar Tutorial

So, apparently importing LIDAR data is one of the hardest things with the TSW Editor.
For the last days I have been trying everything to get it right.
I watched the preview stream which only had very little information about LIDAR import.
DTG Lukas gave some tips on the forum.
But I think I finally got it.
So I made a step by step tutorial

This is just what works for me, I don't know if it works for everyone but at least this is giving a bit more info.
And I guess this is probably not what would be the "official" way.

If you follow this, I expect you to know a bit about the software QGIS (hopefully more than me),
and have GDAL installed (don't know if it comes with QGIS by default).


First off, you want your LIDAR data to extend slightly further than you actually need in the route.
It should overlap a bit with the SRTM only areas.

1. Get your LIDAR data in a format that QGIS understands.
I used LIDAR data from NRW in Germany, which is in the .xyz format.
There are other LIDAR formats which may require conversion first.
Also note the Coordinate Reference System (CRS) that the data is in.
Mine is in UTM zone 32N. The tiles are all 1km x 1km.

2. According to DTG Lukas, import all you LIDAR data into QGIS.
I imported all LIDAR tiles into QGIS (around 600).
Depending on the amount this will take a while and you might need a good PC for this.
Or maybe just to test this, choose a smaller area.

3. After the import is done you need to merge all the layer together that were created on import.
Go to Raster > Miscellaneous > Merge...
Select all layers as input layers using the button on the right side.
You might want to set it so that the merged file gets saved in case something goes wrong.
Then click "Run"
A new layer called "Merged" should have appeared.

5. At this point you should quickly click on the little eye symbol and hide all layers to stop your PC from lagging.
Collapse all layers and remove the imported layers. Only keep the merged layer.

6. Right click on the layer in the list in the bottom left.
Select "Layer CRS > Set Layer CRS..."
In the new window search for the CRS your LIDAR data is in.
Mine is ETRS89 / UTM zone 32N (EPSG:25832).
Click OK.
I honestly don't know if this step is neccessary as we set the source CRS in the next step anyway but just to be sure.
If your layer disappears after this, right click on your layer and click "Zoom to Layer(s)"

7. We now need to reproject the merged layer to the Pseudo Mercator system.
Go to Raster > Projections > Warp (Reproject)...
Select the merged layer as input layer.
Set your source CRS the same as in step 6.
Set the target CRS to "EPSG:3857 - WGS 84 / Pseudo-Mercator".
I set resampling method to bilinear. Choose whatever you want.
Set "Output file resolution in target georeferenced units" to 1.0.
Just like in step 3 maybe you want to set it so that the reprojected file is saved somewhere.
Click "Run".
Now you should have the layer "Reprojected".
You can try and add an OSM layer to see if it reprojected correctly.
OpenStreetMap is in the browser above the layers.

8. Now we need to tile the reprojected layer into smaller pieces.
Open the processing toolbox by clicking Processing > Toolbox.
Then on the right, open GDAL > Raster miscellaneous > Retile.
As input layer, select the reprojected layer.
Enter your tile with and hight. I used 3000x3000 but from a bit of testing, it doesn't seem to matter.
But bigger tiles means less tiles but larger files.
In one post DTG Lukas said he also uses 2500x2500.
Choose your output directory.
The output files will be in the GEOTiff format.

9. The last step is the worst one.
Delete the folder called "1".
Delete every file that is only zero. (Thumbnail is black in Windows Explorer).
Delete every file that is not 3000x3000 or whatever tile size you entered.
I also deleted every tile that is more that half zeros.
Convert every .tif tile to .asc using GDAL translate in QGIS.
Yes, this is almost impossible by hand if you have many tiles, so you could write a script or
search google for QGIS GDAL batch conversion.
You can delete every .aux.xml and every .prj file if you want, as the TSW Editor won't accept those anyway.

10. Check if the .asc files look good.
Rows and cols should be 3000,
cellsize should be 1.0


11. Open the editor and import the files in some empty folder, maybe next to your SRTM data folder.

12. Now it's time to use the data.
Go into Route Building Mode.
Click on Snap Landscape To Track.
I don't know what the "UTMZone" and "Use Pseudo Mercator" options do,
but for me it's "UTMZone" = 32 and "Use Pseudo Mercator" = yes (was the default).
Select all imported tiles in the editor.
Click on "Use Selected Assets".

Now don't right click the LandScapeStreamingProxys in the World Outliner.
Instead right click on a tile in the main viewport.
It must be a tile that you know has Lidar coverage in your data or else it will be flattened.
You might not be able to see that its highlighted with a orange outline,
but it's still selected and the context menu will pop up.
Click on "Apply Lidar data to selected Landscape Proxys".
The tile should now have LIDAR quality and you can clearly see the smooth transition from LIDAR to STRM.



The reason you have to select only one tile is because the editor will
just flatten every selected tile isn't covered by LIDAR in your dataset.
From then you just apply LIDAR tile by tile. Never apply to a tile out of your LIDAR coverage or it will be flattened.

One more tip:
In the top left of the viewport you can click on Perspective then on "Top".
Then click "Lit" and then on "Visibility Collision" or something (Lit doesn't work in top ortho for some reason).
Now you can navigate efficiently using the right mouse button while generating new tiles.
You can also right click on tiles here and apply LIDAR here.




I hope this helps
(and works)

 

Godsend.
I’ll have a crack in a moment (hopefully it doesn’t crash!)

 

You're a genius.

 

Lovely!!! YAY!! I am going to have another stab later on.

 

Lulon do you have also an instruction without SRTM data? I don´t want to register for data which I don´t need...

 

The lower Resolution SRTM creates Distant mountains, and other bits of distant lesser details. As far as I know that is.
(its actually something that makes quite a bit of sense to me)

 

Nice instructions, this will be my weekend project Thank you for sharing your findings!

 

This needs to be stickied. DTG Alex or dtg_jan please.

 

Excellent stuff. I've processed over 600 tiles to .asc this morning thanks to this, and it mostly works. There are some quirks with the LIDAR data which was surprising, but I guess that's the data, not the process.

Thanks a lot for the guide

 

I followed both the tutorial and the video, but there is something strange with my source data.
The coordinates are not translated correctly so the asc files are filled with meaningless numbers.

 

Thanks for the positive responses! Im glad to see people having success with this.
First part of the master class came early

 

Sorry, no. I tried it myself to use only LIDAR but the "Create Landscapes" button doesn't show up.
But you only need to register for 30 m SRTM, not for 90 m SRTM as far as I know.
And as KrisKol said, it's really useful.

 

The only reasons why I think that could happen is that you didn't set the source and target CRS correctly during reprojection
or your source LIDAR data is not in the CRS you think it is.
Also I think it's really important to set output file resolution to 1.0 when reprojecting.

 

I figured out why I had issues. (sadly it seems Discord is down so I can't easily retrieve what I shared there in a group, but what it essentially boiled down to was corrupt/duplicate data within the same tile coordinate).

When you convert TIF to ASC, there is a new folder created, labeled "1". As far as I gather, anything that goes into there is useless. Don't use it. Delete it once the conversion is done.
I did not delete it, I imported the data to unreal, and.. That's when havoc started as I went on to apply LIDAR data on the tiles. It was really messed up in some places.

I re converted but did not use the data in the folder "1", and now everything works perfectly!

 

Oh right, I forgot to mention that. GDAL retile always creates that folder and I think you can't disable it.
I think the folder contains some zoomed out tiles if you want to use it in a google maps like system. Just delete it.

 

Thank you. That was one of the issues and, based on OSM map, the coordinates are now correct.

After the entire procedure, when I save the asc they still say NODATA_value.
ncols 2500
nrows 2500
xllcorner 934221.330116000026
yllcorner 5868595.380293999799
cellsize 1.000000000000
NODATA_value 9.9692099683868690468e+36

 

Okay, how weird. I never had that. And I can't think of a reason why thats happening. Maybe it has something to do with the data types? Leaving the data types as the default in QGIS worked for me. But I would check if the retiled tiles in .tif format are okay first before converting.

 

If I check the properties, after the merge, there it is the strange number.
Band1#long_name=GDAL Band Number 1
Band1#_FillValue=9.96921e+36

Never seen something so variable...

 

Even following text and video step by step, I can´t merge the lidar data. What´s wrong here? Source files come in EPSG:25833 *.xyz

Spoiler: QGIS Error message

QGIS-Version: 3.32.2-Lima
QGIS-Codeversion: c0b8833964
Qt-Version: 5.15.3
Python-Version: 3.9.5
GDAL-Version: 3.7.1
GEOS-Version: 3.12.0-CAPI-1.18.0
PROJ-Version: Rel. 9.2.1, June 1st, 2023
PDAL-Version: 2.5.5 (git-version: f1a9ac)
Algorithmus gestartet um: 2023-09-29T14:54:05
Algorithmus Verschmelzen startet…
Input parameters:
{ 'DATA_TYPE' : 5, 'EXTRA' : '', 'INPUT' : ['M:/TSW/Freiberg_Nossen/DEM/dgm1_33376_5642_2_sn_xyz/dgm1_33376_5642_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33376_5644_2_sn_xyz/dgm1_33376_5644_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33376_5646_2_sn_xyz/dgm1_33376_5646_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33376_5648_2_sn_xyz/dgm1_33376_5648_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33376_5650_2_sn_xyz/dgm1_33376_5650_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33376_5652_2_sn_xyz/dgm1_33376_5652_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33376_5654_2_sn_xyz/dgm1_33376_5654_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33376_5656_2_sn_xyz/dgm1_33376_5656_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33376_5658_2_sn_xyz/dgm1_33376_5658_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33378_5638_2_sn_xyz/dgm1_33378_5638_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33378_5640_2_sn_xyz/dgm1_33378_5640_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33378_5642_2_sn_xyz/dgm1_33378_5642_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33378_5644_2_sn_xyz/dgm1_33378_5644_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33378_5646_2_sn_xyz/dgm1_33378_5646_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33378_5648_2_sn_xyz/dgm1_33378_5648_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33378_5650_2_sn_xyz/dgm1_33378_5650_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33378_5652_2_sn_xyz/dgm1_33378_5652_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33378_5654_2_sn_xyz/dgm1_33378_5654_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33378_5656_2_sn_xyz/dgm1_33378_5656_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33378_5658_2_sn_xyz/dgm1_33378_5658_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33380_5638_2_sn_xyz/dgm1_33380_5638_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33380_5640_2_sn_xyz/dgm1_33380_5640_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33380_5642_2_sn_xyz/dgm1_33380_5642_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33380_5644_2_sn_xyz/dgm1_33380_5644_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33380_5646_2_sn_xyz/dgm1_33380_5646_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33380_5648_2_sn_xyz/dgm1_33380_5648_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33380_5650_2_sn_xyz/dgm1_33380_5650_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33380_5652_2_sn_xyz/dgm1_33380_5652_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33380_5654_2_sn_xyz/dgm1_33380_5654_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33380_5656_2_sn_xyz/dgm1_33380_5656_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33380_5658_2_sn_xyz/dgm1_33380_5658_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33382_5638_2_sn_xyz/dgm1_33382_5638_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33382_5640_2_sn_xyz/dgm1_33382_5640_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33382_5642_2_sn_xyz/dgm1_33382_5642_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33384_5638_2_sn_xyz/dgm1_33384_5638_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33384_5640_2_sn_xyz/dgm1_33384_5640_2_sn.xyz','M:/TSW/Freiberg_Nossen/DEM/dgm1_33384_5642_2_sn_xyz/dgm1_33384_5642_2_sn.xyz'], 'NODATA_INPUT' : None, 'NODATA_OUTPUT' : None, 'OPTIONS' : '', 'OUTPUT' : 'TEMPORARY_OUTPUT', 'PCT' : False, 'SEPARATE' : False }

GDAL-Befehl:
gdal_merge.bat -ot Float32 -of GTiff -o C:/Users/*/AppData/Local/Temp/processing_ddhmYt/774dd81a5a754687b6942eea55992655/OUTPUT.tif --optfile C:/Users/*/AppData/Local/Temp/processing_ddhmYt/120c0354ced84ca8afddc90627a2222b/mergeInputFiles.txt
GDAL-Befehlsausgaben:
Traceback (most recent call last):
File "M:\Program Files\QGIS 3.32.2\apps\Python39\Scripts\gdal_merge.py", line 14, in <module>
sys.exit(main(sys.argv))
File "M:\Program Files\QGIS 3.32.2\apps\Python39\lib\site-packages\osgeo_utils\gdal_merge.py", line 632, in main
return gdal_merge(argv)
File "M:\Program Files\QGIS 3.32.2\apps\Python39\lib\site-packages\osgeo_utils\gdal_merge.py", line 554, in gdal_merge
t_fh = driver.Create(out_file, xsize, ysize, bands, band_type, create_options)
File "M:\Program Files\QGIS 3.32.2\apps\Python39\lib\site-packages\osgeo\gdal.py", line 2764, in Create
return _gdal.Driver_Create(self, *args, **kwargs)
RuntimeError: Attempt to create 10000x-17999 dataset is illegal,sizes must be larger than zero.
Prozess lieferte Fehlercode 1
Execution completed in 115.30 seconds (1 minute 55 seconds)
Results:
{'OUTPUT': 'C:/Users/*/AppData/Local/Temp/processing_ddhmYt/774dd81a5a754687b6942eea55992655/OUTPUT.tif'}

Lade Ergebnis Layer
Die folgenden Layer wurden nicht erzeugt.
• C:/Users/*/AppData/Local/Temp/processing_ddhmYt/774dd81a5a754687b6942eea55992655/OUTPUT.tif
Im 'Protokoll-Fenster' im QGIS-Hauptfenster sind mehr Informationen zur Ausführung des Algorithmus zu finden.

 

The value "Band1#_FillValue=9.96921e+36" was the default to define the empty parts.
But in the end the result is the same.

 

The detail level of LIDAR looks like it might be worth doing even in an area with completely flat terrain. The resolution seems fine enough to to pick up the contour of drainage ditches, road embankments, etc. If that's the case it would be a huge time saver in the long run even if a route has very little terrain variation.

 

Thank you guys for your awesome work and tutorials.
I´m sure there is still something wrong in my QGIS transformation process, because if i import lidar in asc format to TSW, nothing happens or the surface flattens.

I think the problem is the merging process in QGIS. When in merge my Lidar data (1km x 1km - EPSG 25832 / ETRS89 UTM Zone 32N)
in QGIS, the merged plate will be white, instead of highly detailed as shown in the tutorial video.
Example:
Picture 1 = Lidar Tiles (TIF-format) imported to QGIS

Picture 2 = Merged plate

Does anyone know what´s wrong here and how to fix it?
Thanks in advance.

 

Wow! Very nice. It's cool to see examples of success.

 

I don't know why that happens. I just tried it too with Munich data in GeoTiff format and for me it just works. Did you leave the data type as the default Float32?

 

Yes i did. I also used the ETRS89 / UTM zone 32N EPSG: 25832 as Project CRS before merging.

 

Sorry I cant't think of anything else that causes it. QGIS complicated sometimes

 

I just uninstalled QGIS 3.28.11 LTR - i thought the stable version would´ve been the right decision, but it seems it was not.
Tried QGIS 3.32.3, and it seems to be much better now:
tiles:

merged:

But sadly in TSW Editor, the tiles will be flat, when i try to apply lidar on SRTM.
ASC Files are lookin good (no zeros):

Think i´ll wait for a official tutorial with an example from europe / germany.

 

Not sure but I think the minimum tile size is 2500 by 2500, have no idea if that is true but maybe try? Who knows it might work

 

Been trying to import Lidar data from Defra Survey (Gov UK site) for the Southampton to Portsmouth line, and I get all these wonderful file types and upon importing them into QGIS. I end up with nothing and no layers to merge. What files do these need to be in? Does each Lidar data site have different file formats?

I see Matt is working on a tutorial to import Lidar and all of that so fingers crossed that may answer my question. The video did not help as much as I was hoping for, I only got as far as the 4 minute mark before things went strange.

 

Yeah I'm also excited for the official tutorial.

For me the tiles are flat if the tiles you want to apply LIDAR to aren't actually covered by your imported LIDAR tiles. And make sure the reprojection is right. But looks like it from your Asc files

 

The tif files should work

 

Anyone else having Issues, when trying to Warp the merged Data? It always ends directly with error code 1?

 

Yes, see my post earlier in this thread. I can’t even get it merged.

 

Just noticed that in the editor, under "Lidar Settings", "use pseudo Mercator" is an option, which is switched off in my case.
So why it is critical if, as default, is turned off...

 

Is this code something to do with one or more variable sizes violate format constraints?

If so I found that to fix it I needed to save the merged file to a file instead of a temp file.

 

How do I get LIDAR data for USA? Which file format should I use? I found out that the STRM image was hidden behind the OSM image after unchecking the OSM image in QGIS.

 

Thanks for the detailed tutorial which certainly helps with using QGIS.
Sadly it hasn't worked for me as when I got to the last stage and selected my "home" tile the editor crashed with an error message EXCEPTION_ACCESS_VIOLATION reading address 0x0000000000000304.
If I go through the process again the same thing happens.
I downloaded 28 5km x5km tiles from DEFRA website (DTM 1 mtr) and merged them together.
I then reprojected to 3857 and saved the merged landscape.
I then retiled to 2500 x 2500 and ended up with about 280 tiles.(tif format)
Deleted the half tiles from the edges and then batch converted to get 263 asc tiles.
Loaded these into editor and clicked on use selected assets when they appear in the left hand box Ok.
When I right click on one tile it says "applying LIDAR data" and then crashes.
I will try checking the files again and reload into the editor as a couple of times it saved them and then saved them again .
Thanks again for your tutorial / instructions - I think I understand what we're trying to do but not quite there yet !

 

everything worked for me till i created the new route all the data i got worked but when entering it into the editor mine too just showed a flat surface. Matt says he should have a tutorial setup by tuesday so gonna wait till then. I think my problem was i never had an option to choose EPSG:3857 - WGS 84 / Pseudo-Mercator

 

Further to yesterday's post I didn't touch the LIDAR data but I did Uninstall and Reinstall the Editor to get a fresh start.
I then set up a plug in for my route as Matt recommended and put the SMRT and LIDAR data into that. I created the low res tiles and this time it worked perfectly and the LIDAR data loaded - one tile at a time - exactly on the point chosen for the route origin. there are a few "knife Edge" ridges -probably along the edges of the tiles.

I tried to "Save Current Level As" with a name other than Main but when I restarted the editor I can't get the Landscape back even though the tiles (LT x1-y1) etc are all still in the folder. Is there a correct way to save the Landscape or does it have to be remade every time ?

 

I also have the knife edge ridges. I don't know why they're there. But you can remove them with the landscape tool.
Regarding the saving, I just save using Ctrl+Shift+S. When I restart the editor the level file is not visible anymore but you can reopen it with File > Recent Levels > Your Level.

 

Now I can say that the procedure with QGIS is perfect. Thank you OP.

I tried in dozens of ways but nothing changed, until..... I found out my worst decision, ever.

When an asc is ready can be imported into a "lidar" directory under the plugin.
Until this evening I have always imported the file, added it "use selected asset" and tried to apply it to the tiles.
The imported files have to be SAVED into UE4 format before using it, and the "pseudo Mercator" box needs to be TICKED and, finally....



On the right the HGT and on the left the infamous Lidar 1m .

Next problem will be to delete the gigantic walls separating the Lidar data from the non Lidar.
Unfortunately there are no Lidar data for 2/3 of my route

 

This worked pretty good. But why do my tile edges look like this?

 

Ahhh, glad you figured it out in the end

 

My edges also sometimes look like this. I don't know why. I wonder if when Matt posts his LIDAR tutorial, it would happen there too.
But for now you can remove the edges using the landscape tool.

 

I'm making some progress now on importing LIDAR data but it's very hit and miss. Anyone from the UK know which one I should be downloading?

 

I first clicked on the National LIDAR Survey data Download which brings up a map of the UK. You can then home in on the area you want
and there is rectangle selection button in the right hand margin .Select this and click on one corner of your area and it will then draw a red line and you can continue until you have drawn a rectangle around the area.
Then click on "get available tiles" which will bring up a list of different surveys at the top.
I selected Lidar Composite DTM -2022 _ Dtm1m (1 metre) and it will then list all the tiles in the area you have selected. I had 28 and opted to download one at a time.
Hope this helps.

 

I have noted something that could be a bug or at least an issue (If somebody could replicate it).

Terrain generated from HGT, then covered with Lidar data. Saved yesterday and opened this evening.
I decided to extend some of the HGT terrain to have a better view and the usual procedure does not work.
Top part of the image (lidar), then the original HGT terrain. The flat part is the "new" generation.
Same HGT and I'm not outside the data contained in it.


Can be that the Lidar data needs to be applied only after setting completely the base terrain?
If so, no extension can be made in a second moment...

 

Looking to have a crack at this LIDAR business now I've had a play with track/materials etc. Has anyone had success with UK LIDAR data using these methods above? Or should I wait for Matt's tutorial - looks like he may be v busy though.
Many thanks to Lulon the OP for posting this by the way.