I came to QGIS and done all steps so far, downloaded all LIDAR data, installed QGIS and attempt to merge all tiles together in QGIS. But the "Merge" function emits errors and does not create the merged tif file. I don't know what the exact reason for this is. I followed all steps by Lucas in the Tracks & Signals masterclass stream. However, there are two reasons which could lead to this problem I think: 1. The LIDAR tiles itself which due to the route being divided across 2 different states and thus having different LIDAR data tile sets to come together. tif files from Hessen and xyz files from Rhineland-Palatine. I chose DGM 25 for LIDAR data Rhineland-Palatine (offered 25 or 50). 2. Rhineland-Palatine's LIDAR tile set is huge as it covers the entire state. There is no ability to just download (and identify) tiles by municipals just like it is the case for the municipals for the cities at the Hessen side of the route. Can anyone help me with that or give me hint what do with the error messages? I'm not that familar with QGS yet, so I have no clue what to do. Error codes: Generating the merge: Here you see both different LIDAR sets together (left Rhineland-Palatine and right - the small one from Hessen):
QGIS has some trouble with the xyz format. First step should be converting all xyz to xyz by QGIS (sounds weird, but regenerates the xyz files in QGIS friendly format). Then try to merge again.
Thank you for the help. Okay, but how do I convert xyz files to QGIS friendly xyz files? I found the Translate (Convert Format) menu but as it seems I only can convert one tile at once (The three dot menu on the right side does not allow the selection of multiple files). I have 252 xyz files and I can't do one by one. This really would exceed my capacity. I googled for it but only found a way to convert xyz to other formats while using the merge tool which does not work for me. OR is there a different way to convert them? Again, I have no clue about this software. So speak everything you know like to a complete novice.
UPDATE: I tried to merge only the xyz files for the LIDAR data of Rhineland-Palatine, and the result is the same (Failed conversion): On the other hand, I tried to merge only the tif files for LIDAR data of Hessen, AND IT WORKS!!! So, the problem is clearly only with the xyz files, not the tif files and probably also not due to the merge of xyz and tif files together as I assumed before.
Can you send me some of the xyz files so I can try it myself? It tried downloading it on the website but it's too complicated I don't know if you can send me files on the forum? Try attaching them in the next post or send them on discord: lulon
Thank you very much. I actually used this link to download them. The resolution is only DGM25, but the state only offers this as highest resolution. It's requires 1GB of storage. https://vermkv.service24.rlp.de/opendat/dgm25/dgm25.zip This is the page where you can find the links to the DGM25 and DGM50 offered by the state RLP: https://lvermgeo.rlp.de/de/geodaten-geoshop/opendata/
I´m not at my QGIS computer right now, but you can batch convert all xyz files into xyz format again. Try to follow this procedure: https://forums.dovetailgames.com/threads/lidar-tutorial.74374/page-3#post-743212
If you put only one xyz into QGIS it looks like this: The xyz's have two bands. You cannot merge something with two bands. Band 2 is alpha so you need to get rid of it. You can do this using GDAL Rearrange bands. It's in the toolbox under GDAL > Raster conversion > Rearrange bands. In selected bands you need to check only band 1: Then optionally depending on how you want to do this set an output file. Then click run. Then you get the single band rasters. Unfortunately, you can only do this one by one. There is this "run as batch process" button but for me it just stops randomly so I check the command output which is gdal_translate -b 1 -of GTiff "C:/blahblah/DGM25_2905530.xyz" "C:/blahblah/DGM25_2905530.tif" then write a command line script or whatever to run it for every file. Anyway then you can merge them.
So, I tried the guide by CosmicDebris: I tried these steps the whole day, but with the change that I used .xyz files instead of .asc in step 11. I also needed to limit the Lidar tiles as the data set I obtained covering the whole entired state of Rhineland-Palantinate was just too huge to process. So, I needed to find a way to narrow down the area, which I only need for my project, which is the northeast of the map. Since the tiles loaded not in a chronological order but following a pattern from right to left, up to down, but not based upon certain areas, I needed to look at each tile whether or not it fits to the data I need. So I went through all 252 tiles and deselected all tiles which I don't need. So starting from intially this, the complete state map: I went over to deselecting certain tiles to come to this finally: I left a little bit more tiles than needed in case I want to extend the route or to use the set also for some other routes I had in mind. But in the end, I shrinled from 242 to 40 tiles. Now I wanted to wrap all of these and re-convert them into new QGIS friendly xyz files. Unfortunatetly, the Wrap menu had no possibilty to fetch simply all currently selected layers, so I needed to set up each row for each tile by its name on my own (a huge process). Now I gave this a run and it worked. However, for the 40 tiles it created EXTREMELY BIG new xyz files. All in total covering more than 300 GB! - I needed to port the process to an external hard drive in order to do so. Now, I'm stuck when importing these 40 tiles, where the biggest tiles have a size about 14 to 15 GB, back into QGIS. Guess what? QGIS does not react. So, I think this option unfortunaltely does not work for me either. The generated xyz files are way too large to handle them within one project.
So here is the official confirmation, QGIS emitted an error message after loading process "Invalid Data Source: ???.xyz is not a lavid or recognized data source."
I don't know how you can see these 2 bands in the tile. When I look in my QGIS installation, I only see a single layer. Is this a specific setting thing?
It's also not because of the size of the .xyz files. I tried to merge just 5, including setting all with their CRS value to EPSG:3857 (WGS 84 / Pseudo Mercator) or to ETRS89/UTM32 (EPSG:25832). No change at all.
I don't know why it shows me the bands. It doesn't always do it. But you can also see the bands if you double click on the layer. So I converted all of the RLP tiles to single band tif files (they are also much much faster to load in QGIS). And I still wasn't able to merge them. So the solution I posted above was only part of the real solution. But I was curious too so I kept trying. It was very hard to find the actual problem. Your xyz files have so called NS positive pixel resolution and GDAL merge doesn't like that. so I first had to reproject them all to EPSG:4647 - UTM zone 32N (zE-N). In the end I was able to merge it. And because I already did it, you can just take it if you want: https://cdn.discordapp.com/attachments/570719032651808790/1171223617472831508/merged_25832.tif (236 MB, only RLP DGM25 in EPSG:25832 - UTM zone 32N, no-data -9999). I hope it works if I send it that way.
Thank you so much!!! That was so kind of you! I hope that everything works well with it in the Editor now, if I try it out. I thought about converting the .xyz to .tif files this night, but as you said this probably wouldn't have worked either because of the file structure. I almost was at giving up this project and this now brings new hope to it! I will definitely mention you if I can realize one of these routes as contributor to the project. One last question (but if not I will try to obtain it in a different way or just let leave it): What about the SRTM data? I heard it is important for the terrain in the distance. I mean in southwest direction I have enough Lidar data, so I don't need it for there, but what about the SRTM data of southeast NRW and southwest Hessen parts of the map which surround the route? I installed the relative "SRTM downloader" plugin in QGIS but didn't came that far to introduce to my project. Did you embedded that in the file already? Or should I do so in any other way now?
So this is what it looks like now with all tiles from RLP and necessary tiles from Hessen in my QGIS installation. There are blacked out tiles at the edges. I guess this is correct since you made the tiles from RLP smaller and now accord correctly to the territory of Rhineland-Palatine and Nordrhine-Westphalen. I placed the KLM file in the project and on top of all layers, so you can see where the route is. At the northest border of the route, and the last 2 stations around Au (Sieg), there is the border to NRW. Not sure whether I should use NRW tiles for the bit of map around Wiedeck or this will be good enough as it is (maybe just with adding the STRM data it will be just fine). However, I'm just happy as it already is currently. EDIT: I occasionally found out that Au (Sieg) - the final station of my route - is just for a tiny bit of map on NRW territory. But the Lidar tiles from Rhineland-Palatine covered it nonetheless, just the area a bit above Au is not covered. I'm thinking about whether I should take it in or not.
Haha, I'm glad it helped you. Never give up. There is always a way that works. I think setting the LIDAR terrain up is one of the most difficult parts of making a route without a tutorial of if you have weird data. From there on it's just doing the same things over and over again. SRTM is just like the DGMs you have but only with lower resolution. So it is useful for tiles that are far away where one wouldn't notice if it's LIDAR or SRTM. There is SRTM with 90 meter resolution which you can download anywhere and SRTM with 30 m resolution which you can download if you have a free account. Here https://cdn.discordapp.com/attachments/570719032651808790/1171472180652027976/SRTM.zip these two HGT files cover your route. You can use them directly in the editor. These SRTM files have 30 m resolution, so it is just slightly blurrier than your DGM25 with 25 m resolution. And I think you should definitely use the LIDAR from NRW as it is free, easy to download with an interactive map and even has 1 meter resolution. You can download it here if you haven't found it already. Only if it's too far away you should use SRTM because it's easier.
Thank you very much for the download hint. It was like you can read my mind, because I found it way harder to find the point where to get the NRW Lidar data than it was for RLP. Also got confused by Open Lidar NRW project which let you only see the map but now download the tiles. Here is how it looks now. And you had right, it was a good decision to get the NRW tiles in. Because they provided even DGM 1 Meter resolution tiles for some part of my route: In detail:
Nice! Looks like you don't even need the SRTM now apart from maybe a bit in Hessen. I'm looking forward to playing your finished route some day!
The problem is if you just have frustration from the very first beginning, it really breaks your optimism and hope to continue. I didn't set up a single asset yet in 4 days in the project already. The first day I set up the KLM data in Google Earth and for the last three days I'm messing with the first steps to get the Lidar data in right format to convert and reconvert it. I really only can hope that this project will make more fun in the future. Again, many thanks for the HGT files. I didn't quite know whether the SRTM data needed to be embedded/merged with the Lidar data but both can set up independently, that's why I asked. Is there a threshold for how long the SRTM data should cover from the route? 20/30/40 kilometer or something like that?
As you know, I was the one who posted the LIDAR tutorial. Do you know how long it took me to figure this out? Once the editor was released I think it was almost 3 days non-stop trying everything out. But that's okay. Because it's one thing you have to do at the start of making a route and then never again. And if making a route can take a few months to a few years, I don't care if setting the terrain up takes 3 days. Once you get used to laying the tracks and work around the bugs which can also take a couple of days until you learn everything it will be much easier Yeah, you need to create the SRTM tiles first in the editor and then apply LIDAR. But after that you can still add new SRTM tiles. I don't know how far the SRTM has to go, I would start with a few kilometers and then play in editor. If you look at the horizon and see the terrain cuts off, it means you should extend the SRTM tiles. But you can always do this later and I haven't bothered with it yet. But it looks to me like it's about 10 kilometers?
So, I tried to merge everything, the RLP, NRW and Hessen tiles and I still got errors. As I read one of the errors, it says something like that I don't have the necessary disk size for the merge file 154551602500 bytes (154,55 GByte) is at least necessary for it. It says that I can omit the check, but however shouldn't I have the storage available for the merge tif? What should I do now? Truncate the tiles? But the "Cut" Feature in the tool bar is greyed out. How do I shrink the merged tile of RLP? And why the hell do I always get such huge files out of conversions in QGIS, way larger than the source files?
I think because you now have the DGM1 the output file must have a high enough resolution to support it. Yes, I would cut out the not important part. And yes the tiles will get very large at the end. Because of the .asc format that we need at the end, only half of my route which is DGM1 is 70 GB. So either you cut out much of the RLP part and merge everything together which results in everything being DGM1 resolution. Or you merge the NRW and Hessen tiles seperately and import them seperately into the editor which should work, too but you might get slight seams between NRW and RLP in the editor. If you want to cut it, you can use GDAL Clip raster by extend. It lets you draw a rectangle on the canvas. But I had to do it myself so I could tell you with certainty how to do it: https://cdn.discordapp.com/attachments/570719032651808790/1171516435772747796/merged_clipped.tif I hope this area is enough
Thank you for the clipped RLP!!! This saves me so many time to figure the respective tools out! So I merged now all layers independently to save some resources while merging. I merged all NRW and Hessen files separately. Then I added your clipped RLP layer. And tada - it worked to merge all three layers together! However, each layer of the three has now their blacked out tiles burned to the layer and when I try to merge all 3 layers, there always happen to be black tiles doesn't matter in which order I put the layers. This is the best result with clipped RLP at top, and Merged Hessen and Merged NRW beneath. We only need a way now to bring Nordrhein-Westfalen, Rheinland-Pfalz and Hessen seamless together that no "dead" tiles are left over. This is the worst result with clipped RLP beneath merged_hessen and merged_nrw: Should I try it again with all NRW and Hessen files unmerged connecting it with clipped RLP?
I remade the merged file without the black parts: https://cdn.discordapp.com/attachments/570719032651808790/1171535354780725338/merged_new_25832.tif Try putting this on top.
So, I've put your black-tiles cropped RLP merge and added it this morning into QGIS, everything seemed perfect so far (only with the black tiles on the Hessen merge): I've run to merge this and again the same errors as initially. I then thought to go a few steps back and remerge the NRW and Hessen tiles. So I did with NRW and also added a bit more Lidar data from the districts which are pretty close to the route and you talked about it to be covered by the SRTM data before. So've added some more tiles to for the Hessen part along the old Hessen merge: Unfortunately, to merge these new tiles with the NRW, the RLP and the old Hessen merge didn't worked either. After 2 hours and 17 minutes, the merge failed due to an allocation error again (although I had enough space on the disk): Now I thought to restart with the Hessen tiles completely from scretch from this evening and merge the new and the old Hessen tiles together first, then add the NRW and RLP merges. This idea worked well so far and looked like this. Only black tiles at the right side of the new complete Hessen merge: Now the last hour I've spent to try to find a way to merge these last 3 merges of RLP, NRW and Hessen together. But everytime I do so, I get a strange strange error saying "numpy.core._exceptions.MemoryError: Unable to allocate 5.40 GiB for an array with shape (95025, 61025) and data type bool Process returned error code 1" although - and this important to know and how I spent many time in the last munites - I just cleared the hard drive where I store the final merge tif file up to 115 GB of free space. So I do not understand why it says it can't allocate an array / storage with it. I'm completely messed up now, I don't now how to proceed anymore further now. I could start anohter last attempt by maybe get rid of all the merges and try to merge all "pure" tiles from NRW, RLP and Hessen to one single merge. But therefore I would need to know how you exactly did that, converted the original RLP to your merge: To be honest, I'm short before giving up. I spent the last 4 days in QGIS to end up stucking always at the same point over and over again, merging the RLP layer/tiles. What I wonder about is how DTG Lukas could work with the Rhineland-Palatine files without a problem to create the map for Linke Rheinstrecke from Koblenz to Mainz. The Lidar data for Rheinland-Pfalz is so messed up, I have no clue to get anything going with that without even speaking about that the resolution (DGM25 as its best) is so poor in comparison to Hessen and NRW with both DGM1 resolution and the fact you can only download the Lidar data for the whole state, but not a narrowed piece of the map (which also was the reason to some problems we had here).
Sad to hear that it didn't work. The error is probably about your RAM and not you disk space. You need 5.4 GB free RAM to do this. Do you have enough RAM? Although where it says 23195602500 bytes that would be 23 GB RAM. I mean even if you have that it's still kinda overkill and you should try to split the whole thing in half or narrow it down meaning you would then rely on SRTM on tiles that are further away. The problem with merging is that it converts everything to a tif file. And tif is a rectangular file. Even if your route isn't rectangular, it will still save all the blank information that you don't need inside the rectangle around your route. As an alternative you could try to treat NRW, RLP and Hessen independently, i. e. skip merging them together. Only merge NRW, only merge Hessen. Then reproject them, tile them individually, convert to asc and import them into the editor. It should work but might require some work where the regions meet, later. Admittedly, your route is in a suboptimal place stretching over 3 states. And DGM25 isn't great either. My route has a similar problem as half of it is in Lower Saxony where LIDAR data is not free. I guess Lukas did the same stuff as we are doing now. But I can understand your frustration.
Ah, yes, allocating an array is RAM, not hard disk. Confused that. But I wonder about that because I should have 5.4 GB free. But no I do not have more than 23 GB RAM, I currently only have 16 GB RAM - and for the use I did so far it worked pretty well. My desktop PC could technically be upgraded to 64 GB RAM, but I'm not planning to do so only because of this unfortunately. I think I have now 3 options left. Either, 1. I find a way to do get rid of the NS positive pixel resolution and merge the RLP tiles, like you did. But only take the Lidar tiles near the route which I really need before and then merge them and try it all again with the NRW and Hessen files. 2. I import merges as you said separately to the PC Editor. And then I need to smooth and adopt the surface with the tools there. or 3. Leave the project and create a route in Hessen or NRW, but is sad because I have a personal relation to the route I wanted to create and this was a big motivation for me personally to get it finished. But thank you very much for your help so far. That was really amazing and awesome! Noone could expect more from you to help out. I only have one question left about a tool I tried out but didn't get to the result I wanted, which is the tool to cut mergers, the GDAL tool under "Raster -> Extraction -> Clip Raster by Extent". How did you used that exactly to trim the RLP layer? If I select under the drop down menu "Clipping Extent -> Draw On Map Canvas" and I select a rectangle on the map and hit run nothing changes or it even adds more black tiles. How did you use the trim tool?
If you want to narrow down the RLP part, I suggest you do that with the merged file I sent you. Because there you only have to open the GDAL Clip raster by extent menu, to the right of clipping extend click on the little arrow, choose Draw on Map Canvas, and then draw your selection, then click run and it should create a new layer. Remember to hide the old layer as otherwise it looks like nothing happened. I don't know if it works as well with the raw xyz data. If you want to remove the NS positive stuff yourself, all you need to do is to reproject everything to EPSG:4647 - UTM zone 32N (zE-N). Nothing else. You can then just merge. You might need to reproject back to UTM32N afterwards. But you could still use my merged file. If you narrow it down to about 2 km distance from your route, it should be fine. Tiles further away can be SRTM. In the end you can always add more LIDAR if you are finished with your route if done correctly. I get that the route is personal to you, mine is too, just really unfortunate that Rhineland wants money for the DGM1. I can only say the DGM from NRW works perfectly (in case you wanna do NRW). And you're welcome.
Hello, I received this folder from Geodata (Portal) NRW and imported the XYZ folder. Now I get this image. Where and where do I need to reformat the tif files so that I can work with them in QGIS?
So i do like Tut video shows with SMTR downloader an befor OSM Standard, so i get some Layers, but if i Warp and Retile it, it will be blank. Anything i do wrong i think. If i switch out the alpha 2 i get a map in 2d without landscape markers
Hi Steffi, I didn't reply because I didn't understand your problem quite well enough yet. I recommend you to create your own thread and elaborate all the steps you done and tried so far and where exactly which problem occurs. This thread was raised mainly because of a problem I had with merging tiles from Rheinland-Pfalz and Hessen (optionally even NRW) together and as later turned out has something to do with the incorrect codecs of the RLP lidar data and in the end unfortunately it didn't solve my problem in this (I had no time until yet to explain in detail how I actually ended up). I don't think switching out alpha channel it will be good in your case.
