Landscape Models #3 – PLY File Processing

← Part 2

Intro

Once the digital elevation data has been manipulated and edited into a model then any subsequent processing will depend on what the model is to be used for. I have two main uses which result in slightly different workflows:

1. Viewing the resulting model on a screen, e.g. on Sketchfab
2. 3D printing the model

Each of these is described below but to follow my workflow for both you will need some custom software from https://github.com/john-davies/lidar-ply. The two command line executables – lidar2ply and plymenu have been built and tested on Linux only. They should work on Windows as long as you have a C++11 compiler.

Sketchfab Upload

To view a simple 3D surface that created from the LiDAR file then only the lidar2ply executable is needed and is used as follows:

lidar2ply -f landscape.asc -i landscape.png -m

• landscape.asc is the ESRI Arc/Info format file exported in the previous post
• landscape.png is the image overlay exported in the previous post
• the -m option specifies that a mesh is created automatically

A couple of things to note:

• For the -i option each pixel in the image is used to colour the corresponding vertex in the 3D model. For that reason both data and image files have to have the same number of rows and columns.
• The meshing algorithm assumes that there is a rectangular matrix of regularly spaced points. This is not a generic meshing algorithm 🙂

The output should be a 3D model file in PLY format that can then be loaded into Meshlab if any further processing is needed. After that it can be uploaded to Sketchfab for example:

The slightly surreal blocky object in the bottom left hand corner of the island is a low poly representation of the lighthouse 🙂 The sea around the island has been trimmed to reduce the model size below 50 Mbytes.

3D Printing

So far what we’ve created is effectively an infinitely thin surface which is fine for displaying on a screen but is no good for 3D printing as it has zero volume. Somehow we need to create a solid, watertight object preferably with a steady base to make the printing easier.

There’s an old but very informative post on the MathWorks website which describes the problem and suggests a couple of possible solutions as shown by the image on the left.

The surface can either be thickened to give it volume or blocked where the edges are extended downwards then closed off to create a stable base. The post gives some Matlab code which should implement either option but I don’t have  a Matlab licence and I couldn’t get it to work using the open source GNU Octave. Rather than investigate in too much detail and because I wanted to add a few different features I decided to write my own code to implement the “Blocked” approach. Having written some PLY file handling code for the lidar2ply utility then it wasn’t actually too difficult. The resulting program is called plymenu and is also available on the GitHub page. It has a few different features but to use it here the following steps should suffice:

1. Create the PLY file using the lidar2ply utility as above. Unless you have a very expensive 3D printer it’s probably not necessary to add a coloured overlay!
2. Open the resulting PLY file in plymenu
3. Select option “5 – Holes” then “2 – Find holes” followed by “3 – List holes” ( see note 1 )
4. There should be one hole listed with text something line “#1: 2817 verticies”
5. Select option “6 – Fill holes and add a base” and “1” as the hole number
6. You now have an option to extend the bottom edge of the surface to create a base ( the green area in the image above ). This can either be a percentage of the overall height or an actual value. ( see note 2 )
7. Once this is complete then quit back to the main menu and write the new PLY file. It should then be possible to import it into the 3D printing software.

Notes:

1. A key step in extending the surface correctly is to detect the opening at the bottom. This is done by running a hole detection algorithm on the whole mesh and there should be one big hole whose edge is the bottom of the surface. However occasionally there are other smaller holes ( depending on how the mesh was created ) so these are reported as well. Usually these are much smaller than the base so it’s fairly easy to select the correct one. These other small holes will need to be filled before the 3D print slicer will work and something like Meshlab will do this quite efficiently.

2. Two options for extending the base are given to handle the following situations

This is a model of a rock created using photogrammetry. Here the base is a percentage of the height because the actual value doesn’t really matter and it’s just to close off the model and allow it to be printed.

This is one tile from a set of 8 forming a landscape so the height is critical to make sure that all the tiles align so a fixed value was used.

The original recreation for the rock model on the left can be seen here:

See https://theretiredengineer.wordpress.com/2018/10/28/lidar-3d-print-part-2/ for how the 8 tile landscape print was created.

3. The plymenu software assumes that the model is in a sensible orientation with the Z axis running vertically up through the model. This is automatically the case for any models created from LiDAR files but not necessarily true for those created by photogrammetry. In the latter case a tool like Meshlab can be used to reorient the model as necessary.

Conclusion

This series of posts has hopefully summarised my current workflow as regards creating 3D landscapes. Obviously this has only scratched the surface but I’ve been using the same basic process for a while now so I think that it’s reasonably stable. Having said that I’m sure that I’ll be writing an update in the not too distant future 🙂

 

Landscape Models #2 – Processing with SAGA GIS

← Part 1

Part 3 →

Intro

After downloading your data of choice it then usually needs some processing before it can be turned into a 3D model. My process is based around the SAGA GIS tool and I originally chose this tool because I found that it had the best gap filling algorithm. However other tools are available and may now be better. I used version 7.3.0 which is the version currently supported by Ubuntu Mate 20.04.

There are four stages to the process and these are described in more detail below

1. Importing the data
2. Editing the data
3. Adding an image overlay
4. Exporting the data

Note that this is my process developed from experience and reading the manuals. If there’s a better way then please let me know 🙂

Data Import

For this example I’m going to create a model of Skokholm Island ( below left ) off the coat of Pembrokeshire, Wales using 2m resolution LiDAR data. Higher resolution data is available but my free Sketchfab plan is limited to 50 Mbytes per model and even the 2m data will need some compression to fit.

Each LiDAR data file covers an area of 1km x 1km corresponding to the blue squares on an Ordnance Survey map. In this case 6 files were necessary to get the full coverage ( below right ) Strictly speaking only 5 are needed for the island itself but I wanted to crop the final model to a rectangle so the top left one containing just sea was also needed.

Skokhol Island – from Open Street Map

LiDAR Cover

SAGA GIS has a data import tool under Manager > Tools > Import/Export > Grids > Import ESRI Arc/Info Grid but files can also just be dragged and dropped onto the Manager > Data tab and they will be imported automatically ( right ).

The “sm7205” text in the file names corresponds to the Ordnance Survey grid reference.

The final stage of the import is to set the map projection. In this case the data is in OSGB 36 projection but other data may have a different projection. In SAGA GIS this is done via the menu  Geoprocessing -> Projection -> Set Coordinate Reference System menu option, selecting the 6 imported files and setting the projection to the correct value. Note that these files are 500 x 500 pixels which, at 2m resolution, gives a 1km square actual area.

This method of importing works on most DEM files available on the web. the only ones that I had a problem with were the planetary data but in the end I didn’t use SAGA GIS for that particular project. For troublesome formats the open source GDAL tool may prove to be useful.

Data Editing

Once the data is imported then there are a few editing steps that may be needed. Firstly the six separate files need to be combined into one file to make further processing a bit easier. This is done via the Geoprocessing > Grid > Grid System > Mosaicking tool and the result should be a new grid as shown on the left. SAGA GIS will work out the arrangement of the tiles based on the geographic information within the individual files. The new file is 1500 x 1000 pixels in size which is the 3 x 2 arrangement of the 6 original files. The colours are the default mapping which uses blue for the lowest values and red for the highest.

Filling the gaps in the data can be done by running the Geoprocessing > Grid > Gaps > Close gaps tool on the new grid. The only option is “Tension Threshold” which I leave at the default.

The last editing step is to clip the grid to its final size by using the interactive clipping tool. This can be found at Geoprocessing -> Grid -> Grid System -> Clip Grids [Interactive] but beware that it doesn’t have all the nice UI features that you’ve come to expect from a photo editor for example. However once the gap filling and clipping have been done then you should end up with something like on the right.

Note that when using the interactive clip tool there is the option to save the parameters so that the same clip can be applied to other grids. This is useful when overlaying an image from a different source that isn’t downloaded via SAGA GIS ( see below ).

There is one final optional editing step which is to resample the grid ( Geoprocessing -> Grid -> Grid System -> Resampling ) but this should only be necessary if the final model ends up too large. I have used this option but recently I’ve found that a 3D tool like Meshlab does a better job of reducing the model size.

Image Overlay

Once the grid has been edited then it’s time to add a suitable overlay. There are several ways that I’ve done this:

Colour Coding

By default SAGA GIS applies a simple colour code to the values in the grid and can be seen in the screenshots above. It’s possible to fairly easily change these colours using the “Colours” controls in the “Settings” tab of the “Properties window, see right.

There are more possibilities available via the Geoprocessing -> Visualization menu and I used the Grid > Terrain Map View option when creating the 3D map of Wales from the SRTM data:

Map or image via SAGA GIS

SAGA GIS has a feature that can be used to automatically download a map to overlay a grid. This is available via Tools > Import/Export > GDAL/OGR > Import TMS Image and displays a dialog as shown on the right. The main options  that I’ve used are Open Street Map and the various Google offerings.

An example is the area around Neustift im Subaital in Austria where we went on holiday. The cyan and magenta tracks are GPS tracks of two walks that I did ( also imported into SAGA GIS as GPX format files and overlaid onto the elevation grid ).

Two problems that I’ve not managed to solve with this method are:

1. I think that the maps could be downloaded at a higher resolution but I haven’t managed to find how to do that.
2. It should be possible to download from alternative Open Street Map derived sources using the “user defined” option, for example from OpenTopoMap to get contour lines. However I’ve never managed to get this to work.

Map or Image via Separate Source

It’s also possible to overlay an image obtained from a different source as long as the image / map projection is known. If it’s different from the underlying grid data then it can be changed to match the grid using Geoprocessing -> Projection -> Coordinate Transformation ( Grid ).

One obvious source is the USGS Landsat data which is freely available. However this is not without its problems and importing it into SAGA GIS takes a little more processing. I’ve previously written about my attempts using this data – post 1, post 2.

However the overlay is created then it will usually end up as a separate grid within SAGA GIS which can then be exported as necessary, see below.

Data Export

The type of data that needs to be exported will depend on what further processing is needed. For creating 3D models in PLY format that can be uploaded to Sketchfab for example or to be 3D printed then two things are required:

1. The clipped grid needs to be exported in an ESRI Arc/Info format ( the same as the original data input ). This is done via Tools -> Import/Export -> Grids -> Export ESRI ARC/Info Grid
2. Each of the map / image overlays needs to be exported as an image. This is done via Tools > Import/Export > Images > Export image. Usually this is just a single image but for the Neustift example above I had three overlays, the basic Google satellite image and one each for the two GPX tracks that I imported. In that case the three images will need to be combined using a separate image editing tool. The final image should have the same X/Y sizes as the grid export above.

The final part of this series of posts will discuss how I create an actual 3D model from this exported data.