Wurstkoffer
Member
1. About
This workaround allows RTB to obtain the terrain texture and elevation data from another source, which ideally provides more detail than Google. This source can be anything in theory. A GeoTiff, a database or file containing data in XYZ format, another mapping service, such as the many regional WMS[1] servers. It can be used for both public and local services. And yes, it can also be a LIDAR based source. An example is provided which retrieves data from a regional public WMS server. Programming skills may be required to access other sources.
2. Principle
The interface that RTB uses to retrieve terrain texture and elevation data is the Google Maps API[2]. The use of the API is very well described[3]. RTB sends requests for locations to Google via HTTPS and receives replies of type PNG and JSON[4]. This is where the Inject takes place. RTB's requests to Google are redirected to a local service that generates replies based on an alternative source. The redirection is easily done in the hosts file[5]. Since RTB communicates via HTTPS, the local service must pretend to be Google. This is achieved via a self-signed SSL certificate for the address maps.google.com. The certificate must be imported into the Windows Certificate Store as a trusted root certificate authority.
3. Instruction & Example
In this example, a web server is used as the local service that mediates between RTB and the alternative data source. On this web server runs a PHP-based interface that behaves like the Google Maps API to RTB and retrieves the alternative data from a regional public WMS server.
Since some editing is done in files, a good text editor is recommended. Of course the Windows Editor can be used, but this manual always refers to Notepad++.
3.1. The Webserver
3.2. SSL Certificate
3.3. Hosts File
3.4. Test redirection
3.5. The PHP example scripts
3.6. Test fetching in RTB
Since the example Config uses a regional public WMS server from Saxony, Germany, we should choose a location from this region for testing. It delivers laser scanned elevation data in 2m resolution.
4. Finding WMS Servers
Finding other sources is a demanding task. There are a lot of map services out there but you have to find one that delivers elevation data via GetFeatureInfo. Search the web for following keywords: WMS, elevation, LIDAR, GIS, DEM/DSM/DTM and the region you're interested in. A good starting point for Europe is https://www.europeandataportal.eu. If you found a resource, take a look in config.php and enter the appropiate URLs, Layername and Reference System supported by the WMS you found. Everything is described in detail in config.php.
5. Other kind of sources
If you want to use some downloaded data like LIDAR point clouds you need to install and set up a local geoserver and load your data into it. I'm pretty sure that's possible, but I haven't done that myself yet. So you have to do some research. If it's loaded and running you just need to change some settings in config.php.
Edit: @Les Neilson made a short tutorial how to setup GeoServer with downloaded LIDAR data.
6. Code using / improvements
Do whatever you want with the provided PHP code. If you can enhance it further, do it! If you want to rewrite it in proper object oriented code, you're welcome! There are also some frameworks out there for using WMS with PHP. I'm sure there are much better ways to communicate with WMS, for instance evaluating the GetCapatabilities request. It would only be nice, if you share your improvements.
7. Footnotes
[1]
[2]
[3]
[4]
[5]
This workaround allows RTB to obtain the terrain texture and elevation data from another source, which ideally provides more detail than Google. This source can be anything in theory. A GeoTiff, a database or file containing data in XYZ format, another mapping service, such as the many regional WMS[1] servers. It can be used for both public and local services. And yes, it can also be a LIDAR based source. An example is provided which retrieves data from a regional public WMS server. Programming skills may be required to access other sources.
2. Principle
The interface that RTB uses to retrieve terrain texture and elevation data is the Google Maps API[2]. The use of the API is very well described[3]. RTB sends requests for locations to Google via HTTPS and receives replies of type PNG and JSON[4]. This is where the Inject takes place. RTB's requests to Google are redirected to a local service that generates replies based on an alternative source. The redirection is easily done in the hosts file[5]. Since RTB communicates via HTTPS, the local service must pretend to be Google. This is achieved via a self-signed SSL certificate for the address maps.google.com. The certificate must be imported into the Windows Certificate Store as a trusted root certificate authority.
3. Instruction & Example
In this example, a web server is used as the local service that mediates between RTB and the alternative data source. On this web server runs a PHP-based interface that behaves like the Google Maps API to RTB and retrieves the alternative data from a regional public WMS server.
Since some editing is done in files, a good text editor is recommended. Of course the Windows Editor can be used, but this manual always refers to Notepad++.
3.1. The Webserver
- Download XAMPP for Windows
- Install it:
- The required components are Apache and PHP. All others can be deactivated.
- The installation directory used in this example is C:\xampp. Make sure there is enough space on the drive. The example script allows you to save all map tiles to create a high-resolution terrain texture later on. This can generate 3-5GB of image files.
- Further installation offers, such as Bitnami, are adware and should be rejected.
- When the Windows Firewall logs on, allow access to all networks.
- At installation completion, allow the Control Panel to start.
- For faster access to it in the future, anchor the Control Panel in the Windows Start Bar.
- Test it:
- Click the start button of the Apache module and make sure it listens on ports 80 and 443.
- If everything's ok, stop Apache again. If not, ask the web for help.
- Click the start button of the Apache module and make sure it listens on ports 80 and 443.
3.2. SSL Certificate
- Before the certificate can be generated with the batch file makecert.bat located in C:\xampp\apache\, the batch must be modified.
- Rightclick makecert.bat and choose "Edit with Notepad++". Change the line "bin\openssl req -new -out server.csr" to "bin\openssl req -nodes -new -out server.csr". This prevents from stating a password for the certificate. Save it and close Notepadd++.
- run makecert.bat by double click.
- A black window appears and you will be asked to enter some information. The first five can be left blank (just hit return). The sixth asks for "Common Name (e. g. server FQDN or YOUR name)". Enter "maps.google.com" there (without quotation marks, of course). Proceed with blank inputs until "The certificate was provided" appears and complete.
- Go to Internet Options (hit Windows Key, so the Startmenu comes up, type "internet options")
- Switch to "Content" tab and click on "Certificates".
- Switch to "Trusted Root Certification Authorities" tab and click "Import".
- On import hit "browse" and choose server.crt from folder C:\xampp\apache\conf\ssl.crt\
- Proceed to the end. A security warning will appear. Just hit "Yes" to install the certificate.
- After successful import, "maps.google.com" should appear in the list of certificates.
3.3. Hosts File
- Run Notepad++ as Administrator
- Open "hosts" file from folder C:\Windows\System32\drivers\etc
- Add a line without a leading "#": "127.0.0.1 maps.google.com".
- Save the file and exit.
- From now on it is no longer possible to reach "https://maps.google.com/" in your web browser. If it becomes necessary to use Google Maps, alternatively use "https://maps.google.co.uk/" or temporarily deactivate the redirection in the Hosts file by inserting a "#" at the beginning of the previously added line and save the file.
3.4. Test redirection
- Start Apache Module in XAMPP Control Panel
- Open your web browser and visit "http://maps.google.com/". The browser may refuse to call it via HTTPS, so just use HTTP.
- If you still reach the original Google Maps Website try the following:
- restart your brower
- use another browser
- flush dns in command prompt (hit windows key, type cmd, hit enter, type "ipconfig /flushdns").
- If everything's working, a welcome page from XAMPP should be served.
- If you still reach the original Google Maps Website try the following:
- Start RTB (if it's already running, close it and restart)
- At "Create New or Open Venue" window click Google button. An error should appear: "Can't load Google Maps image. The remote server returned an error: (404) Not Found.". That's fine. It means the redirection works.
- Let's take a look if the server received a request. Open "ssl_request.log" file at folder C:\xampp\apache\logs\. It should look like this:
3.5. The PHP example scripts
- Extract the content from RTB_WMS_Inject.zip to C:\xampp\htdocs\ so it looks like this:
- Let's see if the scripts work as expected with default settings.
- Open config.php from folder C:\xampp\htdocs\maps in Notepad++. Scroll down to bottom and add your Google API key by replacing the fake value at $key variable.
- Visit "http://maps.google.com/maps" in your web browser. If you still got problems to reach it, try "http://localhost/maps". A Config Test site should load. If error messages appear, try to understand and solve them
. If everything went well, you'll see 3 map tiles and 3 elevation responses from google, wms and a merged version from localhost:
In the WMS pictures you will spot a white border in terrain. This appears, because the WMS server contains map tiles with bad alpha channel.
3.6. Test fetching in RTB
Since the example Config uses a regional public WMS server from Saxony, Germany, we should choose a location from this region for testing. It delivers laser scanned elevation data in 2m resolution.
- So, switch to RTB (or restart), paste "50.842566,14.7440267" at the "Set Latidude/Longitude" input box and hit the Google button again.
- Zoom in a bit.
- Hit OK, and OK again to let RTB start the project and receive data from WMS.
- While RTB is fetching, take a look at the log file at C:\xampp\htdocs\maps\logs. Every block starts with the request URL that was sent from RTB, followed by some details from Googles and WMS's responses for every single location from the request.
- Back in RTB, when Terrain is loaded, start to sub divide a bit. But be warned. You have to be really patient. The WMS server delivers responses much slower as Google. A way to improve that drastically is to find download resources and use it with a local geoserver.
- If you sub divide till 9 or 10, you will see some ugly level steps. Unfortunately, the WMS simply does not interpolate elevation between measuring points... or rather: a measuring point is a 2x2 meter pixel.
There is nothing we can do about that at the moment, except manual flattening. You can certainly achieve much better results with higher resolution material, which may be available as XYZ points or vector data.
4. Finding WMS Servers
Finding other sources is a demanding task. There are a lot of map services out there but you have to find one that delivers elevation data via GetFeatureInfo. Search the web for following keywords: WMS, elevation, LIDAR, GIS, DEM/DSM/DTM and the region you're interested in. A good starting point for Europe is https://www.europeandataportal.eu. If you found a resource, take a look in config.php and enter the appropiate URLs, Layername and Reference System supported by the WMS you found. Everything is described in detail in config.php.
5. Other kind of sources
If you want to use some downloaded data like LIDAR point clouds you need to install and set up a local geoserver and load your data into it. I'm pretty sure that's possible, but I haven't done that myself yet. So you have to do some research. If it's loaded and running you just need to change some settings in config.php.
Edit: @Les Neilson made a short tutorial how to setup GeoServer with downloaded LIDAR data.
6. Code using / improvements
Do whatever you want with the provided PHP code. If you can enhance it further, do it! If you want to rewrite it in proper object oriented code, you're welcome! There are also some frameworks out there for using WMS with PHP. I'm sure there are much better ways to communicate with WMS, for instance evaluating the GetCapatabilities request. It would only be nice, if you share your improvements.
7. Footnotes
[1]
"A Web Map Service (WMS) is a standard protocol for serving (over the Internet) georeferenced map images which a map server generates using data from a GIS [remark: Geo Information Service] database. [...] GetFeatureInfo - if a layer is marked as 'queryable' then you can request data about a coordinate of the map image." (for example elevation data) - https://en.wikipedia.org/wiki/Web_Map_Service
[2]
"[...] a service for retrieving static map images, [...] and obtaining elevation profiles." - https://en.wikipedia.org/wiki/Google_Maps#Google_Maps_API
[3]
https://developers.google.com/maps/documentation/static-maps/intro
https://developers.google.com/maps/documentation/elevation/start
https://developers.google.com/maps/documentation/elevation/start
[4]
When creating the terrain texture, RTB requests multiple map tiles via Google Static Map. It sends coordinates via Center Lat/Lon, in conjunction with zoom level, format, width & height of the tiles and the API key in this format:
A tile will then be received:
A similar request is made to receive elevation data. Here, however, several locations are transmitted, separated by pipe:
The response comes in JSON format:
These requests could be confirmed with the help of mitmproxy.
Code:
https://maps.google.com/maps/api/staticmap?center=50.8355467,14.7050059&zoom=18&size=512x512&scale=2&format=png&maptype=satellite&sensor=false&key=A similar request is made to receive elevation data. Here, however, several locations are transmitted, separated by pipe:
Code:
https://maps.google.com/maps/api/elevation/json?locations=50.8457672356934,14.7382761104377|50.8477157928223,14.7382759988057|50.8477157928223,14.7409487&key=
Code:
{
"results" : [
{
"elevation" : 450.2297668457031,
"location" : {
"lat" : 50.8457672356934,
"lng" : 14.7382761104377
},
"resolution" : 152.7032318115234
},
{
"elevation" : 494.1800537109375,
"location" : {
"lat" : 50.8477157928223,
"lng" : 14.7382759988057
},
"resolution" : 152.7032318115234
},
{
"elevation" : 485.4239807128906,
"location" : {
"lat" : 50.8477157928223,
"lng" : 14.7409487
},
"resolution" : 152.7032318115234
}
],
"status" : "OK"
}[5]
"The computer file hosts is an operating system file that maps hostnames to IP addresses." - https://en.wikipedia.org/wiki/Hosts_(file)
Last edited:
