TOOL LIDAR data - available for free for England

[assettocorsamods]

Are you a track modder ? Good ! Are you passionate about UK tracks ? Well, even better !
Because the UK's Environment Agency decided to make all the data collected in the last 17 years, available for everyone to use for free. The applications are various, but speaking only for our field of interest, this is huge news for every track modder out there. Getting accurate reference data is one of the most difficult tasks without LS when aiming for a top quality project. Thanks to this decision, it's now possible to access all the 11 terabytes of LIDAR data, and with it most definitely that special corner of England which might interest you .

How does this data look like ? Something like this :

Point clouds are available at different resolutions, ranging from 25 cm to 2 m (as of now 72% of England is covered).​

Some possible projects ideas to get you started:

Truxton

Outlon Park

Croft

Rockingham

...and many more !

Here is the DOWNLOAD link, and for more information (source), here's the official announcement:

"For the last 17 years the Environment Agency has used lasers to map and scan the English landscape from above to help us carry out work such as flood modelling and tracking changing coastal habitats.

This month, for the first time, we are making our LIDAR (Light Detection and Ranging) data available for everyone to use for free. The aim is to help organisations, businesses, and individuals to do everything from manage forests, discover hidden archaeological features, and even create virtual reality worlds for gaming.



All 11 terabytes of our LIDAR data (that’s roughly equivalent to 2,750,000 MP3 songs) will eventually be available through our new Open LIDAR portal under an Open Government Licence, allowing it to be used for any purpose.

We hope that by giving free access to our data businesses and local communities will develop innovative solutions to benefit the environment, grow our thriving rural economy, and boost our world-leading food and farming industry. The possibilities are endless and we hope that making LIDAR data open will be a catalyst for new ideas and innovation.

The Environment Agency Geomatics LIDAR portal has been supplying composite data for the last six years but, with technology moving on and the insatiable appetite for data growing, this was reaching the end of its life. To replace it we had to develop a new way of delivering the data which was completely self service and able to deliver more data to a wider range of people.

Behind the scenes we have been working hard on this new service not only to make as much of the data available as rapidly as possible but also to ensure that our system was robust. At 11 terabytes, the total size of our LIDAR datasets made this quite a logistical and technological challenge and, coupled with working to a tight timescale, this has meant rolling out the data in stages.

Composite LIDAR data, which is annually updated and brings together all data from the last 17 years, has already been released. Composite data is available at 25cm, 50cm, 1m, and 2m resolutions, which in total covers over 72% of England.

The next data to be released will be Tiled LIDAR, which comes from individual dated surveys offering a ‘snapshot’ of locations at specific times. This will be followed by different data formats in the coming months and finally point cloud data.

Our new portal is still a work in progress, and we’re interested in your feedback to see where it is possible to tweak our systems and improve the experience both for existing customers and those who are new to our open data offering.

This release comes as part of our wider Open Data plans, and as part of #OpenDefra, in which we are a key part of helping to unlock 8000 datasets across the Defra network. Defra is changing as an organisation; and this is how we will do business in the future.

LIDAR data – some surprising uses:

• Archaeology: Archaeologists are some of the biggest users of the Environment Agency’s LIDAR data: it enables them to map and visualise ancient landscapes and discover subterranean features. For example, our surveys are helping to find Bronze Age burial mounds in Monmouthshire and Roman roads in Cheshire.
• Computer games: Minecraft players have requested our LIDAR data to help them build virtual worlds: the data could be useful to anyone creating realistic 3D worlds.
• Walking apps: We provide LIDAR data to orienteers who use it to create slope maps. This could potentially be used to create new walking apps.
• Civil engineering: A variety of transport, energy, and utility companies use our data to help them manage their infrastructure and plan engineering projects.
• Forestry: Our data help bodies such as the Forestry Commission map forests as well as count and measure the height of trees so they can manage woodlands more efficiently.

You can find further updates through our Twitter orFacebook or contact us at opendata@environment-agency.gov.uk if you have any feedback or suggestions."

Thank you UK Environment Agency .
And to you modders, good luck, have fun, and show us what wonders you can make out of this !​

 

[Pixelchaser]

been playing about with this. i can convert to dem files, that digital elevated mesh files in TIFF format. these can be used as height maps quite well through 3dmax.

the data can include surface detail or just terrain detail. i.e with or without trees ,houses etc...
3 separate tiles ive processed so far. it must be 1pixe per metre data here. 1000 pixel wide tiles. hopefully myself and Fernando can crack its other more convenient ways to use the data,

 

[luchian]

Very interesting guys ! Looking forward to see more

 

[Nibbles]

Interesting. The Strategic Road Network has, in a number of areas, also been LiDAR recorded which might be interesting to anyone who wants to create a track using some of the routes. You could, for example, create a circuit that does a lap of the M6/M42/M5 around Birmingham

 

[Pixelchaser]

I have thought of that. similarly all roads in Google data have an API too which might be useable in Race Track Builder in the future. between the two sources it would be great to create a track from that.

 

[Nibbles]

Re LiDAR, I do wonder how useful the data would be. As i understand it, LiDAR surveys of racetracks are normally done from static points around a track (perhaps every 50m?) with a full 360 degree scan taking a few minutes. Highways England's LiDAR survey has been carried out using a scanner attached to the back of a vehicle, taking a snapshot every 2m. The data looks a bit like this;

Does that appear to be useful?

 

[luchian]

Yes it does from my point of view. No matter the reference data, you still need to create the meshes manually. But instead of visually approximating everywhere, you have a point cloud to align your geometry to. And this is huuge help when aiming accuracy.

 

[Nibbles]

I've asked the question whether this data would be available, and will let you know the outcome.

Thanks
Steve

 

[Pixelchaser]

that level of information is exactly what is required.

 

[Mr Whippy]

Nibbles, that data looks great!

I've worked on some mobile laser scanned roads for sim use and the data looks comparable.

I've also used photogrammetry based reference data too which gets pretty nice results that look a little like what you posted above.


If that level of data could be made available it'd be great for people wanting to make UK road courses. I know I'd love to make some UK roads... it's been a passion of mine for a long time but accuracy and data collection is always the hardest bit (for free hobby tracks at least!)


Cheers

Dave

 

[Pixelchaser]

well I got a little feel of this level of Laser data over the weekend and its quite incredible. however its not easy data to work with. uncompressed about a mile and a half of road is 5 gig of data points and I don't have a super computer to handle it.

but I am making some real roads for driving on none the less. it wont ever be top quality racing track quality, but real roads and functional it should be.

 

[Mr Whippy]

Pixelchaser, you can just reduce the density in CloudCompare easily, and also combine them together.

Obviously it depends how you want to work with the point cloud data. In Max with Clouds2Max I can drape nice quad meshes onto the points and get nice colour data into textures too, so useful for road markings etc.

But I know those in Blender etc may want to generate meshes in CloudCompare and then get textured meshes in Blender to work from.

Dave

 

[Pixelchaser]

hi mr whippy. you sound delicious !

any hints on cloud compare and reducing the volume there? clouds to max sounds interesting. must look for that. but I need cost free solutions tbh.

 

[Mr Whippy]

Haha

In Cloud Compare there are quite a few reduction methods, statistical analysis reductions (based on analysis of each point vs neighbours etc), to random reductions of points etc.
I forget the buttons now, but it's quite straight forward, I figured it out without the manual... though I did read it later for some other stuff

Yeah Clouds 2 Max is great, but sadly it's not heavily maintained (a bit annoying given the price!) so each version has great features but also some bugs... and swapping between versions is a real faff too unfortunately.
I have work arounds but they're not ideal given basic functionality like gamma correction on point cloud colours should be right.

I even tried Max 2016 demo for point clouds, and it didn't support even basic industry standard formats. No doubt you need some other Autodesk software to process them to work in Max. VERY lame. In my view Autodesk are now just out for money only, the software usability and quality has gone out of the window.



Really I want to learn and move away from Max and go to Blender and develop tools/plugins there for my needs... but every time I try Blender I pull my hair out with the bizzare UI.
It works, but it seems different just for the sake of difference to the 'usual' software interfaces, like left/right click select/properties logic.



But anyway, my script for importing this data. Still no joy with huge tiles all at once. 1/10th works ok. So just now writing code to create chunks and adjust each one a bit at a time.


A few small tests and the data looks nice. A DSM at 1m or lower will be very nice... just so resource heavy vs point cloud data.


Hopefully I'll have some pics in a bit where I can actually see what I'm looking at and gauge the quality

Right now I'm just seeing hills and streams and have no idea if the features look good or not.

Dave

 

[Pixelchaser]

very interesting there.

liquido showed me something the other day, forget what it actually was. but it was a way to load you clouds and classify all the data in them.(new software thing) so tree. buildings. surfaces etc etc. easily filterable. if I remember correctly it was an ArcGIS plugin.

 

[Mr Whippy]

Well I had a read up on the data release, so just to clarify a few points on this free data.

The release is in this order.

First is the *composite* data, which is basically datasets that use the best scan data, or most recent data, for a given area. The datasets are blended together using some interpolation methods, so ultimate accuracy may not be assured if you're working over blended parts.

Next is the *tiles* data, which is basically blocks of scans, so these are dated datasets without blending and various dates may be available. Probably best to grab these for a circuit build project if there are overlaps on the scans since with the composite data you might get blends and very slight discrepancies (or even buildings disappearing over blends if new ones are present in one scan vs the other)

Finally it seems like actual *point cloud* data will be made available last. Given the timings since this data has been made available, maybe this is already available for some areas, but I've not found any yet... still looking.




As per my file loading script, still working on it. I think this is more a scope of maxscript power more than my coding skills. Maxscript dealing with arrays with millions of entries on huge geometry seems to just bog down the process.
Ie, dealing with a 40k vert plane with 40k of height points and it's done in 6 seconds. Make that 80k and it's more like 20 seconds.
Even doing 40k vert chunks on a 1,000,000 vert object and it's taking about 30 seconds.

So current plan is to just break it up into many smaller jobs then recombine.

Eventually I want it fast so I can just churn out prepped meshes for people if they want them, within a matter of minutes.




As per the data, well this is a 1m DSM vs reality in a rural area.




I think you could happily get the top-down data nice from satellite imagery, but then jiggle it all into Z alignment really nicely with this level of reference data. You can almost see the crown of the road here in the DSM data.

The 2m data for rural areas is still quite nice, just softer looking. Probably better to just use the best data you can for the actual roads/tracks... but 2m data is probably ample once you leave the barriers or about 50m away from where you will be driving.

I'm now checking some urban 50cm data which I guess will be pretty damn nice. The 25cm data will be super heavy on resources but I bet you'll be able to pick out some great details, and get race tracks in 25cm areas REALLY nice and accurate.

It'll be interesting to see some point cloud data once I find some!



Cheers

Dave

 

[Mr Whippy]

Yipee, a few hours and I've got the script working nicely.

Just a few tidying up tasks to do and it'll churn through a 1m DSM in about 60 seconds.

I've not tried a 50cm DSM yet...

Fingers crossed haha.



So if anyone needs a DSM/DTM in the grid formats turning into a mesh let me know and I'll send it over!

Dave

 

[Pixelchaser]

interesting stuff there. may I request an area. its not a track, its a road project. and what size of area may I request ? 5km x 5k or is that to large?

 

[liquido]

I'm also working on CloudCompare with the elevation data provide by nibbles.

in fact I merge all the files into 3 big files with LAStools wich is a free software, then go to cloud compare and start to work with the files... my 8gb memory ram has problem with the full pack ( 5 Gb for 2km of road) so I discard a lot of data from the scene to be able to work with.





thanks to DanTDBV and his toturial I was able to delete every point if there is no a surface elevation data and get a mesh from that data... a 2gb mesh.



when I try to simplify the point could data with the subsample tool before make the mesh, (I try almost every option abaliable in the software) I can't achieve something usefull.



Today I will try to simplify the mesh again, but maybe I got the same resuts.

 

[Mr Whippy]

Hi Pixelchaser,

I'm happy to do a 5km x 5km area for you if I get to beta test the track... if it ever gets that far

If you show me a map with the rough area I can get to work on that for you.


I've been working on a road project since about 2003, and the data available and datasets I've collected/created over the years have meant the old stuff is often superseded.
But it's been a great test project for learning and stuff... this new data might help me actually get something really nailed down which is really exciting.



I'd also been testing with photogrammetry capture techniques this summer, and generating point clouds:

All I really need to do is capture from a moving vehicle rather than walking the route with a long pole and remote trigger haha. Some more testing required but as you can see by the end profile in the video, the points form a quality road surface for the major detail (camber/crown), and the rest is great reference for modelling.

Having used some high quality mobile road scans in the past for a commerical project, it was nice to see how close the photogrammetry got. Not close enough for engineering purposes I don't think, but amply good enough for conveying the character of roads!


Cheers

Dave