TOOL LIDAR data - available for free for England

[Mr Whippy]

Liquido,

When I've tried meshing high density LiDAR data (scanned very near road like that data is), the meshes take ages and the usefulness is really not so great since the polygon density is so high.

On the project I worked on I tried this at first over 20km, and it was just impossible to use.

In the end this is why I used a point cloud software to reference point clouds directly in my building software.

Then if you wanted to make a mesh for the road, you can 'drape' the low poly mesh (say 50cm grid), onto the point clouds.



Of course you may get ok results eventually using a meshing process etc, but it's very time consuming and ram/polygon intense!

You're probably better using the LiDAR data from Environment Agency for everything, and then use just this very high quality data for the road surface itself... and set the meshing properties to something like 10,000 polygons per 100m say, or 100,000 polygons per 1km.

That should be ample for Assetto Corsa physics engine I think!

 

[luchian]

Guys, this discussion is really really interesting. I didn't had the chance to work with point clouds and lidar data, but even this short snippets of information from @Mr Whippy and you helps steer my curiosity into the right direction. I still have some things to sort out IRL (for another 1-2 months), but I can't wait to have a go at all this !

@Mr Whippy : I could use these formats into a software I am familiar with (Microstation); how would one obtain one of these from what's available ? (in a few words only).

In case it is already described above, I admit I can't follow each step closely until I don't get a chance to try it live.

Thanks guys and keep it coming .

 

[Pixelchaser]

man that vid is fantastic. you can definitely test what we make man !

area I need is say 4/5 km around this roundabout. and the major roads to and from this point. no interest in buildings really. sings and lamps would be good but mainly surface detail. reference positioning can be done with street view here too.
https://goo.gl/maps/51vcgFMLARp

Liquido has output a few great things tonight, im sure he show them soon. I am convinced its point cloud data that is only viable for usage at the level we need. everything is else is broadly useable too, just not for driveable surfaces.

 

[liquido]

Liquido,

When I've tried meshing high density LiDAR data (scanned very near road like that data is), the meshes take ages and the usefulness is really not so great since the polygon density is so high.

On the project I worked on I tried this at first over 20km, and it was just impossible to use.

In the end this is why I used a point cloud software to reference point clouds directly in my building software.

Then if you wanted to make a mesh for the road, you can 'drape' the low poly mesh (say 50cm grid), onto the point clouds.

Value information for me, thank you... you save a lot of time, It would be remiss not to have your opinion into account.

I need two hours to load the files into my computer and process all the data, in fact I need to delete data from clouds before join them into one big file.

Yesterday Pixselchaser and me were working on that point cloud and after many hours thinking about it, we finish on almost the same conclusion... no way to use that info directly into our proyects.

But we discober that you can make a heihgt maps with CloudCompare, at least you can extract some info from that file.... also has you said ... we can use the point cloud to place object with precission.

Finally we get this mesh from the point cloud provide by nibbles.

 

[Pixelchaser]

exciting to see the miniscule topography going on with a roundabut. the camber and flow becomes apparent.

firstly I think use this for references, then cut down the mesh to the roads only and lay low poly road on top, and use remaining data as invisible physics layer with much more reduction.

also I wonder if we have a way to generate a poly patch and repeat that within a low poly geometry....?

 

[Mr Whippy]

The problem with this kinda data is 'issues' when arbitrarily meshing it.

Ie, statistically, lots of points that make the mesh may not be road, they might be grass and stuff, so edges become less accurate.

Kerbs are a nice hard edge to work up to, but then you need to make sure you cut to the edges nicely too.


There is a LOT of time spent going around tidying meshes of this density for physics... and I'd question how useful or accurate they really are at such densities.

I'm not saying the AC engine isn't good, but is it THAT good to use this much data and give the results you expect? There is every chance your meshing is statistically bad based on the data you feed it (the point clouds, and how you've statistically chopped it/reduced it etc)... maybe you need to filter it to smooth it before meshing it?

So basically I'd be very cautious about throwing any mesh over about 25cm grid size over this data and expecting to get actual correct road topology from it that AC will use and feel correct upon.




Personally (again, no super duper expert, just based on my experience), I'd build a nice mesh that at sub-div level 1, matches the boundary of the road shape you want as the physics surface, AND the visual model.

You can then sub-divide it more to get more or less density later for the physics surface or the visual surface, while keeping the outline perfect and smooth.

You can then experiment easily with the physics mesh but simply draping it onto the very high density mesh (from Cloud Compare), and use varying systems to process that base mesh first (smoothing/sub-sampling etc)

That way you're not explicitly using the high density base mesh, because you can't explicitly rely upon it for getting nice results... BUT, you can increasingly sub-divide your physics mesh and get increasingly closer to that meshed reference surface if you want to.


I think it just gives more control that way, and it's nice to have a direct correlation between the physics mesh and the visual mesh boundaries etc.


This is just an interesting view from someone dealing with this road type scan data and making it feel right for the sim in question.
http://www.cruden.com/blog/road-modelling-for-the-automotive-industry/

I'd argue that it'd be really useful to understand the AC tyre model in some detail to understand how far you need to go with the physics mesh density, and what the limitations of the tyre model are.
There is no point for example modelling every grain of the road asphalt for example, if the tyre is modelled as a point and the physics won't really care if the grid is 0.2cm in size, or 20cm in size!

Dave

 

[Pixelchaser]

well we are about to test that to some degree, going to create a surface with as much as this as possible. for a test. we can look at cpu load in game etc. but I already know its complete overkill. and that if you can achieve a 10 cm mesh, that's perfect for simulating the changes on real roads that are there.

our physical mesh is never graphically drawn btw. always invisible to the graphics engine and your ingame eyes.

 

[Mr Whippy]

Yeah it's well worth testing.

But like I say, even a 10cm grid for physics, although nice and smooth and ideal, statistically you might get it more bumpy than it really is, or vice versa.

Which is where some kinda statistical method of conforming a controlled physics mesh over the 'meshed' point cloud is a better approach, so you can iterate and test lots of variables quickly and easily.
Again, been there and done that with meshing and testing... in the end a 25cm grid felt just as good as a 5cm grid because the noise of the point cloud just became the noise in the rougher edged physics mesh

Perhaps with some really well controlled simulation then you'd feel that extra data, but I really don't think AC is doing anything that smart here. It's a racing simulation first and foremost. It probably has a single point contact patch tyre model, and I'm pretty certain they don't simulate any bushings in suspensions or sub-frames etc.
Anything you 'feel' from a realistic physics mesh surface of a real rough-ish road, is probably just going to feel overkill because of the lack of these things.


Again, lots to test and experiment with I'm sure... just remember AC was designed for smooth race courses with race cars and rose jointed suspensions.
Road cars, with PAS, road suspension, road bushings, on rough surfaces, it's not designed for... so just be sure you don't do too much work getting great physics meshes that match real roads perfectly as AC cars might drive like crap on there

Dave

 

[Mr Whippy]

Damn it.

OK so I can do 2m and 1m grids easily here from the Environment Agency data.

50cm grids, hmmm, crashes Max here. Too many entries for the array.


At this point I'm thinking it'd be easier to just generate meshes externally in a little app in a decent format like fbx.

I can script my grid tool to load in chunks so it never deals with anything too big. Probably a better way to go generally now knowing Max is hitting a limit with 4mill array entries haha


But for now I could do the 50cm or 25cm grids by simply cutting up the ascii files into chunks manually.

I'm just doing more testing here, but I'll put up some samples of 2m, 1m, 50cm and 25cm for a certain area so people can have a see.


Also just FYI, the coverage map is gappy, so even if an area gets shaded, it might not have full coverage within that block.


Cheers

Dave

 

[Pixelchaser]

Yeah it's well worth testing.



Which is where some kinda statistical method of conforming a controlled physics mesh over the 'meshed' point cloud is a better approach, so you can iterate and test lots of variables quickly and easily.

Dave

this is exactly how anyone uses this info. referencing it onto your own game capable mesh. and directly we use a road wide portion directly with manipulation and or extra conforming to surface capability.

 

[Mr Whippy]

Yes it's the best approach if you can't reference the point cloud directly.

But in the end it's still all guess work I think.

The real key is knowing the limits of the physics engine and the limits of the data, and then you can tweak the physics mesh to suit your end needs for that track.

Like I say, for AC it's heavily setup for race cars on smooth race courses, so I'm not sure what kinda data it will or won't like for roads.

But that's no bad thing for you guys haha, it just means more driving AC and doing back to back testing in your favourite road cars

Also maybe going and driving on that roundabout in real life and trying to get a feel for what it's really like hehe

 

[Pixelchaser]

I believe real roads are very smooth and very safe. they are generically designed to provide a uniform basis for navigating and coordinating yourself with. I think this means they can cope with a lot less detail than any scanned surface. and given the speed we are ment to be driving them at. a real road system wouldn't require the full 100% of what a race track requires. but it depends how far you do want to take it. but I just think real roads "should" not be as intensive or complicated as a race track. a real road just needs lots of real roads to be convincing.

 

[Mr Whippy]

Hehe, I know what you mean.

It's really hard to figure out what is right or wrong really.



In practice a 25cm grid is probably ample for most purposes except specific engineering ones, or specific road details (pot holes)

But the risk is that if you sub-sample a high and low point from a point cloud every 25cm with +/- 5mm error in the point data scanning, on certain surfaces you might end up with +/- 10mm noise at each vert.
As a worst case scenario on an otherwise totally smooth surface that would give you a constant square/sine wave with 1:25 gradient up/down/up/down... and give you a bumpy feeling.
But in reality it might be super flat.

It's these subtle noises that you might not want. But then important details may also be hidden in that noise that you might want to capture.


Does AC engine interpolate across the physics mesh, or literally drive on triangles?

Dave

 

[Mr Whippy]

Anyway, here is the 1m DSM for that roundabout area on the A45.

I'll upload a ZIP to my FTP and send a linky to you shortly.

Generally this data is pretty nice. I'm a bit frustrated that the downloader coverage preview is so low quality though, since many blocks you download have pretty big gaps in them, at least away from cities and valleys.

There is a better preview tool here:
https://data.gov.uk/data/map-preview?url=http://www.geostore.com/OGC/OGCInterface?SERVICE=WMS&UID=UDATAGOV2011&PASSWORD=datagov2011&INTERFACE=ENVIRONMENT&LC=1000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000&n=55.816&w=-6.236&e=2.072&s=49.943

I've pre-selected the 1m DSM, but at least you can see the EXACT coverage of each data-set and then decide to download and import or not

But, I'm not sure it does the 'tiles' coverages, so you're back to square one checking those by importing to check coverage

Cheers

Dave

 

[Pixelchaser]

it is very nice.

thanks so much for this. ive had a look in max and it is superb. those rivers, the height deformation on the centre of the roundabout. the houses etc. its stunning man !

 

[Mr Whippy]

The values at Z=0m are 'no value', not actually 0m by the way

I'd have deleted them but thought you might find it useful to have values there for whatever reason.


Yes the data is impressive indeed.

50cm and you're getting road crown/cambers pretty nicely, and houses generally show up chimneys and roof pitch pretty nicely too.

I've not loaded any 25cm data yet but it's got to be pretty damn nice... but wow is it polygon heavy!


I think it's better in many smaller chunks, but the data comes in either same size dimensionally (so 4x more polygons for 50cm, or 16x for 25cm!)... and if you grab the 'tile' datasets, they're often 4x bigger again due to being 2km squares vs 1km squares.

Soooo, my script is getting re-tweaked/written again since I think quite a bit of useful data not on the composites is on the tiled sets, for my needs at least!


Cheers

Dave

 

[Mr Whippy]

Urgh, Maxscript, my nemesis

Strings to names for recursive array creation/(re?)naming. How is this so hard?

I'm obviously missing something obvious I've missed many times before but have since forgotten. Time to go dig in some of my existing scripts.


I think I might try an entirely different approach to this tomorrow, then I can at least load these huge grids. Probably better in small chunks too. Max seems to start to fall down with 4mill vert objects.

 

[liquido]

What a beautiful mesh Dave, looks awesome... I wonder if you know what is the zaxxon method.

Long time ago one guy (zaxxon) how is missing in the foums he used to roam, He wrote a pack of scripts with Octave to elevate custom meshes created with the software (mesh lab) and elevation data from ASC/DEM/LIDAR fromats.

In most of cases you can't use directly the mesh you got from elevation data, a smart solution is to create a mesh in 2d with the splines for the future roads inside the mesh like road paths, and conform your custom 2d mesh with the elevation data of the terrain. this way you can create a custom mesh, more dense near to the road and sections you will be able to drive and less polygons on the far scene... also you keep the splines inside the mesh like road paths so you will able to work on those splines after elevate the whole scene.

This guy go one step further and he develop a script wich rise/elevate the road with the elevation data and also with several parameters to define min/max inclination and relation betwen nodes of the spline that define the road.

also he uses two diferents elevation data references... one more dense elevation data for the road and near terrain, and other elevation data file with less points for the far terrain.

The LIDAR data is a marble... long time waitting for this moment, but I'm disapointed... I can't use that data couse the amount of information is still a problem for common user computer.

 

[Mr Whippy]

Hey Liquido,

Yeah that is almost how I work here on most tracks.

Well, every track is different based on the data you can get.

But if I have a DSM or DTM as a base reference then I just get the mesh, cut away all the un-required bits to save on system resources, and then conform a spline path to that surface to get the Z data into a spline.
I can then do a basic loft, then collapse and work on the road as a sub-d part from then on. Or variations on that theme.

Yes I've also written some scripts to tweak splines, so I can smooth them after conforming. Add more knots, or remove them etc.

Also yes I've used high and low density DSM for near/far terrain. With these high quality DSM, I can now bake normal maps into low res textures, to sit alongside the aerial imagery on the distant terrain, and give me really nice shading and lighting!


Really this is why I like stuff like Max for these jobs. There is no right or wrong process, just the one that works and gets you the results you need.
I've NEVER built a track the same way twice. Every track has unique variables and data available so you work to those needs.




Yes the LIDAR data grids are great. But they ARE heavy loading on the system.

Maxscript isn't really designed for such huge files, so even automating the import is a pain.

Ah well, it's a fun challenge to make it work haha


I think the best way I deal with these huge grids is to hold them in an Xref scene. Then ones I use/work on are just used/optimised and saved in the local file. Mainly just so the Max project files are not saving the same data many times over!

 

[Pixelchaser]

have you tried the Mario Morais plugin scripts for max at all ?