FS9 VTP2 roads explained...possibly

[Guest george_tirebiter]

For those advanced scenery designers... Last week I was looking into the feasibility of using Tiger/line data to improve the horrible default roads and rivers (and now railroads) in FS2002/04. A quick peek inside the scenery files revealed that they didn

[Guest GerrishGray]

Hi George / Lee (?)Great work exercising your hexeditor and programming skill on deciphering some of the mysteries of FS file formats, data structures, etc. VTP roads are not my 'bag', but I'm sure that Christian, Dick and the other workers in that field will find your analysis invaluable.From a purely academic view, I find it interesting that MS have used this sort of data structure to squeeze two SINT11 (!!!) values into a 24-bit data structure (technically, perhaps you should call this a STRUCT24 rather than a UINT24, or even follow the example of MSFS SDK's and just give it a descriptive name such as XYOFF24 :-) Whoops!I have just spotted that I now run the risk of someone suggesting that I should XY-off !!!). It's the kind of data structure that reminds me of the days 35 years ago when I was an assembler programmer on an 18-bit mini (yes, I did say 18-bit - there was sense to 18-bits as a word size in those days) with a truly massive :-lol 8192-word memory space (you could page to further 8192-word memory blocks if you had a machine that powerful - the original Concorde simulator used machines with three such pages if my own creaking memory is correct). We had to construct all sorts of weird and wonderful structures to pack data into the smallest possible number of words, and write code that relied on loads of shifting and masking to pack/decipher it.In this day and age I wouldn't have expected MS to pack data like this. But then again, the process of fetching data files from disk must still be one of the major speed bottlenecks in FS, just as reading it from mag tape was 35 years ago, so perhaps 'packed' data still has its attractions. Maybe there are lots of places in all those other as yet undeciphered parts of FS that also use packed structures ...?Keep up the good work!CheersGerrish

[rhumbaflappy 113]

Hi Lee.This is very close!I can confirm the use of the "Zone" it's LOD8's U,V position in an LOD3 Zone. I think we can acept Zone as it's term.Initially, I can build a single structure with the resereved bits set as 7. The "parent" is a regular VTP2 point... but I haven't yet progressed to verify the 24bit U,V of the 6 children ( I'll get to it eventually ).A problem I have is with the continuation of the line. If it's 13 points, we should have the VTPWidePoint ( with the reserved as 7 ), The width, and then 6 24bit children; then a new VTPWidePoint, no width, and 5 24bit children. But that gives me garbage... the garbage seems spawned by the second VTPWidePoint.:(The three-byte, packed, offset may give some problems in a macro... but we'll see. ( It should be a signed integer for U and V, as the children are relative to the parent in each segment ).Dick

[Guest george_tirebiter]

Hi Dick,>I think we can acept Zone as it's term.It's a little strange to see something I wrote in a sleep-deprived haze become standard terminology ;)>A problem I have is with the continuation of the line. If>it's 13 points, we should have the VTPWidePoint ( with >the reserved as 7 ), The width, and then 6 24bit >children; then a new VTPWidePoint, no width, and 5 24bit>children. But that gives me garbage... the garbage seems >spawned by the second VTPWidePoint.I don't think I was clear enough about the use of the reserved bits. They should be set to indicate the number of children following the VTPWidePoint, not as an on/off flag. So in your example above, they should be set to 6 for the first VTPWidePoint and 5 for the second.>The three-byte, packed, offset may give some problems in >a macro... but we'll see.Could this be why MS left it out of the SDK?I forgot to mention that TMFViewer shows VTPWidePoints as green rectangles and child points as blue rectangles, with a maroon line connecting the points in order. Without this ability to "see" the data, I never would have figured it out.Lee B.

[Guest george_tirebiter]

Hi GerrishThanks for the encouragement. My 'real' name is Lee. George LeRoy Tirebiter was a car-chasing dog at USC who later became an ice-cream-truck-chasing fictional character (I'm sure a few people reading this will know what I'm talking about).It does seem a little strange that MS would introduce packed structures for scenery at the same time that they were changing to floating-point numbers for aircraft and building models, but it might depend on the (in)efficiencies of the FS graphics model. I wasn't sure what to call the 24-bit structure, as I've only bothered to learn enough 'C' to hack the macros used to make gauges for FS.As for your assembly-language experience, you have my sympathies :-)My first programming experience was writing assembly language for the Apple IIe. Not the sort of thing most people do as a hobby, but it does pass the time. It's great that computers have advanced beyond the point where you have to worry about every byte.ThanksLee B.

[rhumbaflappy 113]

Hi Lee.Thanks for the clarification. Each segment may then have a maximum of 8 points ( parent plus 7 children ).; ----------------------------------------; Reproduced RD951250.bgl from FS2002; 17.072496 14.272385; N17* 4.3497' E14* 16.343'; derived initially from BGLAnalyze ©; ----------------------------------------; ----------------------------------------; This file contains terrain data of textured polygon type; ----------------------------------------; Since there are no instructions in SCASM to assemble; such data, the following code can only be assembled; with the BGLC assembler.; You will need the following files to do that:; TDFHeaders.inc; TDFMacros.inc; by Richard Ludowise; ----------------------------------------include TDFMacros.incinclude TDFHeaders.inc; ---------------------------------------- BGLHeader 18, 16, 015, 014, TerrainHeaderStart, VTPHeaderVTPHeader label wordVTPFileHeader 100h, VTPIndexStart, TextureStart, VTPEndVTPDataStart label WordBeginData0 label wordVTPDataArea 1, 1, 30, 15 VTPLayer 6, 0 VTPNumTexturesInLayer 1, 0 VTPTextureId 0, 0 VTPPolyCount 1, 0 VTPPolyMethod2 13, 1, 0; VTPWidePoint 8021, 1, 8050, 7 ; ------- maximum of 8 children to the parent point VTPWidePoint 8051, 1, 8050, 7 VTPWidePointWidth 35 BYTE 0F8h BYTE 0AFh BYTE 000h BYTE 0ECh BYTE 0CFh BYTE 0FDh BYTE 0CAh BYTE 02Eh BYTE 0E6h BYTE 0CAh BYTE 02Eh BYTE 0E6h BYTE 0EAh BYTE 00Fh BYTE 0FEh BYTE 0F4h BYTE 06Fh BYTE 0FFh BYTE 09Ah BYTE 06Dh BYTE 0DBh; VTPWidePoint 7372, 0, 7308, 4 VTPWidePoint 7402, 0, 7308, 4 BYTE 0FEh BYTE 0EFh BYTE 0FFh BYTE 0FEh BYTE 0CFh BYTE 0FFh BYTE 000h BYTE 080h BYTE 0FFh BYTE 0FAh BYTE 0CFh BYTE 000h; ----------------- 2 parent points plus 11 children equal 13 pointsEndData0 label wordVTPIndexStart label wordVTPIndexHeader 1, VTPIndexData, VTPDataStartVTPIndexData label wordCell_414_207 EQU VTPCellID 0, 414, 207VTPIndexEntry Cell_414_207, VTPDataStart, BeginData0, EndData0TextureStart label wordVTPTextureListHeader 1, TextureIndexStart, TextureDataStart, TextureDataEndTextureIndexStart label word VTPTextureListEntry TextureDataStart, BeginTexture0, EndTexture0TextureDataStart label word BeginTexture0 label word VTPTextureName "1166" VTPTextureType 3, 0, 0, 4 EndTexture0 label wordTextureDataEnd label wordVTPEnd label wordhttp://forums.avsim.net/user_files/40967.jpgIt Works! My road is on the right ( simply moved 30 points to the east ). It copies the default exactly. I did find the VTP2 border still trims off the line. I'll work out the 24bit byte packing tomorrow ( perhaps ). Road ends are squared off nicely. It works in FS2004 as well as FS2002.=============I noted the bit-pack is 11 bytes as you indicate, but I suspect the #12 byte signifies plus or minus for a range of -2048 to 2047 (?)Dick

[rhumbaflappy 113]

Hi Lee.I just discovered an important clue.The VTPWidePoint provides the line width, and the default scaling factor for the rest of the points. The points are X,Y spread over 3 bytes, as you wrote. With 35 as a width, the X and Y offsets from the parent are going to be small numbers. Negative numbers are allowed.I suspect there is a scaling factor(s) hidden in the three BYTEs that allow some fine tuning of the offsets... but I can place and draw simple lines with some control. I just need to find the limits of the offsets... whether they are 11 bits, or something smaller. With a scaling adjustment, they could be much smaller numbers.===============Other good news. the use of your Zone data ( The LOD8 position within an LOD3 grid ) makes all our homemade polys and lines visible in TMFviewer!It may also help with problems within the sim, although I haven't noticed any yet.The Zone discovery alone is huge. I hope to have a new TDFCalc2004 version out that will give the Zone data. For the curious, my Delphi Code for the Zone is: LOD3x := Ceil( ( Longitude + 180 ) / 15.0000 ) - 1 ; LOD3y := Ceil( ( 180 - ( Latitude + 90 ) ) / 11.2500 ) - 1 ; LOD8UVx := Ceil( ( Longitude + 180 ) / 0.46875 ) - 1 - ( LOD3x * 32 ); LOD8UVy := Ceil( ( 180 - ( Latitude + 90 ) ) / 0.3515625 ) - 1 - ( LOD3y * 32 );The Zone is : LOD8UVx, LOD8UVyDick

[rhumbaflappy 113]

Hi Lee.For now, it appears this macro will work for BGLC:; TDFMacrosFS9.inc ; -------------------------------------------------------------------VTPChildPoint Macro CHILDX, CHILDY WORD ( CHILDX * 10h ) + ( ( CHILDY / 100h ) ) BYTE ( CHILDY MOD 100h )EndM; -------------------------------------------------------------------You can build a bgl in this manner:[/font color=purple]; ----------------------------------------include TDFMacros.incinclude TDFHeaders.incinclude TDFMacrosFS9.inc ; note the new macro file; ---------------------------------------- BGLHeader 43, 42, -88, -89, TerrainHeaderStart, VTPHeaderVTPHeader label wordVTPFileHeader 100h, VTPIndexStart, TextureStart, VTPEndVTPDataStart label WordBeginData0 label wordVTPDataArea 1, 1, 3, 6 VTPLayer 6, 0 VTPNumTexturesInLayer 1, 0 VTPTextureId 0, 0 VTPPolyCount 1, 0 VTPPolyMethod2 7, 1, 0 VTPWidePoint 4260, 1, 10939, 6 VTPWidePointWidth 35 VTPChildPoint 1, 0 VTPChildPoint -7, 4 VTPChildPoint 0, 4 VTPChildPoint 7, 4 VTPChildPoint 0, 1 VTPChildPoint 1, 0EndData0 label wordVTPIndexStart label wordVTPIndexHeader 1, VTPIndexData, VTPDataStartVTPIndexData label wordCell_195_134 EQU VTPCellID 0, 195, 134VTPIndexEntry Cell_195_134, VTPDataStart, BeginData0, EndData0TextureStart label wordVTPTextureListHeader 1, TextureIndexStart, TextureDataStart, TextureDataEndTextureIndexStart label word VTPTextureListEntry TextureDataStart, BeginTexture0, EndTexture0TextureDataStart label word BeginTexture0 label word VTPTextureName "1166" VTPTextureType 3, 0, 0, 4 EndTexture0 label wordTextureDataEnd label wordVTPEnd label word[/font]In the VTPDataArea structure, you can derive the Zone numbers from the cell numbers by :( 195 MOD 32 ) = 3( 134 MOD 32 ) = 635 is the width and the offset scale factor.http://forums.avsim.net/user_files/41232.jpghttp://forums.avsim.net/user_files/41233.jpgIt should be possible to mix our old VTP2 lines in with this mix, as they simply have 0 children. That would allow the newer style for the line ends, and the older style for more accurate placement of the body of the line... probably MS's intention with this structure.A new version of TDFCalc2004 is available which includes the zone numbers for the VTPDataArea:http://library.avsim.net/esearch.php?DLID=...&CatID=fs2004sdDick

[Guest george_tirebiter]

Hi Dick,Fantastic work! I didn't expect to have the functionality of this info verified so soon, not to mention a new version of TDFCalc to boot! The macro will be of great help; I made a small test road this afternoon and hand-coding the child points too much of a chore for me to want to try it again.Unless or until we can figure out exactly how the scaling factor affects the child points, using them only for endpoints is not a bad idea. As far as I know, tapered ends are the only problem with the old VTP2 roads (now that the Zone info makes them visible in TMFViewer). Saving one byte of file space per point just doesn't seem to be worth the trouble of building the body of the road with child points.Lee B.

[rhumbaflappy 113]

Hi Lee.I'm glad you got it working... and I agree it's too much work and actually less accurate than the way we have been making roads. But it may be a solution for squaring the ends.Time permitting, I'll play with a hybrid line soon... square ends with the children, and the body of the line without children.I still suspect there is more to those three bytes than just offset location.Dick

[rhumbaflappy 113]

Hi Lee.Yes, we can use hybrid lines... mixing the old with the new:http://forums.avsim.net/user_files/41423.jpghttp://forums.avsim.net/user_files/41424.jpg;--------------------------------------------------include TDFMacros.incinclude TDFHeaders.incinclude TDFMacrosFS9.incBGLHeader 1, -1, 1, -1, TerrainHeaderStart, VTPHeaderVTPHeader label word VTPFileHeader 256, VTPIndexStart, TextureStart, VTPEndVTPStart label word datamark_v0 label word VTPDataArea 1, 1, 0, 0 VTPLayer 8, 0 VTPNumTexturesInLayer 1, 0 VTPTextureId 0, 0 VTPPolyCount 1, 0;;; VTPPolyMethod2 31, 1, 0 VTPPolyMethod2Ex 2 VTPWidePoint 4491, 1, 4484, 1 VTPWidePointWidth 35 VTPChildPoint 0, 1 VTPWidePoint 4537,1, 4509, 0 VTPWidePointWidth 35 VTPWidePoint 4540, 0, 4534, 0 VTPWidePoint 4524, 0, 4557, 0 VTPWidePoint 4512, 0, 4570, 0 VTPWidePoint 4501, 0, 4605, 0 VTPWidePoint 4504, 0, 4624, 0 VTPWidePoint 4526, 0, 4633, 0 VTPWidePoint 4565, 0, 4654, 0 VTPWidePoint 4600, 0, 4690, 0 VTPWidePoint 4608, 0, 4761, 0 VTPWidePoint 4583, 0, 4828, 0 VTPWidePoint 4539, 0, 4830, 0 VTPWidePoint 4518, 0, 4744, 0 VTPWidePoint 4532, 0, 4679, 0 VTPWidePoint 4573, 0, 4639, 0 VTPWidePoint 4624, 0, 4656, 0 VTPWidePoint 4663, 0, 4757, 0 VTPWidePoint 4639, 0, 4841, 0 VTPWidePoint 4598, 0, 4883, 0 VTPWidePoint 4572, 0, 4925, 0 VTPWidePoint 4509, 0, 4948, 0 VTPWidePoint 4472, 0, 4900, 0 VTPWidePoint 4445, 0, 4803, 0 VTPWidePoint 4434, 0, 4717, 0 VTPWidePoint 4411, 0, 4650, 0 VTPWidePoint 4400, 0, 4635, 0 VTPWidePoint 4345, 0, 4669, 0 VTPWidePoint 4352, 1, 4730, 0 VTPWidePointWidth 35 VTPWidePoint 4345, 1, 4730, 2 VTPWidePointWidth 17 VTPChildPoint 0, 0 VTPChildPoint 1, 0 datamark_v1 label word;----------------------------------------------Cellv_384_256 EQU VTPCellID 0, 384, 256;---------------------------------------------- VTPIndexStart label word VTPIndexHeader 1, VTPIndexData, VTPStart VTPIndexData label word VTPIndexEntry Cellv_384_256, VTPStart, datamark_v0, datamark_v1;-----------------------------TextureStart label word VTPTextureListHeader 1, TextureIndexStart, TextureDataStart, TextureDataEndTextureIndexStart label word VTPTextureListEntry TextureDataStart, texturemark_0, texturemark_1TextureDataStart label word texturemark_0 label word VTPTextureName "1159" VTPTextureType 0, 0, 0, 0 texturemark_1 label wordTextureDataEnd label word; ------------------------------VTPEnd label wordThis is a hacked Ground2K VTP2 road that is using the extended point format, squared ends, and the more familiar body.This will need a bit of work until we get it down pat, but the idea is sound... and it works.Dick

[Guest christian]

This is great. Thanks, Lee, I never had the stamina to sit this through and figure it out. I'm just back from a work trip and am really tired. I'll have a look over it tommorow or so.I just had a second look. I can't believe how close I got. I basically figured this format out a year ago or so. The missing piece was that regular points would sit in between. Before the SDK come out I had figured out that the sceme must be UINT24, but because they were those other points in between, it just didn't make sense... The second bit (where the SDK says use 0) I also figured out quite early, but never worked out what it's used for...Cheers, Christian

[Guest JR Morgan]

You guys are AMAZING... 'Sounds like this is going to be a "Key" breakthrough... Thanks for your work..J.R.

[Guest christian]

Ok, I've done some testing. Unless I'm doing something wrong, the algorithm isn't right! I've tried to 'read' the HL900210.bgl in the 'ocen' directory. It doesn't work. It looks to me as if there is an extra byte sometimes in between the client points. The line format is really nasty, I can't get any sense into it (yet again). Maybe someone else has a smart idea, Lee, Dick?I had a second look and tried RD900370, which actually works fine. So, maybe this scheme gets used for road lines. Hydro lines must have something special about them though. Since both use the same format we are still missing something...Some more findings:I think the bit pattern is actually not 1:11:1:11, but rather 8:4:8:4. This is just a guess from the little data I have seen, but could be worthwhile following up. Also it is possible that bit 21 is always 0 and if bit 9 is one, then the extra byte follows (maybe this is a width identifier?). So the bit pattern could be 8:1:3:8:1:3. The remaining question would be, what are the 3 bits controlling? Cheers, Christian

[Guest christian]

And more testing....The 8:4:8:4 pattern doesn't seem to be correct. Christian