Microsoft Flight Simulator and Prepar3D. Converging, Diverging or

[sjt 49]

As a change from the myriad of purely speculative posts that have been published here since the announcement of MS Flight Simulator, I thought I’d try to address something a little more serious. Sadly my comments will doubtless lead to some more speculation; I might even speculate a tad myself. Let’s see how this goes…..

One of the subjects that has not attracted too much comment yet has been the subject of SimConnect. To all professional developers and me (an amateur developer) this is an incredibly important topic, so here are some of my thoughts.

For those who are not aware of what SimConnect is I’ll explain simplistically.:

SimConnect is a low-level client server application that enables messages to be sent between a pc- based application (the client) and the flight simulator application (the server).  Typical servers are FSX, FSX Steam and P3D – now Flight Simulator they tell us.  When I describe it as “low level” it means that the average person on the street can’t do much with it unless they know how to write programs in C++, C# or VB.NET typically.

SimConnect should not be confused with FSUIPC. I’m no expert on exactly how Pete Dowson has written FSUIPC and will invite wrath if I get it wrong, but I can reasonably accurately describe it as essentially a higher level shell that surrounds SimConnect. FSUIPC does all the hard work of connecting SimConnect and communication with the server, and exposes the methods and results in a way that does not need you to know how to write programs.

SimConnect uses a fundamental principle of object oriented programming (OOP) called inheritance, or derivation. Every time a new version of SimConnect is released, the new version inherits all of the characteristics of the previous versions. So any new characteristics (properties, methods etc) just get added to what was in the previous version, meaning what’s already known does not need to be rewritten.

The server is the controlling influence. Because of inheritance, any client application that uses an old version of SimConnect  can be used to communicate with a newer server that inherited from it, but it doesn’t work the other way round. If the latest version of SimConnect attempts to communicate with a server written for an earlier version, they will not understand each other

 

 

The last version of SimConnect that was written for FSX was 10.0.62615.0 (I believe). There might be a latter version than that. If there is, I’m not aware of it and in the context of this post, it doesn’t matter anyway.

All subsequent versions of SimConnect have inherited from that version. That includes ALL the versions of SimConnect written for P3D.

So – if you want to communicate with FSX, FSX (Steam) or any Prepar3D vs 1 thru 4.5…… you can use FSX SimConnect version 1.0.62615.0 without any problem whatsoever. You will be limited to using the capabilities of the FSX version of course. So anything new that has been added to the later versions in Prepar3D will not be available to you if you want to use the FSX SimConnect. If you want the latest bells and whistles, you have to use the latest SimConnect. But if you do that – inheritance doesn’t work backwards…… The latest SimConnect for P3D v4 can only communicate with v4! It can’t communicate lower down the hierarchy.

 

 

So where is this leading.......?

Well……. The clear statement has been written that Microsoft Flight Simulator will support SimConnect. Quote “ For those with a preference for using FSX tools, SimConnect will be supported”. Excellent news indeed…… The only thing missing for me is “...OK,,,,but……starting from what version of SimConnect?” . That’s incredibly important to know. If the answer is “10.0.62615.0” it means that every client application written purely for FSX using SimConnect has a chance of communicating with Flight Simulator right out of the box. Gosh, that’s such a huge thing!  But…………but…….. anything written with a LockheedMartin. Simconnect.dll will not work – even though the LM SimConnect versions inherited originally from the FSX version. That’s because SimConnect does not work back down the ancestry – only up the hierarchy.

Most of my own client apps have only been written using the FSX SimConnect because the new versions that LM added to did not give me anything special I desperately needed. So my apps can communicate with FSX, or any version of P3D without any adjustment. In theory they should also work with Flight Simulator too, assuming the SimConnect that the Asobo/Microsoft team says it supports, is inherited from the FSX one (I think that’s pretty much guaranteed)

But wait……. (…….here comes some speculation….this is AVSIM after all……..)……..

What happens if the SimConnect version that Flight Simulator starts with is not the FSX version 10.something.or.other, but actually one of the LockheedMartin ones??? I mean…..surely it’s possible/likely/extremely likely/certain that LM and Microsoft (and Dovetail Games, maybe?) all sat down together and discussed the business benefits of not wasting all of LM’s good work for the past few years going to 64 bit, and adding all sorts of nice capabilities. Wouldn’t it just make a lot of business sense to join hands and go into the future with a single joint-venture that benefits both/all parties without treading on each others’ toes? What – in all honesty – is the point of 2 simulators being in the public domain that came from precisely that same root just 9 years ago, and have a high degree of interoperability and thousands of equally compatible 3^rd party add-ons that should work? What would be the point? Who stands to gain from competing with products that start from the same roots? Maybe those meetings didn’t happen (but should have, imho) or maybe they did and failed. Or maybe we just have to wait and see………

Quote from Prepar3D web site home page:“Prepar3D furthers the development of Microsoft® ESP™ while maintaining compatibility with Microsoft Flight Simulator X, allowing many thousands of add-ons to be used within Prepar3D.” Hmmm......

Have you noticed all the talk about Prepar3D version 5? I haven’t. Only from speculative simmers. Number of statements about it from Lockheed Martin sources = zero. Strange……Wouldn’t now be the time to be telling people (if it existed) in the face of the potential onslaught from Microsoft? Just look at how Intel and AMD get at it once the other announces something. It’s all about “first to market”. Silence is not a terribly good way to go about it, generally.

Now let’s look at the list of Flight Simulator partners. Note they are all aircraft manufacturers.  I see light aircraft, and commuters and we’re getting to a few liners. But wait……where are the jets and military aircraft? So far – zero. Doesn’t Lockheed Martin specialize in them? (Just wondering)

So back to business sense. Oh boy, wouldn’t it be stupid to announce right now if (big IF) that Lockheed Martin was a partner in the Flight Simulator project, because the number of sales of Prepar3D would dry up instantly. Not good business. So let’s hang on a bit and keep the revenue stream going…..until the time is right.

Just wondering, just speculating. Sorry……………..Just throwing out some food for thought. Anyone hungry? I hope the explanation about SimConnect was helpful, if nothing else.

 

 

 

Edited December 10, 2019 by sjt
Typos

[fs4fun 366]

Besides supporting whichever API version there's also the question of what variables and events will be there, and are they just emulating the old variable/event set for backward compatibility and devs will want to migrate to whatever the native interface will look like.

Also, simconnect looks like something designed in the 1990's. It's fine to have it available for backward compatibility but can we get something more modern, maybe gRPC?

[MattNischan 3,778]

Some background on what SimConnect is and is not. SimConnect is actually a binary wire protocol created to communicate asynchronously with the FSX/ESP platform. This protocol has been kept backwards compatible over the years as additional fields and message types get added via enhancements to ESP from various vendors. The original fields and message types are still handled. To my knowledge the protocol specification has not been released.

When the sim is started, by default IPv4, IPv6, and Windows pipe sockets are opened and await communication. This is the SimConnect server, hosted in the sim itself. By default this server listens only locally, but that can be configured. Then, programs using the SimConnect.dll client SDK can call various APIs to set values and control parts of the simulation. This is handled by the SimConnect library making a socket connection to the server and issuing various messages. These messages are queued, handled asynchronously, and then the results are handed back to the client through the socket in response messages (if necessary).

This architecture has a few good benefits. One is that you aren't calling directly into the sim itself via a linked library. This means that you don't need to link specific versions of the DLL for each simulator version. Due to the communication being done not via function calls into the sim but via standard socket communication, there isn't a hard coded dependency between clients and ESP. As long as each client speaks valid SimConnect packets, everyone is happy. And since the protocol has remained backwards compatible, lower versioned clients continue to work in later platforms.

I will add here that the ability for clients to work in this compatible way does not have anything to do with object oriented programming, classic inheritance, or anything else. Not trying to be pedantic, but instead educational, as I know these topics can get very confusing. In fact, if MS open sourced the wire protocol specification for SimConnect client/server, it would be easy to write forwards and backwards compatible client libraries in any language, even those lacking standard inheritance schemes. It's all just basic IP communication.

Heading to FSUIPC, FSUIPC started it's life out as something akin to Cheat Engine, if you've ever used that on PC. Basically, it modified directly the memory contents of the running simulator program. For example, the simulator could have a variable allocated in memory for the speed of the player aircraft in the X direction. Then, FSUIPC could control your X speed by directly modifying the value that is in that memory location of the simulator executable process. This is how FSUIPC did everything, up until SimConnect. Pete Dowson literally, through trial, error, and investigation, found the memory locations of the various variables the simulator was storing and wrote to them directly to control parts of the simulator.

However, today, FSUIPC does most of everything through the SimConnect library itself. The few things that MS or LM never added support for it still uses the old school memory offset writing technique for, but that's now the exception and not the rule. Pete is I think around 76 now and deserves a medal (or 10) for his work; however, for us to launch into another generation of simulation using memory poking would be unfortunate and silly.

My speculation after all that, and that's all any of us could offer at this stage, is that MSFS will launch with a SimConnect server that is backwards compatible with the FSX version, and maybe (but I wouldn't bet on it) some P3D extensions. There's really nothing in the base sim that would affect much in SimConnect; for example, LM moving the platform to 64bit doesn't change the wire protocol at all for SimConnect, so there's no sense that they would be losing out on generational type advancements. The only things would be SimConnect specific API changes and feature additions since the time of FSX.

I will also virtually guarantee that the protocol will be greatly expanded to fulfill the increased and improved needs of the next generation of addons. It would also be amazing if they released the SimConnect protocol openly, but I wouldn't say I'm holding my breath on that.

-Matt

[David Mills 4,737]

I appreciate the time and attention to detail that the OP employed in his explanation of SimConnect. I learned a lot and enjoyed his shared knowledge and informed speculation.

I seem to recall a few months ago that the always-well-informed Rob Ainscough offered a passing speculation that LM's next iteration of P3D might actually be a collaborative, militarized enhancement of MS2020 for military training purposes.

My own completely-uninformed opinion here is that P3D will go one of two opposite directions. Either LM will (1) forgo any future, major updates to P3D and so will essentially drop the project, or (2) they have indeed been collaborating with Microsoft/Asobo and will release P3D V5 simultaneously with MS2020. I've noticed that the LM P3D forum has not been particularly active recently, except for incessant bug reports about 4.5. I've seen several users complain that their support requests forever go unanswered and unresolved. Some of their most popular forum and support moderators have quit without explanation and without even saying "goodbye." I don't think Microsoft would spend all those millions of dollars on development of MS2020 only to allow LM to skim part of the profit. Microsoft earns its money primarily by selling software. LM earns its money primarily by selling hardware. So the new Flight Simulator is a much bigger deal to Microsoft than it is to LM.

My guess therefore is that LM will drop support for P3D. They seem to have done so already.

 

 

 

[RALF9636 1,911]

29 minutes ago, David Mills said:

I appreciate the time and attention to detail that the OP employed in his explanation of SimConnect. I learned a lot and enjoyed his shared knowledge and informed speculation.

I seem to recall a few months ago that the always-well-informed Rob Ainscough offered a passing speculation that LM's next iteration of P3D might actually be a collaborative, militarized enhancement of MS2020 for military training purposes.

My own completely-uninformed opinion here is that P3D will go one of two opposite directions. Either LM will (1) forgo any future, major updates to P3D and so will essentially drop the project, or (2) they have indeed been collaborating with Microsoft/Asobo and will release P3D V5 simultaneously with MS2020. I've noticed that the LM P3D forum has not been particularly active recently, except for incessant bug reports about 4.5. I've seen several users complain that their support requests forever go unanswered and unresolved. Some of their most popular forum and support moderators have quit without explanation and without even saying "goodbye." I don't think Microsoft would spend all those millions of dollars on development of MS2020 only to allow LM to skim part of the profit. Microsoft earns its money primarily by selling software. LM earns its money primarily by selling hardware. So the new Flight Simulator is a much bigger deal to Microsoft than it is to LM.

My guess therefore is that LM will drop support for P3D. They seem to have done so already.

 

 

 

I can very well imagine that as well, at least as far as a product is concerned that factually is available for the consumer market (because obviously LM do nothing to enforce their EULAs in that regard so far).

May main concern in this regard is the license agreement that MS and LM have made when MS provided the ESP engine to LM. I know nothing about that license agreement, but I would be surprised if MS did not include some competition clause in case they re-enter the market.

The LM EULA not allowing to use P3D for personal entertainment might be mirroring that competition clause. As long as MS wasn't really in the market nobody cared for the enforcement of the EULA but that would change as soon as MS enters the market again.

Just speculation of course.

 

Edited December 10, 2019 by RALF9636

[Theboot100 404]

2 hours ago, sjt said:

SimConnect is a low-level client server application that enables messages to be sent between a pc- based application (the client) and the flight simulator application (the server).  Typical servers are FSX, FSX Steam and P3D – now Flight Simulator they tell us.  When I describe it as “low level” it means that the average person on the street can’t do much with it unless they know how to write programs in C++, C# or VB.NET typically.

Not to be pedantic but there's a few misconceptions.  C# and VB.NET are not low level languages.  They are both high level languages using a Just In Time Compiler.

2 hours ago, sjt said:

SimConnect uses a fundamental principle of object oriented programming (OOP) called inheritance, or derivation. Every time a new version of SimConnect is released, the new version inherits all of the characteristics of the previous versions. So any new characteristics (properties, methods etc) just get added to what was in the previous version, meaning what’s already known does not need to be rewritten.

The server is the controlling influence. Because of inheritance, any client application that uses an old version of SimConnect  can be used to communicate with a newer server that inherited from it, but it doesn’t work the other way round. If the latest version of SimConnect attempts to communicate with a server written for an earlier version, they will not understand each other

 

The last version of SimConnect that was written for FSX was 10.0.62615.0 (I believe). There might be a latter version than that. If there is, I’m not aware of it and in the context of this post, it doesn’t matter anyway.

All subsequent versions of SimConnect have inherited from that version. That includes ALL the versions of SimConnect written for P3D.

So – if you want to communicate with FSX, FSX (Steam) or any Prepar3D vs 1 thru 4.5…… you can use FSX SimConnect version 1.0.62615.0 without any problem whatsoever. You will be limited to using the capabilities of the FSX version of course. So anything new that has been added to the later versions in Prepar3D will not be available to you if you want to use the FSX SimConnect. If you want the latest bells and whistles, you have to use the latest SimConnect. But if you do that – inheritance doesn’t work backwards…… The latest SimConnect for P3D v4 can only communicate with v4! It can’t communicate lower down the hierarchy.

OOP is a programming pattern or style.  Just like Functional Programming or even Procedural Programming is.  It has nothing to do with releasing different versions of Sim Connect.  

Being able to use different versions interacting with each other simply is a program design.  The earlier version simply ignores various variables it has been sent that it knows nothing about.

 

[MattNischan 3,778]

11 minutes ago, Rob_Ainscough said:

SimConnect is slow and can easily saturate data paths, it's still there primarily for compatibility and it is NOT "low-level".  If you want to work "low-level" with P3D then use the PDK  and C++ (this is different from the SDK) and PDK stands for "Prepar3D Development Kit" and it's NOT a part of the ESP or anything to do with Microsoft, it's LM's creation.  If you want to gain more access to the core and get things done really fast in P3D, use the PDK.

I just want to be clear on something regarding this point. The speed at which SimConnect can process has nothing to do at what layer it operates or what language of choice it operates in. In 2019 there are OSS socket server implementations in C++, C#, JS, and Python that all do 5M+ requests per second with mere megabytes of memory overhead on commodity machines. Even controlling tens of thousands of variables and events per frame (a huge amount for even a complex addon) would be hardly a blip.

The way forward is not more tight linking with direct library calls into the sim in native code. A proper, fast, low overhead socket server in MSFS powering SimConnect (or some other, more friendly protocol) would allow the most flexibility and language choice while maintaining functionally equal performance. I, for one, hope they go that way.

-Matt

[David Mills 4,737]

For those of you following this thread who may not be professional programmers, a "high level language" is generally not as efficient or fast as a "low level language." Low level languages, such as assembly/assembler/machine language, are much more difficult to learn and to code, but perform in a vastly superior way to "high level" languages. This is quite counter-intuitive. High-level languages are called such because they more closely approximate high-level human speech. For example, to print something in BASIC, you simply type PRINT "whatever." But the computer only really understands two things: ones (1s) and zeroes (0s). Everything else must be interpreted or compiled "downward." So the closer your original programming language comes to the computer's native "low-level" binary code, the faster your code executes. In other words, "low-level" is good. "High level" is inferior but easier to use.

[sjt 49]

32 minutes ago, Rob_Ainscough said:

you have a lot that is NOT correct here

Possibly - but such as?

1. Nothing I said contradicts your point 1

2. I never mentioned speed - just how SimConnect works in principle in laymens' terms for those who have no idea what SimmConnect is. I made no comment about its architecture or whether it should be retained or otherwise. I am not an advocate of SimConnect, but it was Microsoft who have said it will be supported, not me. If you read what I actually said about "low level" I was not using purist programming thinking on that - as you seem to be doing. I clearly explained what I meant by "low level". I said ": When I describe it as “low level” it means that the average person on the street can’t do much with it unless they know how to write programs". Sorry that wasn't clear enough.

3. Believe it or not I am aware of that. I was in aerospace for 30 years and climbed on and in more military aircraft than most. None were LM, as it happened. I fail to see what your comment has to do with anything. I merely pointed out the absence of jet/military manufacturers, and the coincidence that LM does manufacture jets and military aircraft and they aren't on the partner list.

4. I didn't suggest it has.

5. Yes. So? Again I never implied faster or better. I was just trying to explain (in simplistic terms) what is the significance of inheritance.

You come across as being very defensive of P3D and no doubt - to use your words - 

32 minutes ago, Rob_Ainscough said:

not permitted to comment beyond that

The idea of my post was not for the professional developers such as yourself who know vastly more about SimConnect than I admit to knowing merely to knock my statements down to show how clever they are, but to provoke  a few thoughts as to the direction things might be going. I don't care whether I'm right or wrong, but it seemed a bit more constructive than wondering whether Flight Simulator was mere vaporware - which has been quite popular and wasteful.

Edited December 10, 2019 by sjt

[sjt 49]

19 minutes ago, Theboot100 said:

OOP...…..has nothing to do with releasing different versions of Sim Connect.  

No. But inheritance does. That's all I was trying to say.

[MattNischan 3,778]

2 minutes ago, David Mills said:

In other words, "low-level" is good. "High level" is inferior but easier to use.

I would caution anyone against that old "wisdom". The performance difference achievable between languages traditionally labeled "low-level" and those labeled "high-level" can be vanishingly small to non-existent these days. Languages such as C# can operate with manual memory allocation and direct pointer access, despite being usually labeled "high-level". Node servers (which run JS) can be extremely quick to process events.

The language used to write a program is the very last thing that defines it's performance. Knowing how to write highly performant code in managed and/or dynamic languages is no more difficult or esoteric a skill than knowing how to avoid the pitfalls of memory management and cache coherency in unmanaged AOT compiled languages. If you can do one you can do the other. Everything else is religion and not programming.

-Matt

[MattNischan 3,778]

sjt,

I hope you didn't take my comments as knocking your knowledge. Certainly it was not my intention to show anyone up or be excessively clever. I like to teach if the opportunity arises!

It was a thought provoking topic and I appreciate the question and you opening it.

-Matt

[sjt 49]

3 minutes ago, MattNischan said:

hope you didn't take my comments as knocking your knowledge. Certainly it was not my intention to show anyone up or be excessively clever. I like to teach if the opportunity arises!

Gosh no! I thought your amplifying post was super, and I learned a lot from it. Thank you. My "being knocked" was not aimed at you, but at Rob who told me bluntly a lot of what I said was wrong, but then failed to point out anything that actually was.

Edited December 10, 2019 by sjt

[MattNischan 3,778]

1 minute ago, Rob_Ainscough said:

I don't see how a layer transport is going to be faster than a direct hook via DLL ... it's a bit more than just requests and given the nature of MSFS streaming data over that layer I can see limits being hit rather quickly.

Yeah, I would agree that absolute throughput would not necessarily be equal, and I certainly would not want to suggest that somehow a direct call would be inferior to handling and parsing a wire protocol message, but we're already talking about massive amounts of data on the peak end for either architecture. Those socket servers I referenced are saturating 10GbE links. I doubt any addons are moving that kind of data.

-Matt

[Theboot100 404]

15 minutes ago, sjt said:

No. But inheritance does. That's all I was trying to say.

No inheritance has nothing to do with different versions of sim connect. It's a programming pattern.  What is achieved in sim connect is possible with any programming style.

In a nutshell how do they make different versions compatible.  Simple, an older version receiving any 'messages' from a newer version that it knows nothing about, it ignores.

Adding functionality to a program with a later version is just that. Adding extra code to the same code base. Whether that code is done using OOP, functional programming procedural programming makes no difference at all.