Simconnect - interim rating :-)

[Guest Stoney3K]

That would also be a great opportunity for people like Jeroen Hoppenbrouwers and the PS1 community to jump in, since they have been using this system for years (over a TCP network). It IS the most efficient process to update a program's data, since it reduces the amount of polling and network/memory traffic significantly. If I use my application through FSUIPC on FS9 I get trouble-free operation on the same machine, but WideFS is restricting the traffic I can send to my remote machines making them hiccup a little. If (either local or remote) machines were to be simply notifed by FSX itself that some value changed, things would go a lot faster. Windows itself uses the same principle too, instead of every window polling where the mouse is every millisecond (the famous Shrek 2 line "Are we there yet??" applies here) a window is notified if some user clicks on it. With the 50fps we accept as a reasonable frame rate this is more than sufficient, and OpenGL applications can be built on the principle too (via GLUT).My current software is going to a stage where a generic event system will be used, which may be the best addition yet to connecting to SimConnect.On the side: Has anyone tried this on two SimConnects together (with two FS sessions) to do something like WidevieW?

[Pete Dowson 558]

>If I use my application through FSUIPC>on FS9 I get trouble-free operation on the same machine, but>WideFS is restricting the traffic I can send to my remote>machines making them hiccup a little. If (either local or>remote) machines were to be simply notifed by FSX itself that>some value changed, things would go a lot faster.Something is seriously askew with your assessment here. WideFS is, in fact, doing pretty much EXACTLY what SimConnect is doing now -- sending only previously requested data when it changes. It always has been working in that way. The polling is only between the application program and WideClient, certainly not between Wideclient and WideServer as you are assuming.Direct polling is a result of Adan Szofran's original IPC interface, inplemented in FS5IPC, FS6IPC and FSUIPC as well, but most certainly never between WideClient and its Server! The problem with any such client/server system, however, is latency, and we are already seeing this as a problem now with SimConnect, even on the same PC as FSX. I am already getting requests to bypass SimConnect and go directly to the data inside FSX's block-box modules again. I am of course resisting, but nevertheless I can see both sides of the coin here.RegardsPete Dowson

[Guest Stoney3K]

>Something is seriously askew with your assessment here. WideFS>is, in fact, doing pretty much EXACTLY what SimConnect is>doing now -- sending only previously requested data when it>changes. It always has been working in that way. The polling>is only between the application program and WideClient,>certainly not between Wideclient and WideServer as you are>assuming.Hi Pete and thanks for the quick reply,I may be using an old version of FSUIPC and the WideFS combo, and I know the problem is much worse when you use managed code (the .NET variant) than direct C/C++ communication with the FSUIPC library. I have been able to get very decent frame rates through a WideFS connection, however I know that at least older variations of WideFS allow for a larger amount of time between received data frames (I have read a doc saying 500ms should be between requests), but I don't know in how much it changed in newer versions of WideFS. I do know that Project Magenta is using heavy amounts of data interpolation (check the pmRJ EICAS for an example) instead of using the more accurate data directly.>Direct polling is a result of Adan Szofran's original IPC>interface, inplemented in FS5IPC, FS6IPC and FSUIPC as well,>but most certainly never between WideClient and its Server! Does WideClient implement a "Request/Response" mechanism or does WideServer actively monitor FSUIPC data constantly and send out a packet to all clients with the changes once they occur? It's the classic "push vs. pull" discussion that has actually baffled programmers for decades already (and became only more prudent after near-real-time applications like games came to the PC's)>The problem with any such client/server system, however, is>latency, and we are already seeing this as a problem now with>SimConnect, even on the same PC as FSX. I am already getting>requests to bypass SimConnect and go directly to the data>inside FSX's block-box modules again. I am of course>resisting, but nevertheless I can see both sides of the coin>here.I'm not sure where the latency comes from. Since most systems capable of running FSX have a decent amount of computing capacity and enough memory, that should not be as much of an issue (Moore's Law is fast enough here). But SimConnect may be relying on a socket or mapped memory file in the Windows kernel at some point for communicating with FS, and here is where it may go wrong, possibly causing underruns or overruns in the shifting of memory around. If SimConnect somehow interacts with the FSX main loop directly and gets its events there, there shouldn't be much trouble, but it is also possible that SimConnect gets a piece of (mapped) memory from FSX and has to monitor it for changes every cycle, causing latencies.I know that the people working with Aerowinx PS1 have built successful simulators (like Matt Sheil's full-mission 744 flight deck) based on a communication protocol that is as simple as plain text TCP messages. The trick they use is to split up some tasks of the simulator (mainly simulating and visual generation), offloading the load on the "server" machine providing the simulator data, but increasing the amount of network usage. Since most simulators these days are built upon 100MBit or even Gigabit networks, that will not quite form the bottleneck for the years to come. Visual generation on those sims is done through a 'slaved' session of FS, while Aerowinx Precision Simulator is used for the simulation logic. That is also exactly the way professional simulators do their thing and guarantee reliability. If one of the core simulation systems fails, another takes over since it has an upsynced copy of the internal simulator data. Most sim systems work in sets of three (like an autopilot system on a 747/767/777) and constantly correct each other for minor glitches in the FDM.I've been to the SIMONA sim in Delft where i study and there it's clearly visible: a set of six IBM SimFinity systems providing visuals and flight dynamics in groups of three. Since a flight dynamics system itself isn't very taxing on a system (at least not the FDM level used by consumer level sims) it wouldn't be unthinkable to make a threesome of older systems run the FDM and use a more elaborate system for the visuals. Keep in mind that even commercial simulators have graphics that are way beneath the level displayed in FSX -- the only upside is that they are way more smooth and visually accurate in terms of terrain, etc. but no telephone poles, trees, randomly generated buildings or reflective sea water.

[Pete Dowson 558]

>I may be using an old version of FSUIPC and the WideFS combo,>and I know the problem is much worse when you use managed code>(the .NET variant) than direct C/C++ communication with the>FSUIPC library.No idea about managed code. Sorry. It always seemed a dreadful idea to me, most inefficient.>... I know that at>least older variations of WideFS allow for a larger amount of>time between received data frames (I have read a doc saying>500ms should be between requests)No, you misread it then. There's no allowance needed except between the very FIRST request for data -- WideServer does not supply all data willy-nilly (just like SimConnect which does not, either). It has to be requested. Once requested, the data is supplied (Server to Client) whenever it changes, irrespective of how often the client application asks Wideclient for it. However, obviously the first time new data is requested by a particular client there are two transactions required -- a frame to the Server and a response back. That's usually still much less than 100 mSecs altogether, but WideFS always offered an initial delay of 500 to be sure. With most application programs that only occurs during their initialisation.All versions of WideFs, for the last 8 years or so, have operated on the Server/Client basis exactly the same as SimConnect.>I do know that>Project Magenta is using heavy amounts of data interpolation>(check the pmRJ EICAS for an example) instead of using the>more accurate data directly.Becasue it is trying to update the larger gauges at higher frame rates and smaller increments that can be supported by FS's frame and data update rates. Compare the jerky ratcheting FS panel gauges with the smooth PM gauges some time -- that's why Enrico does such things. It's a lesson gauge makers could take on board too.>Does WideClient implement a "Request/Response" mechanism or>does WideServer actively monitor FSUIPC data constantly and>send out a packet to all clients with the changes once they>occur?Er ... that's not fully a sensible question, put like that. WideClient operates an FSUIPC interface to the client applications. Its interface with the Server is exactly as I said -- it requests data the first time an application asks for it, and thereafter the Server keeps it updated without further requests being needed. WideServer maintains seperate data lists for each client, according to their needs, and watches for the changes. This is why WideFF is more efficient using a network via a switch -- it isn't sending the same data to every client.>I'm not sure where the latency comes from. Since most systems>capable of running FSX have a decent amount of computing>capacity and enough memory, that should not be as much of an>issueAsk folks who want to make smooth changes to an aircraft's position, for STOL and VTOL operations, for example. with FS2004 and FSUIPC's direct control over FS's values they can get virtually instant and smooth control. Via SimConnect and its asynchronicity the results are too jerky to be usable. There's too much delay between input and result. I'm hoping MS will be able to imporve it, but I'm not sure how much they can do.>But SimConnect may be>relying on a socket or mapped memory file in the Windows>kernel at some point for communicating with FSSimconnect is using Windows sockets, TCP protocol, in both directions. I thought that was well known by now.RegardsPete

[Guest Stoney3K]

>>I may be using an old version of FSUIPC and the WideFS>combo,>>and I know the problem is much worse when you use managed>code>>(the .NET variant) than direct C/C++ communication with the>>FSUIPC library.>>No idea about managed code. Sorry. It always seemed a dreadful>idea to me, most inefficient.Tell me about it, that sort of thing is never meant to run even at near real-time applications. I've used a port of the FSUIPC user library to .NET and can tell you it is dreadfully slow, which is to be expected.>>... I know that at>>least older variations of WideFS allow for a larger amount>of>>time between received data frames (I have read a doc saying>>500ms should be between requests)>>No, you misread it then. There's no allowance needed except>between the very FIRST request for data -- WideServer does not>supply all data willy-nilly (just like SimConnect which does>not, either). It has to be requested. Once requested, the data>is supplied (Server to Client) whenever it changes,>irrespective of how often the client application asks>Wideclient for it. However, obviously the first time new data>is requested by a particular client there are two transactions>required -- a frame to the Server and a response back. That's>usually still much less than 100 mSecs altogether, but WideFS>always offered an initial delay of 500 to be sure. With most>application programs that only occurs during their>initialisation.I did misread it then, and that may have also been the cause of my performance issues -- since in the first versions of my app I used more than one request to pull out several aspects of the aircraft's data out of FS -- in the latest version I'm using one request for everything. Isn't it possible to pull a whole chunk of memory (from the beginning of the FSUIPC data to the end) in one go by any way? That way I could process it in-memory later which may even be faster than using token variables.>All versions of WideFs, for the last 8 years or so, have>operated on the Server/Client basis exactly the same as>SimConnect.>>>I do know that>>Project Magenta is using heavy amounts of data interpolation>>(check the pmRJ EICAS for an example) instead of using the>>more accurate data directly.>>Becasue it is trying to update the larger gauges at higher>frame rates and smaller increments that can be supported by>FS's frame and data update rates. Compare the jerky ratcheting>FS panel gauges with the smooth PM gauges some time -- that's>why Enrico does such things. It's a lesson gauge makers could>take on board too.I'm doing bits of interpolation on my displays too, but they're small bits and I don't want to interpolate large changes as this results in a virtual "inertia" on your gauges (e.g. giving them a maximum rate of change). >>Er ... that's not fully a sensible question, put like that.>WideClient operates an FSUIPC interface to the client>applications. Its interface with the Server is exactly as I>said -- it requests data the first time an application asks>for it, and thereafter the Server keeps it updated without>further requests being needed. WideServer maintains seperate>data lists for each client, according to their needs, and>watches for the changes. This is why WideFF is more efficient>using a network via a switch -- it isn't sending the same data>to every client.Okay, so the server always keeps tabs on the data that is requested by some application or another, and sends out a packet when neccessary. I know WideClient is just a front end for applications that use FSUIPC natively, is it possible to connect to WideServer directly? (obviously for that you will need the data / protocol format, but I'm not sure if you're willing to share that)>Ask folks who want to make smooth changes to an aircraft's>position, for STOL and VTOL operations, for example. with>FS2004 and FSUIPC's direct control over FS's values they can>get virtually instant and smooth control. Via SimConnect and>its asynchronicity the results are too jerky to be usable.>There's too much delay between input and result. I'm hoping MS>will be able to imporve it, but I'm not sure how much they can>do.For the fastest works, use FS04 with FSUIPC on the same as FS is running on. Works for me all the time.>Simconnect is using Windows sockets, TCP protocol, in both>directions. I thought that was well known by now.If they use TCP for local connections also, then you have practically isolated the problem since Winsock is a really dirty piece of software. Especially when you use TCP to communicate data that is in a different part of the same machine's memory you're shooting a mosquito with an AA gun -- a very slow one, that is. Count to that the common programming mistakes first-time socket programmers make (e.g. writing or reading one byte at a time through TCP) and you're introducing seconds of latency along the way. Using sockets to communicate locally is not incorrect, however, since UNIX servers do it all the time and have no problems with it, but it's not exactly suited for real-time applications. For a real-time app you'll be wanting a simple UDP-based interface (something changed? Drop a UDP packet), since most UDP packets are translated into Ethernet packets almost directly (especially on a local network) and can be very smartly routed by the switch. You can do broadcast and multicast on UDP, something you cannot do on TCP, and UDP packets are message based (advantage, allows you to send/recv an entire struct in one go instead of waiting for a boundary character to come by), which reduces the amount of hassle on the network layer by orders of magnitude. Flight simulators like X-Plane and FlightGear have been using UDP packets since the start of their development, the only downside of UDP is the lack of reliability (that can be overcome by a simple parity/checksum mechanism).

[Pete Dowson 558]

>that may have also been the cause>of my performance issues -- since in the first versions of my>app I used more than one request to pull out several aspects>of the aircraft's data out of FS -- in the latest version I'm>using one request for everything.Once you've asked for any item once, any further requests do not cause any extra traffic between WideClient and the Server in any case, so this is only an initialisation delay.The inefficiency in using multiple "FSUIPC_Process" calls instead of one per cycle (whatever your cycle may be) in in the process switching in that PC between your process and Wideclient (or FS+FSUIPC in the FS PC). This is the same no matter where you run your application. When you use "FSUIPC_Read" you are only adding another request to a list in memory, which is dealt with locally by Wideclient or FSUIPC when you do the Process call only.>Isn't it possible to pull a>whole chunk of memory (from the beginning of the FSUIPC data>to the end) in one go by any way?Yes. Have you never tried to use FSInterrogate and done this there?But it is EXTREMELY inefficient. Within those 65536 bytes many different ones will be changing, and everything that is changing will have to be sent every time, even if you aren't really interested. It will slow FS down enormously as well as your local PC. It is ALWAYS much much more efficient to build up a minimum list of your needs by FSUIPC_Reads, then do the one Process call. On a Client PC the first time this may take hundreds of milliseconds, but after that it will be fast.>Okay, so the server always keeps tabs on the data that is>requested by some application or another, and sends out a>packet when neccessary. I know WideClient is just a front end>for applications that use FSUIPC natively, is it possible to>connect to WideServer directly? (obviously for that you will>need the data / protocol format, but I'm not sure if you're>willing to share that)Why would you want to? It's a *connected* link, so if you are connected no other application on the same PC will be able to. That is very limiting!I've no documentation of the WideFS packet formatting and protocols, they were evolved rather than designed, but I have heard of folks decoding them for implementation on, for example, MACs. I don't object to any of that, but I'm really not willing to spend the time documenting them or supporting other implementations. This is especially so now that SimConnect provides the function and I would expect, over time, folks to change to that.>If they use TCP for local connections also, then you have>practically isolated the problem since Winsock is a really>dirty piece of software. Especially when you use TCP to>communicate data that is in a different part of the same>machine's memory you're shooting a mosquito with an AA gun -->a very slow one, that is.Do you think I don't know this? I am thumping on MS's door all the time about this method of connection. I expressed my doubts way back, but they assured us that their performance checks showed otherwise. I'm not sure what they were measuring, but it really didn't turn out good enough -- yet. I am still hoping.>For a real-time app>you'll be wanting a simple UDP-based interface (something>changed? Drop a UDP packet)I use the UDP option for WideFS on most of my 10-PC cockpit Network, but you have to watch it. There's no checking, it isn't guaranteed to arrive, and packets can actually arrive out of order (especially, for instance, in a firewire looped network as I used once).It isn't too bad for simple things like gauge updating, because if you miss one the next will be along soon enough, and out of order packets self-correct next time too. But controlled switching and so on is much more critical.I check my WideFS logs regularly -- if any errors are reported (it logs missing frames and out-of-order sequencing) I have to find and fix the cause or change that PC back to TCP. I use TCP for my pmSystems and ActiveSky PCs in any case since almost all of their work involves writing TO FS, not reading back, and order is more critical and omissions a pain.RegardsPete

[DadJokeCinema 145]

>>No idea about managed code. Sorry. It always seemed a>dreadful>>idea to me, most inefficient.>>Tell me about it, that sort of thing is never meant to run>even at near real-time applications. I've used a port of the>FSUIPC user library to .NET and can tell you it is dreadfully>slow, which is to be expected.I'm playing catch up a bit on this conversation, and I apologize for being late to the party, but I think things like the new XNA framework and the older Managed DirectX kind of show that it is feasible, when using the built in garbage collection properly, to run real-time applications with managed code.

[Guest Stoney3K]

>>I'm playing catch up a bit on this conversation, and I>apologize for being late to the party, but I think things like>the new XNA framework and the older Managed DirectX kind of>show that it is feasible, when using the built in garbage>collection properly, to run real-time applications with>managed code.I really want to give XNA a try, maybe it can turn out into something really useful for developers. I know you use it, but have you got any working examples (binaries) that I can toy around with? Maybe it'll run even better on Vista.

[DadJokeCinema 145]

I've got a pretty sketchy half finished primary flight display. Right now it's just the artificial horizon, but it works.I had to create a WinForms object and hide it, just to get back to the system messaging, since the XNA framework doesn't support it or hides it really well.Other than that, I've been making a 2D game engine. I have some free time tonight, what sort of thing would you like to see get done with it?

[Guest Stoney3K]

>I have some free time tonight, what sort of thing would you>like to see get done with it?Just putting it online somewhere would be great, so I can have a test and all. My first experiments with glass cockpits were in GDI+ and Visual Studio .NET and I found it really easy to manipulate code or to do things quickly. I later switched to OpenGL for performance reasons.

[DadJokeCinema 145]

I took a look at it and I need to update it. It was written in the beta.I'll let you know as soon as I get it done.

[beatle 0]

<>Well, then look at that code I sent you a while back for doing XNA without their fullscreen framework :->I think there's a way to access a message only window, but that may have only been in the Beta, haven't looked for it in the final release version.

[DadJokeCinema 145]

><<>I had to create a WinForms object and hide it, just to get>back to the system messaging, since the XNA framework doesn't>support it or hides it really well.>>>>>Well, then look at that code I sent you a while back for doing>XNA without their fullscreen framework :->>>I think there's a way to access a message only window, but>that may have only been in the Beta, haven't looked for it in>the final release version.>>Actually, I am looking at it for my re-coding. Technically, "maximize" is close enough to fullscreen for those purposes, I think. It was just a question of which was more important to the process, if I wrap the whole thing in there just to gain access to the system messages or not.