Character Encoding (XML Files)

[Cruachan 705]

1 hour ago, SteveW said:

Then you paste in some text that contains characters inconsistent with the present format.

Hi Steve,

I do recall using my usual double slash to comment out a couple of default lines in Prepar3D.cfg before adding those same lines containing modified numerical values. I then decided to use a semicolon instead of the slashes. Do either of these characters qualify as characters inconsistent with the native format?

Best regards,

Mike

[SteveW 1,792]

5 hours ago, Cruachan said:

Hi Steve,

I do recall using my usual double slash to comment out a couple of default lines in Prepar3D.cfg before adding those same lines containing modified numerical values. I then decided to use a semicolon instead of the slashes. Do either of these characters qualify as characters inconsistent with the native format?

Best regards,

Mike

Those are OK. But copy paste from Windows calculator maybe could do it. If you delete Prepar3D.cfg and let it create a new one, then use native Notepad.exe to edit the file you should be OK. I would never use Notepad++ to edit P3D files and do not recommend it.

Incidentally, using Notepad++, I think, mainly stems from not being able to edit files in the read only folders SimObjects, with aircraft.cfg and panel.cfg etc. When editing files in read only folders it's best to set the User Group modify allow permission on those folders. That is the proper way to do it rather than use run as admin or special tools. Only authenticated access is allowed with the User Group so that's the only safe way to do it.

Only edit with native Notepad and if you get the popup to change format, back out and type in from the keyboard don't paste. If you get the read only popup, set the folder User Group to Modify allow and don't edit as admin.

Edited July 29, 2020 by SteveW

[Cruachan 705]

21 hours ago, SteveW said:

Add-on xml files are Windows CRLF UTF-8

dll.xml exe.xml and Displays.xml files are Windows CRLF UTF-8 with BOM

cfg files like display.cfg in C:\ProgramData\Lockheed Martin\Prepar3D v5 are Windows CRLF UTF-16LE

Using Notepad++ is a cause of encoding errors in these files . Using native Windows Notepad.exe is preferred.

Just as the fog is slowly lifting to reveal greater comprehension I discover that I am actually managing to learn about some quite useful stuff!

For example, regarding the use of CR and LF:

https://en.m.wikipedia.org/wiki/Newline#History

It’s a long article but restricting reading to the first few sections should suffice as these are pertinent to this discussion.

Mike

Edited July 29, 2020 by Cruachan

[SteveW 1,792]

1 hour ago, Cruachan said:

regarding the use of CR and LF:

Hi Mike,

Correct. In the pascal language I use:

Cr = #13;
Lf = #10;
Crlf = Cr + Lf;

Regards

Steve
 

[SteveW 1,792]

...I once had a callout to a problem that was thought to be a Windows problem corrupting files. Very basically someone editing Windows files on a MAC (which incidentally I think they are going to ARM processors), and sending the result back to a Windows system. As you can see from reading the Wiki that might not work properly if line ends are altered.

[Lorby_SI 2,042]

19 hours ago, Cruachan said:

is it safe to change that encoding

No. It never is. I can only recommend to refrain from editing "technical" files (=files that normally are created and processed by software alone) manually. With P3D this shouldn't be necessary.

I have taken a look at the files that you sent me. They are encoded in plain UTF-8, most don't have a text declaration and they don't have a BOM. This results in P4AO not being able to determine the actual Encoding (that is not as easy as one might think). And that is the reason why it is reporting them - I should probably rephrase that message. If you would press "Save", then P4AO would add a text declaration to each file and use the encoding UTF-8 with BOM. Not because I want that to happen, but because the XML parsing and writing logic in P4AO is not my own. The program uses the System.Xml.Serialization assembly from Microsoft - which I presume works as per W3C XML specification.

Best regards

 

Edited July 29, 2020 by Lorby_SI

[RamonB 449]

Oliver, One question?
Even though BOM is allowed in UTF-8, It was my understanding (from reading unicode.org) that BOM was neither required nor recommended for UTF-8, unless some type of data conversion was done.

 

Edited July 29, 2020 by RamonB

[Lorby_SI 2,042]

21 minutes ago, RamonB said:

Oliver, One question?
Even though BOM is allowed in UTF-8, It was my understanding (from reading unicode.org) that BOM was neither required nor recommended for UTF-8, unless some type of data conversion was done.

 

That is true. But as I wrote, I don't have control over that. I am not going to gobble together my own code when there is a perfectly valid solution from Microsoft. The BOM isn't necessary, but it is not forbidden either. 

"Unicode" is about text files in general. The specification that applies to add-on.xml files is the XML spec from W3C (otherwise they aren't XML files at all)

https://www.w3.org/TR/xml/#charencoding

Quote

Each external parsed entity in an XML document may use a different encoding for its characters. All XML processors must be able to read entities in both the UTF-8 and UTF-16 encodings. The terms "UTF-8" and "UTF-16" in this specification do not apply to related character encodings, including but not limited to UTF-16BE, UTF-16LE, or CESU-8.

Entities encoded in UTF-16 must and entities encoded in UTF-8 may begin with the Byte Order Mark described by Annex H of [ISO/IEC 10646:2000], section 16.8 of [Unicode] (the ZERO WIDTH NO-BREAK SPACE character, #xFEFF). This is an encoding signature, not part of either the markup or the character data of the XML document. XML processors must be able to use this character to differentiate between UTF-8 and UTF-16 encoded documents.

 

Edited July 29, 2020 by Lorby_SI

[SteveW 1,792]

As I mentioned earlier the BOM might be required with xml files but presents a problem for standard Windows code in certain .ini files, that is, those files with section names between square brackets..

[Lorby_SI 2,042]

46 minutes ago, RamonB said:

that BOM was neither required nor recommended for UTF-8, unless some type of data conversion was done.

I don't really agree with that point of view. IMHO that is a matter of code safety. If there is no BOM, the encoding can only be "guessed". I prefer solutions where a program can determine the file Encoding without a doubt right from the start.

Best regards

 

Edited July 29, 2020 by Lorby_SI

[SteveW 1,792]

For an example; have a look at an FSX or P3D .PLN flight plan file in Notepad.exe and we can see that these are .xml files of the format Windows CRLF UTF-8 with BOM.

[Lorby_SI 2,042]

Hallo @ll,

my personal view about the Notepad/ ++ discussion is the following:

- As long as you are editing plain text files, like .cfg or .ini, that don't really have a technical specification backing them, you "only" have to make sure that you don't change / screw up the Encoding. That means being careful with copy/pasting text from other applications (especially your web browser!), and generally keeping a backup copy just in case. You can use any reasonably solid text editor for that (but with most of them you can't change the encoding). And naturally, you must adhere to the rules that the program that reads these files has set. For P3D that would require a thorough read of the Learning Center. If there is no spec, then you are on your own. Your guess is as good as anybody else's. 

- With XML files matters are not that simple. As the W3C spec shows, these are not just text files. Strictly speaking, Notepad.exe is unsuitable for editing them, because it is not a proper XML parser. "++" at least is aware of the XML spec. But still, you should never write/edit them manually, unless you are familiar with the W3C spec and know exactly what you are doing and the way to do it. Even in the old days there were countless examples of people or installers screwing up the dll.xml and exe.xml, and those were just the two files. Now we have dozens of xml files everywhere.

With P3D special care should also be taken not to edit the add-ons.cfg files, not by a user and neither by an installer. Doing this interferes with automated features of P3D. Instead, P3D has the autodiscovery and the CLI for adding content - there is no need to mess with the two config files.

I often hear people making the argument "the simulator reads it fine, so it can't be wrong". I can't agree to that either. The simulator was built to read basically anything, but it has its limits too (as demonstrated for example by the three bugs that were present in the add-on.xml processing, two of which prevented the sim from even starting). Errors in files cause error handling in the sim. With a file that is only read once, that is not a big deal. But a file that is read all the time, it is. For example a wonky gauge code that is executed every 50 milliseconds = 18 times every second the sim has to process this error. Or other content errors, like broken sound files, broken effects, screwy MDL or BGL files... etc. And yet people never cease to put all the blame for stutters and bad frame rates on the sim itself (and turn off the content error log).

Best regards

Edited July 29, 2020 by Lorby_SI