Sign in to follow this  
hesynergy

Tens of GB of Orbx texture duplicates found across product line

Recommended Posts

Hey y'all,

 

I thought would look around and started cleaning up my hard drives, especially my flight sim folders, versions of which span six drives. I found a huge number of what appeared to be duplicate files, and decided to concentrate on my current sim, PREPAR3d v2.5.

 

I decided to do graphic file search using a specialized program called DupeGuru....what I found is extremely puzzling..., where almost 50 GB of graphic texture files found on JUST my P3d v2.5 FOLDER... appear to be duplicated across the range of Orbix products that i own;see below.

 

FSS0222940 FTX GLOBAL ,

FSS0204119 EU Wales,

EU England,

NA Blue USA/Canada Pacific Northwest,

NZ North Island, NZ South Island,  

FSS0269217-Global VECTOR,

FSS0270854-NA Gold USA Northern California

 

Using Irfanview as my main viewer, which has been set up to launch the external editors, PSP, AND DXT2bmp, I analized some of the reported duplicate textures in DXT2bmp, and found a lot of those examined to be in fact duplicates...such as 027cfa1.BMP BMP(DXT1):1024x1024 16 bit , found in the England and Wales folders, is virtualy visually identical, and size and type wise, identical.

 

Starting with FTXGlobal, which if i recall correctly, wants you to backup your texture folders befor installing itself. A huge number of files are installed....all brandy new FTXG textures.

 

Add in the packages cited above, what appears to happen is one package doesn't appear to know about the other, and merrily installs itself and its textures....many of which are duplicates of previously installed  packages.

 

Therefore, each installed package compounds the problem....yielding for me close to 50 GB of spurious duplicates. DupeGuru has a "Picture Edition", so i am re-doing the scan... of 117,613 graphic files; it is only 15% through with the analysis...I will report back soon, hoping that the results will be significantly different....i would be pleased-as-punch, to be wrong.

 

Who is culpable here?  did i mess up and not follow the installation instructions?

 

or did Orbx kindly give me 50GB "spares"

 

I am happy to know if I messed up...if someone can tell me how to "UN-Mess it"

 

Am I not alone here?...with my little 50GB Orbx giftee.

 

Or...am I making a Mountain.BMP out of a molehill.BMP??

 

i can provide html, csv, or inate .DupeGuru report files for your inspection.

 

regards to all,

 

from a puzzled ...

 

Chas

Share this post


Link to post
Help AVSIM continue to serve you!
Please donate today!

Sounds like you are interpreting things wrongly. FTX copies textures to the sim, but stores them in the ORBX folder. In  addition you have the user-optional backup folder. These are not duplicates, they are primary install files, so they have to sit somewhere on the computer. By default that's under the main sim folder.

 

Obvious thing to do is `offsite` the ORBX folder, using a make link instruction, - we have covered this on numerous occasions in  this forum

 

Try this 

http://forum.avsim.net/topic/438234-installing-fsx-orbx-scenery-on-separate-drives/

 

In any event, apart from dedicated files called by Regions and the like, the total install size in the sim is driven by the size of the FTX Library download, not anything else. And they are not additional files, in most cases they are alternatives to what was installed before.

Share this post


Link to post

Mountain out of a molehill. Each region can call it's own version of a land classification texture, if it has it's own landclass.bgl file. So any texture can and should appear more than once, ideally each time a region is installed. Add in that FSX/P3D will have it's own World-based set of the same textures and you will add up many instances of the same file name.

Share this post


Link to post

Did it ever occur to you that maybe 033b2su4.bmp in the FTX_NA_NRM08_CUSTOM\texture folder might actually be different than the 033b2su4.bmp located in FTX_NA_NCA08_CUSTOM\texture? Maybe that's how FTX makes one region look different than another region - as in their real world counterparts?

Share this post


Link to post

Did it ever occur to you ...

No jim...with no snarkiness...never occurred to me...

 

"such as 027cfa1.BMP BMP(DXT1):1024x1024 16 bit , found in the England and Wales folders, is virtualy visually identical, and size and type wise, identical."

 

 

C

Share this post


Link to post

Some might be. Many won't.

FTX manipulates BOTH landclass AND textures with the textures mapped to work with the Regional landclass. That is why you have so much variation on a theme and why ANY attempt to de-duplicate will cause problems with non-neighbouring, lack of colour-matched or just missing textures.

 

What everyone here is trying to tell you, all snarkiness aside, is that if you don't know what you are doing, leave it alone.

ALL textures are NECESSARY

 

 

If you want to know more, ask the developers in their own forum, as they don't answer technical questions here.

Share this post


Link to post

such as 027cfa1.BMP BMP(DXT1):1024x1024 16 bit , found in the England and Wales folders, is virtualy visually identical, and size and type wise, identical

 

Nobody said there won't be duplicates, I believe if you were tasked with creating 3000+ custom LC textures for a region you'd probably be looking for ways to reuse anything you could as well, no? The real question though is how do you make a landclass .bgl in England\custom\scenery call a texture from Wales\custom\texture so as not to have duplicates? You can't, and even if it were possible, what about folks that own Wales but not England? The Wales people are going to need need a complete texture set too? Imagine the crying/screaming/nasty accusations that would ensue within the FS community if Wales required a previous installation of England in order to have a complete texture set!

Share this post


Link to post

Some might be. Many won't.

FTX manipulates BOTH landclass AND textures with the textures mapped to work with the Regional landclass. That is why you have so much variation on a theme and why ANY attempt to de-duplicate will cause problems with non-neighbouring, lack of colour-matched or just missing textures.

 

What everyone here is trying to tell you, all snarkiness aside, is that if you don't know what you are doing, leave it alone.

ALL textures are NECESSARY

 

 

If you want to know more, ask the developers in their own forum, as they don't answer technical questions here.

Nobody said there won't be duplicates, I believe if you were tasked with creating 3000+ custom LC textures for a region you'd probably be looking for ways to reuse anything you could as well, no? The real question though is how do you make a landclass .bgl in England\custom\scenery call a texture from Wales\custom\texture so as not to have duplicates? You can't, and even if it were possible, what about folks that own Wales but not England? The Wales people are going to need need a complete texture set too? Imagine the crying/screaming/nasty accusations that would ensue within the FS community if Wales required a previous installation of England in order to have a complete texture set!

Gotcha to all of you!

 

Just like CODE...RE-Use it's the operative keyword here, NOT redundancy.

 

...if it scenery file use was predicated on selection from a central database format, kinda like FTX Central is now with ...with An SQL or NOTSQL with relational indices, pseudocoding the idea,

 

CaseOwnerOfEngland then

Share CommonEngWales Xtexture

CaseOwnerOfNZ

Download XtextureNZNorthNoshare

case...

Case...

EndCase

 

Pre-staging common files based upon what you already have.

 

it could be figured out algorithmically so that there were no redundancies of texture tiles..

 

...playing devils advocate with myself, it could be that process might slow things down unacceptably, but I would really be interested in the scenery engine code which decides what...

 

...goes where...

 

.....and when.

 

I guess we can put this puppy to bed ...

 

...but before we put it to sleep, does anyone know whether scenery/world/texture files have changed since version 1.4?...I can still download 1.4, and re-install it to find out, but a quick answer would sure save me some time.

 

I'm afraid that I may have messed up in my many copies over from various versions of fs,fs9,fsx, fsXv??,p3d 1,1.4,2,2.1,etc,etc...and ended up with redundancies of my own doing .

 

...I appreciate all the input!

 

 

Chas

Share this post


Link to post

Gotcha? How's that? I understand the concept of querying a database if that's what you mean. Sure it would probably be possible, it's probably possible for humans to inhabit Mars. It's not practical however and it certainly wouldn't be practical for some company that has sale-able products packaged up and ready for download to tear them apart, blow them to smithereens, and attempt to restructure everything on titles that have been out for 5 years or more and have therefore run their course financially - to save someone on average what, maybe 5 Gb of disk space optimistically? When you can buy a 500 Gb SSD for $200 USD? And now we're back to selecting a region before we start the sim so some utility can copy textures first? It's a nice concept but I'm afraid I'd rather forfeit the disk space and have the team busy working on SCA.

Share this post


Link to post

Each product is designed and expected to work as a standalone. each product is also expected to be able to integrate with the next, seamlessly. Database manipulation aside, and it's no way near as simple as you describe, it's simply impractical to think that a developer could develop a solution that not only meets that varying requirement, but also configures appropriately with the pre-existing framework of textures AND allows switching between various "modules", local, global or regional each of which can call different textures separately, depending on different landclass calls that are structured differently in the aftermarket with deliberate intent.

 

You still don't seem to grasp the complexities of the interrelationship between the existing and alternate textures, and the alternative to the alternate textures, and the alternate textures for the alternate textures for the landclass calls, the alternative-to-those textures that are called dependant on whether other products from the same, or another developer, are installed. The list goes on and on.

 

What you do need to grasp is that textures are only called when required and otherwise sit dormant on the hard drive, much like your FSX installation can sit alongside your P3D installation, with many textures of the same name as in P3D but with zero impact on the drawcall requirement. It is precisely the same as default - which has numerous files of the same name to deal with the `five seasons` concept that is at the heart of the visual presentation of P3D, as it was for FSX before it. 

 

If this is all about hard drive space then, save up your pennies and buy another one, remove what you don't need to save space or, if you have any doubt as to the fidelity of the P3D installation the solution is equally simple - uninstall and start again. That will have the biggest impact on performance. Although still not much, as it happens.

 

If you have suggestions to make to L-M about the structure of their file, folder and texture dependencies, this isn't the place for it and they have a whole forum dedicated to it on their website.

 

Likewise, for FTX and their product range. Good luck with getting JV to explain their detailed decision-making process about file structure ! 

 

Here, you're just wasting your time. Nothing you have said will predicate any change in anything whatsoever. You have your choices. Choose one.

Share this post


Link to post

Just my 2c worth here. As far as I know there's nothing to stop you combining all your scenery files into one scenery folder and all your texture files into another texture folder (over-writing as you go).  (This is what I do for add-on scenery items such as miscellaneous airports I download.)  However this might slow down the sim as it looks in large folders for required files.  However the ability to action individual sceneries will be severely reduced, and any time you download updates etc you will need to manually move the files around.  But if you have a week to spare to experiment it would be fun to see what effect amalgamation would have.

Share this post


Link to post

Chas,

 

I believe what you are looking for is something like a .xml or .cfg file that can be used on the base files so all that needs to change would be path reference and avoid the copying of files around.  This should NOT cause any performance issue and should save on disk space.

 

Biggest problem with implementing something like that is it WILL break compatibility IF you have 3rd party vendor doing it the new way and another do it the file copy way ... but I believe that does sometimes happen anyway with existing process hence why some 3rd party products are not compatible with other 3rd party products (Look what happened when just the Terrain.cfg moved in P3D 2.5, that was a BIG deal for some 3rd party).

 

Would such a change be worth it?  With dropping SSD prices and higher capacity, I'd say probably not.  Fortunately photo realistic scenery has no such limitation other than the more you enable in the scenery.cfg the more likely you'll get issues around loading of terrain textures (aka the blurries).

 

Cheers, Rob.

Share this post


Link to post

I believe what you are looking for is something like a .xml or .cfg file that can be used on the base files so all that needs to change would be path reference and avoid the copying of files around.  This should NOT cause any performance issue and should save on disk space.

 

...

Would such a change be worth it?  ...

 

Thanks Rob!

The whole purpose of this thread was to prep for Win X...in other words...clean up my mess ups of file manipulations from pre-FS9 to v2.5 of P3D…

And all these snarky meanies...jump on my arse...I'm goin' out to the garden and eat worms .

 

Chas

Share this post


Link to post

Sound advice, freely given to correct your misapprehension and lack of understanding, is snarky, is it ?

 

Good luck with the disaster that you are about to inflict on your system. Everyone has warned you. It's down to you to heed the warnings.

Share this post


Link to post

I think the OP has expressed a valid concern and opened a discussion on a number of valid issues. :Shocked:

 

IMHO, AVSIM should use some reverse psychology, and bestow a "SNAVE Award" to the person(s) who consistently post the most reprehensible content in these forums. :Idea:

 

Perhaps a "SNAVE alert" button can be provided under every post to allow easier voting by forum members ? :rolleyes:

 

GaryGB

Share this post


Link to post

GaryGB

 

The OP made a request for information that has been fully met by a number of posters. It's his attitude to the advice offered that is really in question.

 

What's your advice on the subject then ? 

Share this post


Link to post

GaryGB

 

The OP made a request for information that has been fully met by a number of posters. It's his attitude to the advice offered that is really in question.

 

What's your advice on the subject then ?

* Accept the fact that people now have (and IMHO, should always have) the right to ask questions on- and discuss- issues related to OrbX products in the AVSIM "Unofficial FTX | ORBX Support Forum" (or anywhere else on the 'inter-web') ...in addition to- or instead of- any "Official FTX | ORBX Support Forum".

 

 

* Recognize that the manner in which authorized Orbx representatives and/or "FTX fan-boys" reply to threads in the AVSIM "Unofficial FTX | ORBX Support Forum" (or anywhere else on the 'inter-web' including any "Official FTX | ORBX Support Forum") ...may have a greater impact on potential future OrbX FTX product sales, than any questions raised / opinions expressed / aspersions cast / rumors spread etc. ...in posts by existing or prospective end users of such products.

 

 

My reply in this thread is (thus far) not intended to address issues involving the very real, and IIUC, in most cases "purposeful" logistical and methodological complexities involved with how OrbX FTX products are distributed / configured / updated on an end users system, and only has to do with the ongoing issues cited above. :wink:

 

GaryGB

Share this post


Link to post

Then one must ask, while you are so busy critiquing the critiques - all of which are completely correct - are you not prepared to offer advice of your own ?

 

There are no fan-boys here, only those with experience and knowledge of the FTX product range, and what complications can and do arise from a non-standard approach to installation and operation. 

 

Your latest reply still contributes nothing to the OP's original question, neither in support nor contradiction, so I am reporting your offensive post about forum members to the mods.

 

Hopefully the topic stays alive.

Share this post


Link to post

While I do believe it would be a good idea for MSFS / P3D add-on developers to include their own "Do's and Don'ts" document within their product manuals, I am further compelled by what I have read thus far in this thread to now:

"...address issues involving the very real, and IIUC, in most cases 'purposeful' logistical and methodological complexities involved with how (ex:) OrbX FTX products are distributed / configured / updated on an end users system". :Idea:



I believe that the best course of action for the OP and others similarly concerned about installing / maintaining / updating MSFS / P3D add-on products such as OrbX FTX, is simply to use a hard drive / SSD which is sufficiently large enough to accommodate the existing MSFS / P3D folder chain, and to allow for future expansion of installed add-on content etc..


Backups of any files which are not intended to be subject to changes by configuration / update procedures of any MSFS / P3D add-on products such as OrbX FTX, certainly may be kept on separate physical or logical storage drives formatted with the NTFS file system.


In theory, 'certain' files and/or folders kept on such separate physical or logical storage drives formatted with the NTFS file system may be "linked" to nested ex: "OrbX" sub-folder locations in one's ex: MSFS / P3D folder chain.


However, I would caution against using "link" technology for any files / folders which are- or which in the future "may be"- subject to changes ...by configuration / update procedures of any such MSFS / P3D add-on products such as OrbX FTX.


When files and/or folders kept on separate physical or logical storage drives formatted with the NTFS file system are "linked" to folder locations in one's ex: MSFS / P3D folder chain, without 'proper' Windows NTFS security permissions, those 'remote' files may- or may not- be accessible ...to allow intended configuration / update procedures of any such MSFS / P3D add-on products such as OrbX FTX.


And when proper Windows NTFS security permissions are in place, while those "linked" 'remote' files may then be accessible to allow intended configuration / update procedures of any such MSFS / P3D add-on products such as OrbX FTX, they may also be vulnerable to mishaps of un-intended file management ...as recently discussed in this thread:

http://www.fsdeveloper.com/forum/threads/windows-10.434667/page-2#post-717285

 

http://www.fsdeveloper.com/forum/threads/windows-10.434667/page-2#post-717302


For those who feel they 'must' utilize linked files / folders (hopefully on either a very carefully considered and/or IMHO, "temporary" basis ex: while saving up for a larger storage drive), use of "Symbolic Link Clones " via the excellent Link Shell Extension (aka "LSE") utility by Hermann Schinagl, rather than more commonly used link technology, might be a 'comparatively safer' option to consider ...as also discussed in the above thread.

 

http://schinagl.priv.at/nt/hardlinkshellext/linkshellextension.html
 

 

However, even this latter 'comparatively safer' method would subsequently involve additional very carefully considered manual maintenance by the end user, to copy and/or move files from the ex: OrbX nested sub-folder chain following intended configuration / update procedures of any such MSFS / P3D add-on products such as OrbX FTX.


IMHO, this latter scenario would be a process fraught with risk of error, and would likely be a time-consuming and redundant task.


Thus, I would again advise 'most' end users to not attempt using "link" technology for any files / folders which are- or which in the future "may be"- subject to changes ...by configuration / update procedures of any such MSFS / P3D add-on products such as OrbX FTX. :Nerd:



Another concern which I am compelled to address is the (well-intended) use of a utility such as "DupeGuru - Picture Edition" with MSFS / P3D to analyze / identify "duplicate" image files among the multiple legacy, current (and future ?) FS image file formats, which in many cases are either completely- or semi- proprietary, and may have unique internal attributes and data structures.

 

I would advise against attempting to use such a utility with MSFS / P3D image file formats, as it may not be able to properly identify the numerous 'unique attributes and data structures' required / utilized in MSFS / P3D image files.


Even if one used a utility which was 'fully-compatible' with all the multiple legacy, current (and future ?) image file formats of MSFS / P3D (ex: some utilities by Martin Wright) to identify "true duplicates", all those image files are located in specific locations under specific file names, for reasons of implementing scenery display according to the developers' intended method of file utilization, to allow proper rendering at run time in MSFS / P3D.

Thus, one would risk disrupting operation of the MSFS / P3D rendering engine by changing / moving / deleting certain files. :wink:


For the same reasons, IMHO, one should not attempt to combine "all your scenery files into one scenery folder and all your texture files into another texture folder" as suggested earlier in this thread: :excl:

http://www.avsim.com/topic/472281-tens-of-gb-of-orbx-texture-duplicates-found-across-product-line/#entry3277419.


Again, AFAIK, the configuration / update procedures of any such MSFS / P3D add-on products such as OrbX FTX may not be able to perform their intended functions, and one would risk disrupting operation of the MSFS / P3D rendering engine by changing / moving / deleting certain files.


I hope this info may prove helpful to all the boys (and girls) who are "fans" of MSFS / P3D add-on products such as OrbX FTX. :smile:

GaryGB

Share this post


Link to post
While I do believe it would be a good idea for MSFS / P3D add-on developers to include their own "Do's and Don'ts" document within their product manuals, I am further compelled by what I have read thus far in this thread to now: [\quote]

 

Terrific response and well researched as well.

 

If I didn't mention it, I used further comparison by using DXTBMT by Martin Wright.

 

Regards to all,

 

Chas

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this