Has anyone made a VR home cockpit via hand tracking?

[Gaiiden 98]

okay the concept is simple: you have a home cockpit - basic plywood frame cut to the dimensions of the aircraft, hardware knobs, buttons, and switches, properly located controls (yoke, throttles, etc) - but there are no gauges or displays or monitors for external views because you have your VR headset. The headset is tracking your hands and showing their positions. So you just reach out and grab the controls and manipulate them - in real life. As with a normal home cockpit, the controls are linked to the computer to send commands to the sim. This would be a relatively cheap solution in general and also for being able to build multiple cockpits as many of the hardware could be re-used with a different plywood frame. So you don't need MSFS to support hand tracking, you just need the headset to show your hand position atop the game. This should be able to be done separately by the headset itself? So again, this is not someone asking for hand tracking support in MSFS. This is something I haven't seen anywhere yet and I think maybe cause hand tracking is still not that great?

[himmelhorse 1,276]

Gaiiden,

Thanks for introducing this conversation, as I was very close to doing just that myself, albeit from a different perspective.

My initial response is that what you are asking for is really taken care of by muscle memory. I have a desk set up with my trim wheel and two quadrants set up beside me. I have a third quadrant set up for Landing Gear up/down located on the desk to my right and slightly ahead of my yoke (Installed under the top of the desk.) The trim wheel and quadrants roughly align to the right, with the yoke handles. The rudder pedals are where one would expect them to be. I only use the keyboard (spacebar) for ATC activation which I use very seldom anyway and "v" to activate VR in MSFS. This is done before the headset is put on. In reality, this is ALL just muscle memory. As a newbie to VR, I found it relatively simple to manipulate my hardware controls using this method ... enough for it not to be a problem anyway.

Secondly, I bought the Oculus Quest 2 specifically for it hand tracking.  I wanted this to be able to manipulate levers switches and knobs in the cockpit.  To my dismay, I subsequently found that none of our sim platforms support hand tracking.  I think, and firmly believe, that the adoption of hand tracking in our sims would be one of the very best innovations to VR in flightsimming. It could eventually lead to the only hardware required would be the rudder pedals and possibly the yoke, if only to rest your hands on LOL I would simply find an incredible immersion factor in being able to manipulate all my switches, levers and knobs by hand using handtracking instead of the back and forth manipulation of keyboard and mouse.

It would be interesting for much more experienced VR users to join this conversation.  I vaguely recall sweviver (on youtube) playing a sim using hand tracking, I think.  I cannot find that video again so this in a supposition. It may well have been using the controllers.

1 hour ago, Gaiiden said:

So you don't need MSFS to support hand tracking, you just need the headset to show your hand position atop the game.

This is an intriguing proposition too, although, again muscle memory would be enough to negate this need. Also, in this case, you would also need an accurate cockpit set-up and I do not believe that is actually required unless it is of decorative value to the user, but something you cannot see once the headset goes on.

Looking forward to further input with this topic.

Regards

Tony

[Gaiiden 98]

On 7/5/2021 at 7:26 PM, himmelhorse said:

My initial response is that what you are asking for is really taken care of by muscle memory. I have a desk set up with my trim wheel and two quadrants set up beside me. I have a third quadrant set up for Landing Gear up/down located on the desk to my right and slightly ahead of my yoke (Installed under the top of the desk.) The trim wheel and quadrants roughly align to the right, with the yoke handles. The rudder pedals are where one would expect them to be. I only use the keyboard (spacebar) for ATC activation which I use very seldom anyway and "v" to activate VR in MSFS. This is done before the headset is put on. In reality, this is ALL just muscle memory. As a newbie to VR, I found it relatively simple to manipulate my hardware controls using this method ... enough for it not to be a problem anyway

I agree for the large controls that you mention but when you get down to radio dials, MFD buttons, switches, etc - things become much more tricky. It would be way too easy to break immersion. If I built the cockpit right I could always reach out close to what I want to manipulate but - for example the light switches on a C152. Okay I reach down under the yoke, which I have a hand on so I know where it is, and I put my finger on a switch - but which switch is it? I reach out towards the radio column to dial in a frequency and again I know the general location and I can probably grab a dial first try but until I spin it I won't know for sure if it's the correct dial since they are spatially close to each other. So yea, for a basic yoke/pedal/levers/trim wheel setup I can see this working, but for smaller controls on the cockpit dash, I imagine myself fumbling around a lot, which would be annoying. I'm a gymnast & martial artist, I have excellent spatial awareness and I still wouldn't like to do it this way

On 7/5/2021 at 7:26 PM, himmelhorse said:

Secondly, I bought the Oculus Quest 2 specifically for it hand tracking.  I wanted this to be able to manipulate levers switches and knobs in the cockpit.  To my dismay, I subsequently found that none of our sim platforms support hand tracking.  I think, and firmly believe, that the adoption of hand tracking in our sims would be one of the very best innovations to VR in flightsimming. It could eventually lead to the only hardware required would be the rudder pedals and possibly the yoke, if only to rest your hands on LOL I would simply find an incredible immersion factor in being able to manipulate all my switches, levers and knobs by hand using handtracking instead of the back and forth manipulation of keyboard and mouse.

this is going to depend on personal preference - for me manipulating air isn't very engaging and just seems gimmicky

On 7/5/2021 at 7:26 PM, himmelhorse said:

Also, in this case, you would also need an accurate cockpit set-up and I do not believe that is actually required unless it is of decorative value to the user, but something you cannot see once the headset goes on.

Yes that's the plan because that's how this all ends up working. The wooden frame is built to the exact specifications of the cockpit so where the headset shows my hands places them right where they are in the real world in relation to the cockpit controls. The in-game aircraft models are not scaled down or otherwise changed from their actual counterparts

[himmelhorse 1,276]

6 minutes ago, Gaiiden said:

this is going to depend on personal preference - for me manipulating air isn't very engaging and just seems gimmicky

I do not, personally, see the difference to using the mouse to click on switches.

Gaiiden, I am not disagreeing with you in any of this.  However, I was just referring to muscle memory in connection with the throttles, yoke, trim wheel and pedals.

The thing is though, that if we had hand tracking support in any of the sims, you do have your hand represented on everything you touch, twist or manipulate. For me, instead of using the mouse to do exactly the same, this would be very immersive or at least very much better than what we have available now. 

I do not know what VR headset you have, but the Quest 2 has a system whereby you can see through the headset by tapping on the side of it and you can actually see your desktop and surrounds (in your case it obviously would be your cockpit) and that may alleviate your problem somewhat.

I like the idea of your cockpit, but I think I would find it limiting to which ever cockpit you choose. I really like to fly anything and everything I can get my hands on.  By that I mean it is hard to adapt a Cessna 172 purpose built cockpit and wanting to fly a 737 for instance. I found much the same thing when in FS2004 I had 7 monitors set up for a Boeing 737/747 or Airbus cockpit with a dedicated monitor for the OHP (portrait) 4 x monitors for the main panel with outside view, 1 x 14inch monitor for the CDU and another 27inch portrait monitor for the pedestal. I later included 1 x 32 inch monitor on each side to represent a side view. Flying any small GA aircraft using that system just somehow looked strange LOL

Anyway, best of luck with your project.

Regards

Tony

[Gaiiden 98]

18 hours ago, himmelhorse said:

I do not, personally, see the difference to using the mouse to click on switches.

You're right, but I wouldn't be using a mouse either - I would instead be manipulating actual switches on the cockpit panel in front of me. Which I can't see, which is why I need the headset to show me where my hands are

18 hours ago, himmelhorse said:

The thing is though, that if we had hand tracking support in any of the sims

This topic has nothing to do with hand tracking support in the sim. This is not necessary for what I'm planning to do. I just need the headset to show me were my hands are in relation to my body while I have it on

18 hours ago, himmelhorse said:

like the idea of your cockpit, but I think I would find it limiting to which ever cockpit you choose

No, it would be limiting to the type of aircraft I choose but within that category would be very freeform. So I prefer small GA craft right now (C152 and Savage Cub, Seneca for twins, etc) and I can re-use many of the switches, dials and buttons between them, I just need to take them out of one cockpit panel and put them in another. It can all be very modular. Now, trying to take the small amount of switches, buttons and dials from a GA aircraft and use them for like an airliner cockpit, I'd come up pretty short 😛

Also I still get the impression that you're not quite getting what I've laid out in my initial post. I tried asking this over in the MSFS forums as well and the only response I got was someone trying to talk about hand tracking support in the sim. I don't need that! Maybe I need to come up with a more visual explainer...

[himmelhorse 1,276]

Gaiiden,

I think I know exactly what you want...I just do not know how you could achieve it and have simply expanded on that. 

Good luck mate

Tony

 

[MarkJHarris 128]

It's an age old problem and hand tracking must be a right pain to get right, because a lot of companies have been trying for many years and still it eludes them.

One of the problems is the controls in an airliner are quite a distance from you. Remember though, you rarely touch most of them. I doubt many of us would want to put themselves into a PDMG airliner and play the PM role. So in that case use FS2Crew!  If I'm flying, I'll never touch the gear lever or the flap lever! I only touch the thrust levers taxiing out and in, and for the first few seconds, and last minute of flight. For physical controls really, what I need are the electric pitch trimmer on the yoke, and most importantly, a rotary sprung rudder trimmer hald on the right of my rig spaced correctly. 

Some of our 737s have a weird lower electric panel layout, and the trim unit is mounted forward of the rear position that is standard. It's a real bind reaching for it and finding it somewhere else!!!!!

It's a muscle memory thing I guess, I would pay a small fortune to get that tactile trim knob for my sim!  Different aircraft need particular controls and they need to sit in the correct position and move correctly to feel "right". Most do not and in reality you hardly ever use them, the other guy does!

For 2 crew aircraft, FS2Crew is the answer.