Jittery Controls when Idle XP10 & 11

So I've dealt with this for a long time and finally decided maybe I should give this a post and see what others have to say.  In both XP10 and 11 I seem to have jittery or noisy controls, Mostly throttle, spoilers and flaps.  Yoke and rudder seem fine.  The Yoke and rudder are Saitek and the the throttle quadrant is Goflight.  P3D doesn't seem to have this issue as far as I can tell, only XP sims.  If I leave the flaps/spoilers/throttles alone, they kind of jump around in place, not enough to disturb flight, but enough to become an annoyance.  For instance it gets quite annoying in the JAR A320 as the FO occasionally goes in to a loop o "Flaps 1, Flaps 0, Flaps 1, Flaps 0".  I've tried re-calibrating numerous times, no matter what I do it is the same condition.  Any ideas what I can do?