I have exactly the same issue and it happens with both Airbus aircrafts - Aerosoft A330 and FSLABS A320.
I have ran voice training many, many times. This may seem strange but I actually saw my speech recognition deteriorate after installing FS2Crew for Airbus. Commands that never had any issues with suddenly became an issue like 'Set', 'Yes' 'Up'. It may not be related but strange that it started to happen after installing the Airbus products. I've even recorded my own voice as part of the speech recognition functionality and although it has got better it's still not consistent. Perhaps it's a Windows issue? I don't know. Tried with different headsets as well and same issue.