I don't know if this is similar, but wanted to throw it out there. In the PMDG 737, when I fly really long legs (an hour plus between waypoints) it will start doing the back and forth thing. I've never let it go too long, correcting it by going back direct to the current waypoint, or killing the A/P. As long as I have a waypoint every couple hundred miles, I have not had it be an issue.
Were you flying really long legs between waypoints when this happens?
If I get a long trip in the 777 before resolution, I'll post my observations...