I see many transatlantic flights that don't follow the NAT tracks, especially flights arriving or departing from the US west coast from and to Europe. How would I go about and determine a custom route like that, one that goes over north Greenland, even into the polar regions? Is there a standard procedure for determining that type of a route, e.g. waypoints being apart by a certain amount of degrees or does it all come down to the wind component and distance?

Welcome to Avsim!

It depends on a lot of variables including prevailing weather conditions, aircraft capabilities and airline's ETOPS certification, plus an airline's route planning software and its own internal operating rules.

As mentioned in this link, the non-NAT routings are using 'Great Circle' routes taking them far to the north:
https://askthepilot.com/questionanswers/great-circles/

A really good free route planning site is SimBrief, though it can appear complicated if you are just starting out flight simming.

In the meantime, a simple way of route planning is using the recently filed real world routes tool here:
https://uk.flightaware.com/statistics/ifr-route/
 

The unfortunate thing about Flightaware is, it does not give you the whole flight plan when going west (sometimes). EGLL-KLAX it only showed the North America part of the flight plan.

EGLL-KIAD it showed the whole flight plan going west. So i guess it varies on the city pairings.

nebojsa