cessna 182 elevator trim

having been flying with the default loadout for so long and now trying lighter weights like only the pilot and one passenger and a quarter tank of gas, i'm finding the elevator trim doesn't seem to have as much effect as when the plane is fully loaded. in a traffic pattern at around 80 knots, trying to fly level, i've almost used all the trim so i have to add power. i don't remember having to do that as much when it was fully loaded. i used a bit of trim and that was that but with a lighter load i have to use alot mostly all of it to stay level at slow speeds. does that sound right? or should i be using more power and less trim? william