I discovered Tileproxy only recently and I was very impressed by the photoreal scenery which it produces. Indeed, it has revived my interest in flightsim! However, there were some major problems near the shoreline: the first was the blue rectangles appearing in the sea, seemingly at random, as described by Ken. In addition, I had bands of varying shades of blue extending from the shore out into the sea, and triangles of land jutting into the sea along the coast, like the teeth of a saw. By trial and error I was able to solve most of these, and I give my solutions below in the hope they might help Ken or others who are having similar problems.
1. Within FSX, ensure that Settings\Display\Scenery\Water effects is set to at least Low 2.x.
2. In ProxyUser.ini, change the water_rgb value to water_rgb=#00341A
The above modifications should get rid of the blue rectangles, at least they did in my case. For the problem of the blue bands and the triangles of land jutting out, try making the following changes to ProxyUser.ini:
blend_distance=50.0
blend_exponent=2
alpha_mix=0.0700
alpha_max=0.3348
The effects of these changes was to get rid of the blue bands completely and to distort and blur the green triangles of land so much that they become virtually unnoticeable, unless you are looking for them.
I am not at all technically-minded, so I have no idea why the above changes worked, but if someone would like to explain I would be very interested.