The idea as presented wouldn't make sense from a physics standpoint to have the water doing damage. Remember, water freezes at 0 C. So you'll never find water below 0 C (unless it's got anti-freeze). Ocean water may freeze about a degree lower than that.
Now, the reason cold water causes hypothermia in real life is what happens after you leave the water. If there's wind, the air blowing over your wet skin causes evaporative cooling, which eventually causes hypothermia minutes later. In fact, the temperature of the water is largely irrelevant. In cold air, it doesn't matter if the water is 0 degrees or 20 degrees.
So...we can have a situation where all bodies of water, regardless of water temperature, cause damage, but only 5 minutes after you leave the water...or we can just forget about it.
Normal oceanwater (not close to a glacier or a river etc) freezes at around 1.9-2.0 degrees celsius. And that is cold.
Still - water which is only little above 0 degrees, would still drain your body heat in a very short time - and thereby cause "physical damage". But yes - the part after you leave the water is also nasty-cold.