First of all, setbacks are never measured from the water. They are measured from the Ordinary High Water Mark as determined by Zoning Office staff or the DNR.
The measurements are taken from the closest point of the structure to the closest point of the Ordinary High Water Mark (OHWM)—even if that point is on a neighbor’s property. The OHWM is also measured from boatslips (even water inside a boathouse) or from indentations along the shoreline.
You cannot draw a straight line from the houses on either side of your lot and use that to establish the reduced setback.