If your idea of a good vacation means being by the water, make a beeline for Constitution Island. No matter what you choose to do here, you're unlikely to get bored of those lovely views.
You'll need to find your way to Cold Spring to visit this water-surrounded land mass. If you plan to do some sightseeing afterward, why not head for United States Military Academy? It won't take long to get to from Constitution Island.
