Whatever the religion or whoever they're dedicated to, shrines have a way of making you reflect. Check out Nogi Shrine and share a moment of silence with the local faithful.
You're spoiled for sightseeing choice in Minato. Nogi Shrine is a good place to start here, but there are lots of other interesting spots to enjoy. Tokyo Skytree, Tokyo Tower, and Tokyo Imperial Palace are close to the sacred place and will fill in your day nicely.