In the East and South, it's the humidity plus heat that's rough to get used to. It's hard to breathe, sweat does not evaporate, and being athletic in it feels like running in a steam room at times. No trade winds like Hawaii. Those accustomed to desert dry heat or West Coast lack of humidity can be hit hard by a New York July, especially for the sudden transition into plebe summer.
Yes, it gets very cold, but there is more time to get used to it. That said, cadets from non-snow/ice hometowns get a few hard lessons in walking on ice, learning to layer and understanding the high humidity of summer is positively bone-chilling in the winter version. Add in winds off the nearest body of water anywhere in the NE, and it's just...raw.