Hey, there's a bit of truth to every rumor. At the academy there certainly seems to be a hierarchy to the engineering majors. Each one presents their own set of challenges. For example, mechanical engineering is a very broad topic that can be taken many different directions depending on the student. Systems (robotics) is very coding intensive, while also incorporating aspects of mechanical. Electrical engineering is its own beast that strongly diverges from the other majors.
Aero has its own quirks that make it difficult, and I will be speaking to the aeronautical side (rather than astronautical). We start very similarly to the mech's, learning a foundation of structural engineering skills. However, we quickly diverge through a series of fluid dyanmics courses (Aerodynamics I & II, Gas Dynamics, etc) that deal with high speed compressible flows that are typically encountered in aeronautical applications. These courses largely focus on engineering at the smallest level (think about individual molecules in a flow).
There is also the category of classes regarding the aircraft as a whole. The most prominent of these are Intro to Aero, Performance, Stability and Control, and Flight Propulsion. These are unique from most other engineering classes, as we analyze the large scale behavior of vehicles, whether they be propellor, jet, or rocket driven. However, they give an excellent blend of knowledge to be taken into real world aviation careers.
Lastly, and perhaps most unique to aero, are the writing and analytic skills classes. The gauntlet for many aero majors is Wind Tunnels, where we make several presentations, write a 40-60 page engineering paper with LaTeX, and design our own experiment to run and evaluate by the end of the semester. Then, there is Flight Test Engineering, which is an excellent blend of every aero class taken thusfar, and involves several flight operations in a Be55 Baron followed by extensive data reduction and reports.
So, as you can see, the disciplines required for study in aero are broad. However, that being said, the actual difficulty is up to the individual person. I couldn't see myself doing any other major, no matter what school I chose attend. And if you stay on top of your work, you should never have a night past 9 or 10 pm. As a pre-requisite to the major, I would definitely agree that strong skills in calculus and physics are a necessity. On top of that, you will find that effective communication and writing skills make your life easier in any engineering major.
Hope this gives you some insight to the major!