Lest you think it's tough today, we took 20/21 semester hours as a plebe. Personally, I think the 16/17 load today is much better. Seriously. During our 2/C year, going into the EE (wires) final, 60% of the class was failing, as in getting an F. With the prospect of having to accommodate hundreds in summer school, they gave a very straightforward final, allowing most of the class to slip by with Cs and Ds.
But enough with the war stories.
I think the reasons USNA is so hard academically -- OTHER THAN the military commitments, etc, which are substantial -- include:
1. You MUST take the classes when you're told to. You can't move most of your classes to an "easier" semester.
2. As the above poster said, they don't grade on the curve. Many mids struggle to earn Cs and even Ds.
3. You are always carrying a heavy course load. At most civilian colleges, you can't take more than 15 hrs w/o special permission. At USNA, you can't take fewer than 15 period. Most mids are carrying 18-22 hours + labs every semester after plebe year.
4. There are no easy courses. Even putting aside the "jock" courses, most civilian colleges (and many grad schools) have courses that let you pad your schedule. USNA has no such thing. Every course is demanding.
5. You must pass every course and you must graduate in 4 yrs. At most schools, if a course is too hard, you can drop it. You can't do that at USNA for your core courses or required majors courses. And you must finish in 4 yrs, not 5 or 6 or . . .
6. Most core courses have a single final exam (and often quarterly exams) so that having a "gouge prof" doesn't help much.
Bottom line is that USNA is a really tough academic experience which is why USNA encourages folks to take challenging h.s. science, math, and English. Having that preparation really helps you.