There will be different opinions from each person on this subject. All Alumni are very passionate about their respective Service Academies and that is what helps make them so great. Coast Guard (I think) statistically is the toughest to gain admission. But, they also do not require a nomination, and that process is unique itself. Each Academy also has its own set of academics and majors. If you want to major in Operations Research, don't go to Navy (unless something changed). Not sure if any single one is tougher than anothers academically. I had an Air Force exchange cadet and a West Point exchange cadet that were my neighbors when I was a 2/C. The Air Force cadet did well academcially and the Army cadet struggled. The Air Force cadet thought they were about equal and Army's cadet said it was much tougher. Now that is just one very small example and doesn't paint a picture as a whole.
I think all the Academies indoc summers are pretty tough. If you don't want to run around the woods, then Army is not the place for someone. If the idea of living on a ship is the worst thing you can think of, do not go to Navy. I do think West Point takes the military side of things more seriously than Navy or Air Force, but that is just my opinion based upon input from friends and my exposure. At Air Force I think you get more privledges for civilian clothes and driving earlier on than the other other schools (not 100% positive on that).
The most important thing to factor in when selecting a SA is what do you want to do when you graduate? If your passion is submarines, well there is only one place to go and select subs. If your goal is to just serve and you will sort it out later, well any Academy would probably work. There are great parts to each school. In my mind the thing that sticks out about Navy compared to the other Academies is its location, beautiful campus, and large commissioning opportunities between the Navy and Marine Corps.