Keep in mind that the Army is primarily a ground-based service. You're going to spend a lot of time at USMA marching, being out in the field, firing weapons, etc. so that you are prepared to go into fields such as artillery, infantry, etc. Of course, the USA does other things, but that's its core mission. The USN is of course primarily sea based. At USNA you spend a lot of time on boats and ships so that when you graduate you are ready to . . . go on ships or subs. Even aviation is primarily (not exclusively) sea-based. Most people have a strong preference for land vs. sea.
The closer call is for those interested in the USA and USMC.
Finally, do NOT go to USNA expecting to be an MD. There are 12 slots each year. You might be able to snag one but it's a long shot and you must be fully prepared to go ships, subs, aviation, or USMC. A decent number of USNA grads attend med school after spending some time in the fleet. [Don't know the story for USMA so won't try to opine]
I advise the following: Imagine you are given a sealed envelope. Inside is the name of the SA that some greater power has decided you will attend. When you open the envelope and read the name of the SA, are you thrilled or a bit disappointed? As someone above said, don't try to rationalize away that gut feeling, because it's probably right.