@jchow ,
I am non-military father of an Army officer who commissioned through AROTC, so take this for what its worth.
I know two young men who entered the BUD/S pipeline via enlistment. They grew up in the same neighborhood in solid families. They went to the same high school and graduated one year apart.
Young man #1, the older, had an unremarkable HS career with little or no sports. He hated college and dropped out after Freshman year. He enlisted in the navy, much to his parents chagrin. They were disappointed that he was forgoing college and that he was enlisting in any branch of the military. He entered the BUD/S pipeline immediately and is a SEAL still today eight years later. His parents are proud, but his mother still can't get use to her son carrying a gun.
Young man #2 was a recruited swimmer to an NCAA D1 school, from which he graduated on time. He enlisted, seeing that as the quickest route to doing cool guy stuff. He voluntarily dropped out during Hell Week. He couldn't handle the combination of exhaustion, hunger and the ice cold Pacific Ocean. (Who can?) Fast forward several years and he was a Navy air traffic controller, hating his job and counting the days until he could get out and start his life, but not as an air traffic controller.
There are a number of lessons here, several of which are mentioned by the previous posters. To me the most important:
There is no unimpeded glide path into the SO community
The attrition rate is staggering, despite the fact that everyone shows up in good/great shape
No one can predict how he/she will respond to intense physical, mental, emotional and environmental stress--piled on all at once
I would say that while young man #2 had the advantage of superior physical conditioning, young man #1 had the advantage of making the more difficult and consequential choices--with societal and parental push back--at a critical time in his life.
I think most people would advise you that if SOAS or BUDS don't work out, you would be happier and more challenged meeting the needs of the Navy as an officer.
Congratulations on Norwich and the Citadel. Best of Luck!