CAPT MJ is, as always, spot on. A few other thoughts.
If you are really, really serious about being an
Olympic swimmer, you don't go to a SA. As a BGO, I worked with a recruited swimmer not too long ago. This person set all sorts of records at USNA. Went to Nationals and finished in the top 16. Finished in the top 50 of the graduating class. That person is now a submariner.
A family friend swam at an ACC school. He was on (and briefly captain of) the US national team. After a few years, he realized that he wasn't good enough to win at the Olympics (he swam in the time of Matt Biondi

). He is now an executive with a major bank.
The point is that, especially today, being a world-class swimmer is a profession. If that is your ultimate goal, you should go to a civilian school that is known for producing world-class swimmers. You focus on that morning, noon and night while hopefully getting an education that will set you up for the long term. When you graduate, you continue swimming as a profession (assuming you're good enough to get endorsement deals).
The overwhelming majority of swimmers aren't in that elite class and never will be. They love the sport and want to do it in college but know that, when they graduate, they're going to do something else for the rest of their lives. They have great college careers, make lots of friends, maybe set a few records. But then they move on.
You mentioned traveling. I can't speak for USAFA, but at USNA, academics are hard. Being away from class is tough -- while traveling is nice to an extent, it's definitely not glamorous and takes you away from valuable class time. I traveled with the swim team (as a non-swimmer) and everyone studied on the bus. They studied on the pool deck. They studied in the motel. They all loved swimming, but they were at USNA to become officers not professional swimmers. And, all in all, they'd rather swim at home than away.
I suggest you start by evaluating whether it's more important to have an
elite swimming career or to prepare to be an officer, assuming you're good enough to be an elite swimmer. I'm not being sarcastic or flip, just realistic, b/c it's very hard to do both at any SA.
If you want to attend an SA, next is which one. There are many good reasons to choose one SA over another. Being close to home is a terrible one.
I would start by asking yourself: if I weren't a swimmer (or varsity athlete), which SA would I want to attend and in which service would I want to serve? See how strongly you feel about that decision. If it's USAFA AND you think your swimming experience will be better as well . . . decision made. The fact that you're debating tells me you're more interested in USNA as a SA and service. If that's the case, you have to weigh 4 years of a perceived "better" swimming experience at USAFA with 4 years of being a mid and 5+ years as an officer.