BTW, there are two issues here -- both the lack of "varsity" sports and the sport you've chosen.
USNA likes varsity athletes for several reasons. First, most varsity sports consist of teams in some form and USNA likes to see teamwork. Second, they like to see competition/"combat" on the athletic field (or in the pool, etc.), because that tends to produce the type of person who will do well in the military. Third, someone who likes and excels at sports is probably physically in good shape and that bodes well for USNA, where athletics is a huge part of the program.
You might consider a team sport over the summer -- some sort of youth league (city, Catholic, police, etc.) -- in a sport that requires running, such as soccer. You need not be terrific, but will be able to add athletic teamwork, etc. to your experiences.
You need to do REALLY well on the CFA. Train for it. Consult a school coach or a personal trainer and figure out what you need to do to max out (or get close in) every event. If you don't participate in varsity sports, USNA at least looks VERY HARD at the CFA.
Finally, be prepared to explain to your BGO and your MOC nominating committees why you didn't do other sports at school. There are at least three sports seasons (fall/winter/summer) which allows most athletes to play multiple sports. You didn't and the sport you chose is one that some may not consider a "real" sport -- BTW, I'm not making value judgments, merely stating what you might face. You should be prepared to explain Frisbee -- how that keeps you in shape, builds teamwork, etc. b/c to many people it seems like something you do at the beach. You need to disabuse folks of that idea.
The reasons for not doing other sports are also important. Do you not like team sports? Does your school not have many varsity sports and thus none at which you were good? Is your personal/family situation such that you weren't able to participate in after-school activities (i.e., parent(s) working and no after school transportation)?
You need NOT explain your reasons here but reflect on them and consider whether, in light of them, you will be happy at a school where organized sports are both mandatory and incredibly important.