There's enough at USNA (or any SA) that makes you miserable. Like P-rades, plebe chemistry, pro knowledge tests. Your sport/ECAs shouldn't be among them. Rather than worry about what others think, consider what interests you. What will lift your spirits when you're down. What will enable you to meet other mids outside of your company doing something together that you enjoy. What do you want to do in the limited free time that you have?
I would say it is probably easier to start with a varsity sport and then drop it if you lose interest, find it takes too much time, don't enjoy it, etc. than to try to join up after plebe year. But not impossible. I know someone who was recruited for a sport, lost interest plebe year and quit, then rejoined and had a great three final years in the sport.
WRT ECAs, I don't think most mids pay much attention to what other mids are doing. There are lots of ECAs and most mids find one or two that really speak to them for whatever reason. Unless things have changed, most mids other than maybe your roommate don't even know what other mids are doing for ECAs, other than maybe those requiring travel like D&B and some of the musical groups.