Rowing is a great sport. It is one sport at USNA that really welcomes "walk-ons," some of whom have ended up making it onto the Varsity "A" boat. There are many different sports opportunities at USNA, and you will have chances to try many of them, including crew, during sports periods in Plebe Summer.
However, rowing, or being on any varsity team, may not be the right thing for a given Mid to do. There are some real pluses and minuses to being a varsity athlete at USNA. The team camaraderie is great. You're treated as a team member, not just as a Plebe, and having a "refuge" can be salvation for a Plebe. However, being on a varsity team requires a LOT more time away from your company. You may like having a reason for missing SMT, but if you don't work as hard as possible to contribute w/i your company, it can have negative ramifications on your rank in company/military aptitude/OOM.
This is not to say that any given person should or should not do crew (or any other varsity sport). Just keep in mind that people can have many valid reasons for why they may decide not to continue in a particular sport.
As for being willing to give up spring break (or not), you might feel differently about that after about 6 months of having 5-12 hours per week of liberty.