It's easy as a h.s. junior or senior to think you want to go to grad school right after undergrad. I can tell you that, after four tough years of undergrad (for a total of 16 straight years of school), a lot of folks are tired of school and ready to start "working." (Recognizing that most USNA grads will go on to "school" for aviation, subs, etc.).
You either think you need to start grad school right away b/c your knowledge is fresh or you worry that you'll get out of the "school" mode and won't do as well later. For the most part and for most people, neither is true. MANY people go to grad school, even for very technical subjects such as engineering or even medicine, 5, 10, 20 or more years later and do very, very well. On the second point, while it may take a few weeks or months to "get back into the swing" of school once you've been out for awhile, other things such as maturity, enjoying school, etc. quickly make up for it.
Naval officers are encouraged to attend grad school later in their careers and, for advancement to (I think) O-5 and above, it's mandatory. But, in the short term, USNA is turning out warfare officers, so consider that when making your decision.