Don't worry about being the old guy
At 23 you are hardly old, but compared to my 17 year old who will still be 17 when in July, you will be be like some sort of yogi guru. When I was at the Naval Academy (in the last millennium), the guys who came out of the Fleet or the Corps rose to the top in terms of leadership of our class. Your six extra years of life experience will definitely work to your advantage. The older midshipmen in my class tended to struggle more academically because math and science were not their strongest areas--but the challenges were not insurmountable if the ultimate objective is not to get a 4.0 but to pass with dignity and graduate.