My personal opinion? No, it's not too small. In fact, I think smaller is better. You will know every classmate, and basically everyone a year ahead and behind you. You will know a majority of the people two years ahead and behind you. And you will know some of the people three years ahead and behind you. A single class at the Naval Academy is larger than the entire Coast Guard Academy (all four classes).
Draw back? No getting lost in the crowd (that's not really a drawback). You CAN be a wallflower, and some manage to skate by at a USNA-size service academy. You can't really do that at CGA.
The officer corps in the Coast Guard is tighter than any other service. You will see some of the same people over and over in your career.
That means if your a bad officer, people WILL know. And if you're good, people WILL know.
There are 50,000 Navy officers. There are 6,000 Coast Guard officers. You would expect a smaller service academy to feed such a small service. Can't have 1,000 new officers dumped into the fleet each year.
Also, consider this, there is no ROTC program for the Coast Guard. So, 40% of the officer corps is fed by the Academy. The other 60% comes from OCS and direct commissions.
I'd say that's a pretty select group of officers. But then I am biased.