Not sure what you mean by a systems/weapons engineer. There isn't really such a job -- at least not for junior officers coming out of USNA. If you mean a Civil Engineer (CEC), since they don't drive ships, I believe you could do that if you are color blind.
The prohibition in the USN on those with color blindness is aviation, submarines and surface. You should be able to do most restricted line and staff corps jobs, which includes CEC. (You can Google to find what other Navy jobs fall into these categories). However, getting a CEC billet isn't guaranteed (even if colorblind). You typically need to major in engineering as a start. And then you have to be sufficiently competitive.
If you're looking to design ships/weapons -- as noted, that's not going to happen right away for any junior officer. If you're looking to be in charge of weapons on ships, that's not going to happen if you're colorblind.