Your best bet to fly is in the Army, Navy, or Marines, though you still may not be able to in those branches. The only reason I say you have a better chance there is because they offer another "color-blind" test called the "Falant" if you fail the Isihara dot system(which most color blind people do, even if you're slightly color-blind). You have to pass the Falant or the Isihara just to serve in the Navy, and both will qualify you to fly, even if you fail the Isihara. The falant will also be giving as a remedial test in the Army and Marine Corps(for aviation), if you fail the Isihara. If you still want to join the military and not fly, then the Marines and Army offer the vivid green, vivid red test, which will qualify you for most ground MOSs(including Combat Arms). The air force will not let you fly if you fail the Isihara, but you can still join and do ground duties.
From a fellow color-blind person, good luck!
Edit:
Same rules pretty much apply to these branchs' respective academies: West Point, you just have to pass the vivid test to be qualified. Air Force, you can go, but you aren't qualified to fly. You have to pass the Falant or Isihara to go to the Naval Academy. On the latter, if you are wanting to go Marines, who only require the vivid test, you may be able to get a waiver to go to USNA, though I hear it's unlikely to get this waiver. You can still enroll in NROTC-Marine Option without a waiver.