You're not good in math because you don't try hard or you're just bad in math period (i.e. left side of your brain is weak)? Unlike a typical university
everyone at the academy,
regardless of their major, have to
take the same math and science classes, which is equivalent to upper division college math and science. For example, if you were a History major at a university, you will not be required to take Calculus 1-4 (1-6 if you're on a quarter system). Can you say differential equation? I personally think that USAFA's curriculum is overkill since most will not end up with an engineering degree. More importantly, very few, I mean, very few engineers in the Air Force do any sort of actual engineering work. You will, however, become masters of engineering briefings and inventing techniques sleeping with your eyes open during those endless briefings. I digress. Remember, the Air Force expects everyone to become a general.
To answer your question, I would not go to the academy unless you can greatly enhance your interest in mathematics and become comfortable in calculus by the time you set your foot in the zoo. Keep your focus on the end goal and not the school/institution.