I am not in your situation, so I have no way of knowing all the information. However, based on what you've mentioned, it seems like a pretty clear (albeit tricky) situation to me. I am guessing that one of the following two scenarios is true, most likely #1.
1. Your professor is "green". Perhaps this is one of his first classes, and he wants to make sure to adequately challenge his students. Or maybe none of his students has ever respectfully and thoughtfully told him that his teaching style can be frustrating. If I were in your situation, I would talk--privately--with the professor, and explain that I was struggling with the tests even though I was studying hard and doing all the homework. Ask for his advice; tell him you really want to earn a good grade in this class and are willing to work hard for it. If he is a legitimate teacher, he will listen and try to assist you. You may be surprised at how much a short, candid conversation can help!
A couple of years ago I was in a similar situation with one of my own teachers. I had the same conversation with her--she encouraged me to keep up my efforts, and told me she was satisfied with my progress so far. I ended up earning an A in that class--probably one of my proudest A's.
2. Your professor really doesn't know how to teach, and/or is so set in his ways that he won't even consider reevaluating his methodology. Bear in mind that the only way to arrive at this conclusion is by trying #1 first. This latter possibility is the more difficult scenario, since you are still in his class and certainly don't want to accidentally "tick him off". In this case, I would go directly to the dean/department chair and talk, again,
privately. Explain your concerns, your efforts to resolve the situation on your own, and your professor's reaction. As long as you are careful not to sound obnoxious or whiny, he/she should be glad to help or at least offer advice.
Just my $.02. Keep in mind that I do not have all the facts myself, so this is just a suggestion to consider. Good luck to you, and I hope you and your professor are able to work out the difficulties you are experiencing.