The doctor at your physical is not the one who makes the decision if you passed or not. DODMERB makes that decision. Once DoDMERB receives the paperwork from the doctor, something will be posted to your DoDMERB account, so keep checking that. I don't know what's common timewise, but my DS had the following:
15-JUL-13 Actual date of exam
18-JUL-13 Date exam received at DoDMERB
18-JUL-13 Medical data entry
19-JUL-13 Date exam reviewed
22-JUL-13 Date letter sent - Remedial
14-AUG-13 Date exam reviewed
15-AUG-13 Date letter sent - Qualified
Depending on when your exam is, and how quickly the doctor sends in the paperwork, the initial process doesn't take long. If you are taking your exam at a busy time for DoDMERB, it will take longer. And if you have one or more Remedials or are initially DQ'd, it will take longer. Everyone is going to have a different experience.