The simple answer, as others have indicated, is yes, you can be sent home during swab summer as well as at any time during your four years there. During swab summer, the most common causes of attrition are medical reasons and people choosing to leave. The medical reasons are typically due to pre-existing conditions or more chronic medical issues. When I was a senior, a new policy was implemented that stated that a swab could only be on limited duty (i.e. restricted physically due to some type of injury) for a certain period of time before disenrollment was considered. I'm not sure if that policy is still in place as it's been a few years since I've had any visibility on swab summer. If someone is disenrolled, there is the possibility of them receiving a conditional appointment and deferment to the next incoming class the following year, as long as they are cleared medically and fulfill a few other requirements such as taking certain academic courses, retaking the PFE, etc. I was medically deferred from the class of 2013 to the class of 2014 during swab summer. I met all of the requirements and returned to complete swab summer with my class. Yes, I was at home all year and attended community college as it was too late to be accepted or attend a four year institution but I can say with complete honestly that some of my community college courses provides the most diverse learning experiences I've ever had. To be in a class with not only recent high school graduates but adults returning to school at 40, single parents, active duty military, etc. was a great opportunity to hear different perspectives.
The possibility for injury is also why some on this forum would recommend maintaining a Plan B, often by putting down a deposit with a school and asking the institution if it would be possible for a spot to held for your DS/DD through the summer. As far as I know, a person would not receive a deferment for conditions such as color blindness or weight issues. I had a classmate kicked out the week of graduation for not making weight. There are standards and everyone is required to meet them.