The question is . . . what is the "job?" Is the "job" being a USN officer or serving in a particular warfare specialty? Or being a midshipman?
Clearly a USN SEAL needs a higher level of fitness to perform his job than an officer serving on a submarine or ship or aircraft (or being a CEC officer, nurse, supply corps officer, etc.). The SEAL job has very high physical demands; driving ships/boats/airplanes requires many skills, but being able to run 4 miles in the wet sand in combat boots in 28 minutes isn't one of them. In order to be a ship driver, etc. there aren't a lot of physical skills required -- the USN really wants you not to be a fat slob -- so it imposes a test that measures fitness, and fitness has different standards for men and women in the USN for the same reason men and women don't compete vs. one another in most athletics.
If the USN imposed the SEAL standard on all officers, there wouldn't be enough people to fill the officer (or enlisted) billets. WRT mids, one must ask what physical skills are necessary to serve as a midshipman, who is not going to war while at USNA or even serving in a unit (other than during the summer). When they pick their warfare specialty, they will be tested for the required physical skills at the appropriate time and must pass, regardless of gender.
A USN PFT is designed to ensure that military members stay in shape. Because the test is administered to people of many ages across many, many disciplines, it is a general fitness test and not designed to measure anything more than that. Those members of the USN who need to be in better shape to do their jobs must meet the requirements of their particular warfare specialty -- e.g., SEALs, EODs, etc., -- and not only to get in but to stay in.
As an aside, I can only speak for the USN from personal experience. It may be that other services have a greater need for all officers to have meet certain physical standards in order to do their jobs. .