Primary care physicians, by necessity, are used to dealing with the general population. Go down to the local Wal-Mart, post office, or DMV, stand in line and look at all the "normal" people around you, these are the people your doctor is used to seeing on a daily basis, and most of his or her training is based on that general population.
Most of the patients that most doctors see are fat, don't work out, live off junk food and soda, and have mostly sedentary lives where many of their problems are based on their lack of physical activity and a healthy diet. Primary care physicians are used to treating the problems of the general population on a daily basis, and most physicians don't have much training or experience in the more esoteric problems faced by the fitness, bodybuilding, or TRT/HRT community.
It's hard work but worth the effort to seek out a primary care physician who is used to dealing with a more athletic population, just so they have more experience and a better appreciation for dealing with the specific problems that we face.
I've learned, with all my doctors, to take what they say with a grain of salt, and then to seek out other opinions from people who are experts in dealing with my specific concerns. Doctors are human beings, and just like anybody else, they make mistakes and are not omniscient, they don't know everything about every problem under the sun. But generally, they do mean well, and from their perspective they are trying to give you the best advice they can on improving your health, which may be in opposition to your athletic or physique enhancement goals.
Try to find doctors with the most relevant experience you can in dealing with your specific problems, have them order the bloodwork or get your own bloodwork done, and look at the numbers, take them to different experts and try to get a consensus opinion on the best course of action for your specific situation.