Over the last few decades i have talked to many people about self image. and the thread that i seem to find is that the image people develop of themselves in their early to mid teens seems to stay with them for the rest of their life for the most part no matter how well or badly they do later in life. Not scientific just my observation.For me it's as much about how I view myself as how others view me. For example, I know my wife finds me sexy, but if I don't feel sexy, my ability to enjoy sex is greatly reduced.
What you said about finding all the imperfections and not being satisfied no matter how much you improve.. this is true. In fact it seems like people who never try to improve their appearance (besides things like clothing), are way more confident and satisfied with their appearance than when they start trying to fix things.