Just wear what you like and don't worry about others. Although the answer to your question all depends upon a person's body type. The fit of your clothing will also make a huge difference. Fit doesn't just have to mean just tight or baggy. If someone is big and lean with good aesthetics then they will 100% look great in proper fitted clothing. Now too tight and you make look like a douche to many. I personally really like wearing tight fitted dress shirts (short sleeved) and it definitely adds to your look especially if you have good arms.
Anyone who is big and bulky may have to be careful and dressing somewhere in the middle is usually best for them. Big and baggy never looks good especially in a formal or social setting. Guys with large guts even if they are huge will just look plain bad in a tightly fitted top. So if you want to look your best for others you have to dress according to your exact body shape/type. Now for the gym or anywhere else I just wear big hoodies and cover up at all times.