I never do...unless I am close to max weight. I like the fact that my lower back is strong and can handle the weight that I squat or shrug. I had a herniated disc back in the day and I wore one then, but when it all healed, I slowly got off the use of the belt to keep my lower back strong.
If you have an injury, I recommend one when you get to 50% of your max, but when you are handling the lighter weights to warm-up, it is good to get those muscles strong again and not rely on belt for the support...to me I feel that the back will not get stronger wearing a belt for all movements.