I'd expect both of them to lower IGF-I, though I'm not aware of any research on them. Most anti-estrogens, including arimidex and letrozole, have been shown to lower IGF-I levels.
If you're interested, DatBTrue has said "it would seem that male bodybuilders who use low dose Tamox in a PCT need not be concerned about inhibition [of IGF-1 production]." The endocrinologist Swale then stated "I wish this simple point would spread across the Boards to dispel this widely accepted nonsense."
Personally, I wouldn't be worrying about IGF-I and the use of an anti-estrogen. If you need an anti-estrogen, then you need one. Just use it.