JUST speculating...
I personally love milk, used to drink raw milk all the time but it was too far to get and too expensive.
I'm JUST speculating... I wonder if for SOME people, there is an immune-type generalized inflammatory response? Think of your small intestine as a screen door, it allows certain sized particles to pass through, but if there is a problem with the small intestine, like a screen door having a hole in it, it can allow larger particles to pass through, if this happens, the particle can be seen as "foreign" b/c it's not something normally found in the blood. Now, if someone is sensitive to milk proteins and they can pass through the small intestine not fully digested, I wonder if some can get that "smooth" look some get concerned about. Dunno...
NOT saying this is how it, just throwing some idea's out there.