You can get away with faster pans or tracking shots as long as the subject in the frame is relatively stable in the shot - i.e. a person running stays in roughly the same screen area as the camera pans with them - because the viewer will focus on them and be less aware of the judder in the background. It's really only distracting to the viewer when everything in the frame is moving at once like your sample shot. For quick pans in a fast cut/moving sequence it's not going to be too noticeable either, it's worst when you force the audience to watch a single fast panning shot for too long.
But again - this is normal. It's present in pretty much every feature film to hit theaters. It's more noticeable on the big screen than on a television, but if you're careful to shoot around it (based on the guidelines in the ASC reference jax mentioned) people just won't notice.
If you play the same clip over and over for your friends asking if they notice it then yes, everyone's going to notice it - in fact, they probably won't be able to un-see it, and you've now ruined that shot for them. Which brings us to another important rule - if there's something about a shot or your film which bothers you, never ever ever point it out to the audience. Chances are they won't even notice it, but once you've told them to look for it they probably can't miss it.