Our society already normalizes countless types of violence, from police brutality to domestic abuse to constant warring overseas. Even violent video games and slasher films are insidious examples of this effect. If these instances of normalized violence have not degraded the character of our society, there is no reason to assume punching Nazis would be the final nail in the coffin. If anything, this act of aggression is more justified and virtuous than the other examples in society today.
Further, this argument relies on the slippery-slope fallacy, implying that if we consider punching Nazis acceptable, our culture will instantly fall into anarchy, with people beating each other up for no reason in the streets. Needless to say, this massive jump is an exaggeration of the real effect Nazi-punching has on our society.