Fundamentally, this question rests on whether you think that Nazism and related philosophies are superior philosophies that will inevitably peacefully gain supporters until stopped by superior force -- or, alternatively, whether the line is from "people saying stupid shit" to "people engaged in active planning of violence, harassment, and other criminal activity" and where in that it's reasonable to stop someone by force.