As a point of fact, regardless of whether one believes neo-Nazis should have the right to express their deplorable beliefs, it has long been settled in the United States that they have the right to do so in a non-violent fashion.
They were calling for violent reprisals against Jews and African Americans.
Per Brandenburg v. Ohio, the government cannot curtail speech unless it is "directed to inciting or producing imminent lawless action and is likely to incite or produce such action".
I think the speech in Charlottesville easily passed that test. The mob did, in fact, mob counter protesters and toss tiki torches, attacked other counter protesters the next day, and we also had the death of one of the counter protesters.
I think, especially post the Charlottesville events, such a rally, with its very specific messaging, its very specific beliefs, can and should be seen as literally aiming to incite the violence they call for in their speech. The violence the preach is imminent, as they clearly proved.