BOSTON (CBS/AP) — One of the planned speakers of the “Free Speech Rally” on Boston Common said the event “fell apart.”

But it lasted long enough to deliver a message that was missing any signs of racism.

READ MORE: Weekend Reservations Needed At Brookline Christmas Tree Farm Due To Demand

About 40,000 demonstrators chanting anti-Nazi slogans converged Saturday on downtown Boston in a boisterous repudiation of white nationalism, dwarfing a small group of conservatives who cut short their planned “free speech rally” a week after a gathering of hate groups led to bloodshed in Virginia.

Related: Some Counter Protesters Scuffle With Police After Boston ‘Free Speech Rally’

The counterprotesters marched through the city to historic Boston Common, where many gathered near a bandstand abandoned early by conservatives who had planned to deliver a series of speeches.

An aerial view of protesters on Boston Common demonstrating against a so-called ‘free speech’ rally on the Parkman Bandstand Saturday, August 19, 2017. (WBZ-TV)

Police vans later escorted the conservatives out of the area, and angry counterprotesters scuffled with armed officers trying to maintain order.

Photos: Boston Free Speech Rally And Counter Protests

Organizers of the midday event, billed as a “Free Speech Rally,” have publicly distanced themselves from the neo-Nazis, white supremacists and others who fomented violence in Charlottesville on Aug. 12.

READ MORE: POLL: Which Will Be The Toughest Of The Patriots 4 Remaining Games?

The ‘free speech’ rally was confined to the Parkman Bandstand on Boston Common Saturday as barriers and police held back a massive protest. (WBZ-TV)

TV cameras showed a group of boisterous counterprotesters on the Common chasing a man with a Trump campaign banner and cap, shouting and swearing at him.

Read: President Trump Chides ‘Anti-Police Agitators’

But other counterprotesters intervened and helped the man safely over a fence into the area where the conservative rally was to be staged. Black-clad counterprotesters also grabbed an American flag out of an elderly woman’s hands, and she stumbled and fell to the ground.

Thousands of protesters march in Boston against a ‘Free Speech Rally’ on Boston Common, August 19, 2017. (Photo by Spencer Platt/Getty Images)

Yet Saturday’s showdown was mostly peaceable, and after demonstrators dispersed, a picnic atmosphere took over with stragglers tossing beach balls, banging on bongo drums and playing reggae music.

The Boston Free Speech Coalition, which organized the event, said it has nothing to do with white nationalism or racism and its group is not affiliated with the Charlottesville rally organizers in any way.

“We are strictly about free speech,” the group said on its Facebook page. “We denounce the politics of supremacy and violence.”

Rallies also were planned in cities across the country, including Dallas, Atlanta and New Orleans.

MORE NEWS: Patriots Beat Bills 14-10, Remain Top Seed In AFC

(© Copyright 2017 CBS Broadcasting Inc. All Rights Reserved. The Associated Press contributed to this report.)