Who signed the letter should not matter as much as what the letter said. The fact that some controversial figures like Ian Buruma and J.K. Rowling signed the letter does not mean that its messages of free speech and liberalism aren't legitimate. Regardless of its signatories, it is still important to discuss how cancel culture has affected people's ability to speak and express their opinions. To oppose a worthy message simply because of the identity of its signatories only feeds into people's complaints about identity politics.
The letter was published as part of a long-running discussion on speech and freedoms with the author and various historians and academics. The timing of the letter and the recent rise of the Black Lives Matter movement was coincidental. Not all signatories came from privileged and wealthy backgrounds. Its primary writer, Thomas Chatterton Williams, is African American himself, and the letter was deliberately circulated to collect signatures from a more diverse range of people. The signatories included leading Black intellectuals, such as Nell Irvin Painter, John McWhorter, and Reginald Dwayne Betts, as well as many LGBTQ+ academics.
Some signatories may have been controversial, but many were also qualified to speak on the subject. There is no point making this a fight over who should be able to talk about what.