There are a lot of slippery slope arguments going on about statue removal - "Andrew Jackson today, who's next, George Washington?" kind of thing.
Suggestion - Israel has an honorific of "Righteous Among the Nations" to identify non-Jews who significantly contributed to save Jews during the Nazi era. One of the most famous of these is probably Oskar Schindler, who was a member of the Nazi Party, but saved over 1,200 Jews during the war. His Nazi status did not disqualify him - his actions on behalf of his workers spoke louder than his offenses.
Perhaps there can be some kind of summit of People of Color who are historians, educators and elder statesmen to make some similar distinctions to advocate which memorials are to be maintained and which are to be altered. The imprimatur of individuals with this moral authority could help guide our reshaping of the public historical tributes.