To say that Black Lives Matter is not to say that all non-Black lives do not matter. The crucial idea here is that the movement sheds light on the racism that Black people experience in virtue of being Black, which has endangered many people on a global scale. 
Removing statues reflects an important part of the current protests, which is acknowledging how the legacy of the British empire has contributed to inequality within our actual society. Thus, the removal of the statues is an attack on inequality, racism, and essentially the beliefs of the British empire, such as Black people being a lesser kind of human being. An ‘All Lives Matter’ approach to the Black Lives Matter movement diminishes the harsh reality that the anti-racism movement seeks to communicate.