Cultural appropriation is a form of colonialism
When the dominant culture appropriates a minority culture, it is exerting its power and privilege over the other culture. Stealing elements of the minority culture and incorporating it into the majority undermines their unique cultural differences and is a form of neo-colonization.
Non-whites have a history of being oppressed through their fashion choices. Throughout history, colonizers have oppressed native minorities through the control of clothing and dress. As a result, when members of the dominant European culture steal elements of minority cultures with a history of oppression, they are engaging in a form of neo-colonialism. In taking elements of their dress and incorporating them into the majority culture, they are robbing minorities of elements of their unique cultural identity. If a white British female chose to wear a sari out to a bar, she would be appropriating a minority culture that suffered a great deal of oppression at the hands of their British colonizers. In appropriating that culture and attempting to bring it into the dominant culture, the white British female is taking an element of cultural expression from that minority culture through which it identifies itself. In the same way, if that same British female chooses to wear a Lederhosen, the Germanic traditional dress, it would not be an example of cultural appropriation. Germanic people have not been oppressed, particularly not through control of their dress. Therefore, their culture is not as vulnerable to appropriation. 
Power balances throughout history are not static. They ebb and flow. What is now a power imbalance has not always been so. For example, when Kezia Daum wore a Chinese prom dress to her school dance in Utah, she was accused of cultural appropriation because as a member of the dominant race/culture (white American), she was appropriating a historical minority culture (Chinese). However, this outlook fails to consider the fact that the original design for the Chinese dress originated in the Manchurian conquest of China in 1648. The Manchurian rulers in Beijing forced native Chinese to adopt Manchurian styles of dress (like the one worn by Kezia). When considered in this historical light, Kezia is not borrowing from a minority culture, but a dominant historical culture. 
[P1] Dominant cultures exert their power over minority cultures when they appropriate their culture. [P2] Therefore, cultural appropriation is an extension of colonialist behaviour.
Rejecting the premises
[Rejecting P1] The interaction between dominant and minority cultures is not straightforward.