I think we COULD say "Ok, I accept that you may not have meant to do harm- but here's why you managed to, anyway and why you need to not do that again". Seems to met that would be just as effective as it currently is to treat a white person who did inadvertent harm as if they did it knowingly and on purpose, if not more so.