Your integrity could be called into question. Though this might make you angry, you also know that there's no question that you've behaved impeccably. The bigger question is why someone would suggest otherwise. It could be that you're caught in the flack of someone else's argument. At a different (exam) level, you might be quite happy to have your expertise subjected to analysis.