It depends on the contractual relationship between voter and representative.
If you are elected as a delegate then you are mandated to follow the wishes and preferences, 'the expressed will', of the electors. You have no status as an independent thinker and discriminator. If the expressed will is unalterable by argument, and to carry it out would violate your conscience, then you should resign. You are reneging on your 'previous sworn duty'.
In contrast if you are a representative then the electorate elects your judgement and conscience into the contract. It is then perfectly proper to act as an independent thinker and discriminator - and to take the consequences at the next election.
What I am echoing here is the Burkean view of representation, after Edmund Burke (1729 - 1797), the Irish politician and philosopher. When Burke was elected to represent the English city of Bristol in 1774 he addressed them as follows :
Certainly, gentlemen, it ought to be the happiness and glory of a representative to live in the strictest union, the closest correspondence, and the most unreserved communication with his constituents. Their wishes ought to have great weight with him; their opinion, high respect; their business, unremitted attention. It is his duty to sacrifice his repose, his pleasures, his satisfactions, to theirs; and above all, ever, and in all cases, to prefer their interest to his own. But his unbiassed opinion, his mature judgment, his enlightened conscience, he ought not to sacrifice to you, to any man, or to any set of men living. These he does not derive from your pleasure; no, nor from the law and the constitution. They are a trust from Providence, for the abuse of which he is deeply answerable. Your representative owes you, not his industry only, but his judgment; and he betrays, instead of serving you, if he sacrifices it to your opinion. (http://peter-moore.co.uk/blog/edmund-burke-speech-to-the-electors-of-bristol-1774)
Elitist language but the British Constitution still works essentially on this view of representation as regards Members of Parliament. It can also produce good results as when parliamentary representatives pass (at least occasionally) humanitarian legislation with which their electors do not agree.
This view of representation is consistent with the self-certainty of, 'I know better that X, which you [the electors] support, is unethical', but its more normal and less self-righteous, less dramatic version is just, 'I believe that X is wrong and on reflection can't support it - and you elected me to use my judgment'. In this case you are precisely acting on your 'previous sworn duty'. That duty is to judge independently of the electors when you consider it appropriate to do so.