I can’t imagine anyone having any real justification for blaming you for them clicking the wrong link – UNLESS, as @swader infers, that your unsub link is particularly small and/or close to another link they use.
That’s a solvable layout issue.
Double opt-out always feels a bit insulting to me. Why not show them an Unsub confirmation screen, but give them a ‘I made a mistake - Re-subscribe me’ button?