Not only should you do the right thing and comply with this request for the sake of the site owner, but it's also in your own interest to do so.
If you have multiple links to his site, it's more than likely you have multiple links to other sites. If Google has alerted him, then it's likely to have alerted others. And if Google notices this pattern of linking, it will regard your site as nothing more than a link farm, and you may find you are also being penalised. So it's in your own interest to sort out the situation swiftly, before it reaches this stage.