I'm assuming you don't have any sort of written agreement? Do you have any communication (like email) that describes the project and the client agreeing to the project?
If you have some written proof that the client accepted specific parts of the project (like the domain registration, hosting, etc) then I do not see a problem informing the client that there are parts of the project that have already been started and will need to be taken out of the deposit check. If you registered their domain and paid for a month of hosting, those are owned by the client and thus are warranted to be paid.
However, this doesn't mean that if they "agreed" via email on the project that you are entitled to the full amount of the project. If you haven't started on the design or any other aspect of it, then you should refund the client the full amount as you have not completed any work to warrant payment. Sure, it is a project you were anticipating working on, but if they back out, just return their deposit and move on to a client that actually wants to work with you.
If you have started on the design, you can try to work it out with the client that you already started and that they owe you for the time you put in so far ... but that is going to be a tough sell to most clients. I would only do so if it were a lot of time into a project (of course I have written agreements that explain what would happen if they backed out so it's all covered anyway).
Personally I would deduct any physical costs for the project (domain/hosting) and refund the rest. I doubt it would be worth your time or the hassle to try to keep anything else. But that's just my take on it.