I don't know if it's possible to stop Netbeans from opening a new window on closing xdebug session but it's possible to stop Netbeans from opening any browser windows, both on starting and stopping the session. Then you would need a browser extension to handle this for you. I use Easy Xdebug in my Seamonkey (this is actually a Firefox extension) and it works well. You have to configure Netbeans not to open any browser windows so they don't bother you any more (see instructions on easy xdebug page).
Debugging now works like this for me:
In Netbeans I press the button to start debugging - and then no new pages are opened in my browser, which I like because most of the time I already have my web application open so I don't need the additional clutter. Also, Netbeans by default always opens the home page (configured in project settings), which doesn't make sense since I usually want to debug a different page.
My browser still functions normally, that is the pages are being opened without debugging. It is only when I enable xdebug session by pressing the button in the browser toolbar that the debugging session starts - from then on any new page opened in the browser is debugged in Netbeans.
When ending the session I press the button in Netbeans and also stop the session in my browser.
This may seem like little improvement because I have to also start and stop debugging in my browser but overall I find it better and cleaner without all those new windows/tabs that I have to close immediately anyway. Also, it's good that Netbeans doesn't start debugging my home page every time I start the xdebug session.