Mozilla based browsers support adding modal=yes to the third argument in a window.open and that will keep the opened window in front of the one that opened it (but doesn’t really force it to be modal). There is nothing that will get other browsers to even do this much.
The solution is to open an overlay in the same window that stops the rest of the page being accessed until it is closed.