I found a part of the interview that may shed some light on the "scrap everything and restart fresh" decision. Maybe it wa s not as simple as "letīs use XPTO engine, it will look better!":
One of the main reasons people enjoyed Duke3 so much was the level of interactivity. (Blowing up monitors, shaking boobs) Will you be targeting the same audience with Duke4? Do you find it easier or more difficult to do this with the Quake2 engine?
God yes. It has been utterly painful to make the Quake stuff as interactive as we want. But it's not really Quake's fault, it's just what we want out of the game. Quake is awesome if you want to do one thing. Make Quake games along similar lines and constraints.
If you want to move much beyond that,things start breaking hard. We have a running gag here of hitting the "Limit of the Day"
We've hit max brush limits, memory limits on bsp's, entities, brush planes, func walls...you name it, we've broken it. But we do extend and expand the engine in every case we can. Some are as easy as a #define in the code and others will take 3-4 weeks to do. In the end, I suspect we'll use Quake as a rendering and net engine, and everything else will be gone.
There are a zillion little things that happen, for instance: You add masking textures to the game (for fences, grates etc). Great. Except they don't take light, so you have to mod the engine to do that. The water doesn't take light, so that has to be added. It's just a never ending battle of getting the engine to do what you want it to. But we're getting pretty good at wrangling Quake