The real problem lies with the modern use of diesel cars, they are simply not designed to be used predomantly in town or for short runs
Thta's not quite true. They are designed with this in mind, but you will get into trouble if you only do really really short runs. Or if you sit idling for ages If you drive around town all day you should be fine.
One cause of this is that a regen (which is when you see all the soot blow out) uses a liquid, if that runs out you get no warning until you end up in limp mode.
Only on some cars. If the DPF isn't hot enough to burn off the soot, the car needs to make the exhaust hotter. Some cars (VAG) do this by squirting fuel in the exhaust stroke, and a catalyst combusts these unburned hydrocarbons to increase the temperature enough to then burn the soot out of the DPF. Other cars (Peugot and Ford/Mazda) need this additive to do the job instead, and the catalyst is integrated into the filter.
My 407 went from a 48 mpg average to 60 mpg once the dpf and egr were modified
It'll also be producing loads more NOx too, which is a harmful irritant and causes smog. Nasty stuff.
I suspect that a modern diesel, even after having a DPF removed, is a lot cleaner than most 10+ year old diesels that I see chugging around with clouds of smoke around them
Probably not, it's probably the same. The engines work in the exact same way - squirt diesel into hot air - so if they are running at low load the results will be the same.
If you go back 20 years then injector pressures were lower and fuel atomisation wasn't as good, so at higher loads they produced more smoke, but under normal driving there's plenty of air in there so the fuel droplets combust more or less completely. That's why older diesels originally had much bigger displacement than equivalent petrol engines - more air to spread the fuel around in = less smoke.
If you floor a diesel from low revs there's a moment where the turbo isn't spooled up, so the ECU (to improve driveability) dumps extra fuel in there which doesn't burn completely, but still burns a bit to get the turbo going. This results in the puff of smoke you used to see. And I suspect that eco label diesels don't do this, which is why the reviews said that they had less low-end punch.