Soot isn’t really the problem with diesels, it is NOx – London and many other UK cities are way over the European legislation pollution limits.
The tailpipe NOx limits are pretty low for both diesel and petrol engines, but the test cycle is not representative of a real drive cycle (which is why it is rare to get near the fuel economy figures).
A petrol, however it is driven has low level of NOx emissions. A diesel however produces huge amounts of NOx at higher loads – these regions are not tested.
Later in this decade more realistic drive cycles will be introduced, along with Urea/adblue based SCR systems on many diesels which reduce the NOx emitted from the tailpipe.
Truck/bus engines are tested in a different manner, which is why for Eu6 they all have SCR systems already.