A plain bivy bag of any description while it may keep you dry is basically just a plastic/nylon bag so won’t do very much for insulation. If it rains then the material will be pressed against you and you will cool to the external temperature – you’ll be dry but cool/cold. Imagine wearing just a waterproof top when it rains – that’s effectively what you are doing.
Similarly regarding a sleeping mat: 80-100Kg of flesh vs several trillion, trillion tonnes of rock and earth isn’t a fair contest. A sleeping mat of some description goes some way to lowering the heat loss via that channel, in fact gramme for gramme they are more effective than a sleeping bag. The modern inflatable mats are effective, light and compact when packed away. The X-frame is 240g and about a litre volume when packed but sod all R-value, something like the Exped Winterlite is 400g, only a little bigger and has an R-value of 6.
Getting in to something dry before getting in to the bivy bag helps as well – hence the thermals and down vest. Try and get bedded down whilst your body is warm but not sweaty (a micro-fibre towel to dry you down is also a good idea) so you aren’t having to warm everything up once you are tucked in. Thermals or silk liner, both do the same job but the thermals can be used in other situations so for me that’s a win.
No one right answer, one night you could be fine and pleasant the next you could be miserable.