Decathlon do some reasonable bivvy kit. I use a decathlon Tarp that did need a bit of additional work to customise it as it didnt have enough eyelets. Decathlon also do a bivvy bag too though I have no idea of how well its made.
The alpkit gear is good, all seems to be of the no-frills but works very well variety. Hunka bivvy bags are good value, well made, pack down small and are highly reccomended. The sleeping bags are good too.
Small stoves are probably the big issue, I use a Primus Eta packlite stove that is flippin brilliant but maybe a little bit pricy if you are just starting out bivvying, MSR pocket rockets are cheaper and can be got for about £20-£30 from Millets (a bit of a faff if its windy) just avoid the cheapo solid fuel stoves as they are, frankly, rubbish.
Otherwise, get a Hunka bivvy bag and any ok-ish sleeping bag, some good warm clothing a big thermos of tea or soup and a clear weather forecast and you're sorted