The main problem with going to the cheap end of the market is that the kit is either heavy, bulky or both. There are exceptions but you need to search for them.
Sleeping mat. Cheapest would be a closed cell foam mat (Karrimat or similar), main problem is that they are bulky with little/no option to get them packed down smaller. Having said that I took one on the Rovaniemi race and simply rolled it around the bag holding my sleeping bag and managed to get it mounted in a Wildcat Lion handlebar harness, this was on a fat bike so didn’t look too silly. Advantages of a CCF mat are that they don’t get punctures. There are uninsulated inflatable mats for around £20 which are fine for summer and late spring/early autumn use. My wife has one and it’s been fine. I’ve a Klymit X-frame which is similarly lacking in insulation but a lot lighter. The insulated mats come in at £70/£80 and upwards.
Sleeping bag: Look at Cumulus (the bags are branded as Criterion here in the UK), excellent value down bags and quilts from Poland. I’ve the 150 quilt which is fine for most of the year then during the colder months I use it as an over-blanket/wrapper for my summer down bag.
Shelter: Even though you say no Alpkit, the Rig3.5 tarp is decent and well thought out (as far as a sheet of silnylon ever can be). If you do use a tarp then the bivy bag only needs to be lightly water/shower proof so something like the Rab Survival zone should be fine.
