It’s all about the luggage really, what you want to carry etc. the rack is a small weight in comparison, especially if you don’t go for a monster, something light like a Tubus Fly is a good option (or Airy if you’ve got the cash!), it’s light, quite low profile and versatile as can take normal pair of panniers, or a smaller drybag strapped on the top.
Small saddle bags are fine for little loads, but if you find you outgrow it then you’ll be into the realms of bigger drybag style stuff anyway and at that point it’s sometimes nicer to have the option of a rack, especially if you might be using panniers on other days during the week.
Don’t rule out a small porteur style rack and bag at the front, can be very versatile and easier access while on the move, depending on the frame geometry and especially front end/trail it will have very little effect on handling, and in some cases can improve things.