Definitely worth getting it checked out if it seems to come and go. I had what I believed to be a deeply-buried arse spot that would grow and decrease in size depending on how much riding I was doing.
When it grew, I'd give a squeeze in the hope of a satisfying burst of pus, but it never yielded anything until the day came when jets of watery blood accompanied the squeezing. Cue me hovering over a mirror placed on the floor, feeling rather concerned, with a view of my backeye that I'd prefer not to have ever seen.
At this point I elected to get involved with medical professionals, and ultimately ended up being diagnosed with a sebaceous cyst, which required surgery. Rather painful surgery. I'd let it fester for so long that, rather than stab open the mound and let the goo run out, the surgeon actually had to scrape out the damaged flesh inside and leave me with an auxiliary bumhole.
To be honest, the recovery wasn't too bad. What wasn't so good was the embarrassment of having a nurse turn up to my house the day after the operation, draw the curtains, make me drop my kecks and then insert the packing into my second ringpiece, all whilst trying to make polite conversation.