We have been twice, once without kids (about 15 years ago) and once with kids (3 years ago).
Both great holidays.
We stayed in Negril both times. The first time we stayed on the West End Road (Negril). A bit out of the way but quite chilled, we stayed in a small room only type place and ate at very small shacks/restaurants.
Recently we went back with the kids and decided all inclusive was probably a better option with the nippers. We stayed at Rui on the beach at Negril, another fantastic holiday.
I would highly recommend Orane at Antsman Tours (link) if you go. We hired him for a couple of days and gave him a list of all of the places we wanted to visit and he arranged two very memorable days.
There are places to avoid but stick to the tourist areas and you will be fine. Use someone like Orane if you want to explore (much cheaper than hotel/tour operator trips).
The first time we went I ended up playing football and having a few beers with one of the locals I met at his village. A very friendly country and I will go back again at some point.
Avoid Spring Break as it will be full of pissed up American college types!