I don’t know what your interests as a family are. Richmond was voted Best River City in the USA last year for it’s river sports. It’s got a hip edge to it, a great state park on the edge. I think it’s the most tattooed city in the USA as well!
Do take the weather into your plans. The East coast gets hot and humid in the summer and that sticky heat can be unbearable and just saps your energy. I’m not a fan of the heat but the other 3 seasons make up for it here. There are plenty things to do in Richmond (mountain biking, white water rafting, SUP, kayaking etc) and that’s all downtown! There are festivals every weekend, Virginia Beach is 100 miles away, DC is 100 miles North of here (with viable trains as an option) and the mountains are 100 miles away.
http://www.visitrichmondva.com/
In summary – a great city to visit, loads to see and do, but I would not recommend that visit in the middle of summer.