I think government contractors can stay at the rec center, they just can't be the one renting the facility. Anyone with a CAC can get on the base.
Anyone can stay as a guest of someone authorized to rent, Military active/reserve/retired, DOD employees active/retired, and I believe other government employees.
Both of you are correct. The only hitch is, if you do not have a DOD ID card (or similar), someone has to sign you on the base each time.
That means that if a civilian is staying there and they want to run back to pick something up, they have to be escorted through the gate.