you are correct. You can go out on Friday - but in the past, 4th class cadets have been required to report back Friday night. We always stayed in the Manitou Springs area and just hung out. Our family had vacationed to Colorado before, so most of the tourist type stuff was not of interest. If you do want to take in the local sites, there are plenty of things to do from Garden of the Gods, to Pikes Peak, to Cripple Creek, or even the Royal Gorge.
The downside for most 4th class cadets is the requirement that they be in their Service Dress uniform when out in public, or some form of uniform if the activity does not lend itself to Service Dress - like bike riding.
As such, some families elect to get away to Denver or Boulder. In these cases, you are still likely to run into other cadets, but the uniform requirement is generally not enforced the further you get away from campus. You could always run into a hardcore cadet that doesn't say anything in front of the parents, but let's them have it when they are back at the hill.
I have been both a 4* and a sponsor parent, so I have seen a pretty big spectrum of activities planned for that weekend. Depending on the start time of the football game, your Saturday is tough to plan because you may not get out of the stadium until mid afternoon. I would plan the weekend, but be prepared to possibly just hang out in your hotel depending on how your cadet feels. Some will just want to sleep and eat and may seem really quite as opposed to when they lived with you, while other cadets are not phased by their experiences up to that point and will behave just like they did when they lived under your roof.
You should check with your local Parents Club to see if they have any activities planned for families from your area. Some will plan dinners and give the cadets an opportunity to meet up with other folks they may know from your area that they may not have seen since getting to the Academy.
Good luck and congrats!