While it not a movie I particularly cared for (I may be alone in the opinion), there is a quote from Forrest Gump that come to mind – “Mama always said life was like a box of chocolates. You never know what you're gonna get.” https://www.rottentomatoes.com/m/forrest_gump/quotes/
It isn’t clear from your question if you are talking about physical rigor or academic rigor separately, or just the rigor of the sundry institutions. If your circumstances permit I’d urge you to visit the institutions you are interested in. They have similarities and significant differences and no one institution is the best choice for every person. Our son’s best friend is about to graduate from VMI and it was the right choice for him. Small classes, demanding academics, he’s done well as a math major.
The point about small classes and no teaching assistants is well made and something to consider. What are your strengths? One our Scouts also went to VMI and did well early on but as the ratline disappeared and the “aca-line” (or words to the effect) came into play it was harder for him to stay focused. There is more structure at A&M in the first two years of Corps-life there based on what our two kids have me than in the last two – and again, some have a tendency to fall away from the academic pursuits with the addition opportunity for self-directed diversions. That said, many of the A&M folks are in heavy math, science, engineering and related pursuits and do well. A&M and Virginia Tech have tremendous researched-based programs in those disciplines.
I guess I’m trying to say you never know what you are going to get at a particular school, it is all a bit of something-shoot, and it would behoove you if possible to visit the schools to see both the Corps life AND the academic majors that you are interested in. Seriously, those are demanding majors regardless of the school you choose – choose the school you feel comfortable in and the rest should, along with a lot of doses of coffee, take care of itself. There is that old saw – you can do any two of these three things – do well in school, have a life outside of the classroom, and get enough sleep. Good luck and best wishes.