It's much more complex than that. I'd really like to blame SOMEONE but a lot of what has happened is the combination of many thousands of little decisions made by ordinary consumers and low level corporate folks who are just trying to get by.
As an example: I spent most of my non-military career in the aerospace and defense sector and saw what I'm about to describe happen on a continuing basis.
For about five years, I worked as a Program Manger on the PHALANX CIWS program which was a key Navy weapons program that had a very demanding schedule. The Customer (USN) had two suppliers (Prime Contractors) who they pitted against each other by price with the winner getting the lions share of production and the loser getting just enough to keep the lights on. When my company first qualified as a producer, we made all of our wire harnesses and circuit cards in house as did our competitor. The next contract, we sharpened our pencils and moved the wire harnesses to a lower cost subcontractor in a Arkansas while they moved harnesses to a special opportunity zone on an indian reservation. this went on and on and by the time I left, our circuit cards were being made at a subcontractor in Mexico (paying $2 per hour) and wire harnesses at a subsidized manufacturer on Vieques Island while our competitor puled their entire operation out of Southern California and moved it to a much lower cost state.
This was not driven (directly) by our CEO or stockholders (some of who were the elites) but instead by our desire to stay in business because if we couldn't get our price below "theirs", we'd all be out of work. By the way, the end state that I described was our final competition where the Navy decided to have a "winner take all" type shootout which the competitor won so the entire workforce that I'd worked with eventually went to zero.
.
Generally with regard to consumer goods like clothing, the cost of whatever labor is involved pretty much dictates it being outsourced to offshore and this is becoming even more evident now with a sharply rising minimum wage. I'm not getting political here and understand people's need to make enough money to function but just understand that a $15 minimum wage is competing with places that are much much lower and even IF there is greater productivity/quality for the $15, the sharp difference in rates will push the work to cheaper places. That is before environmental regulations, costs of power, OSHA concerns, and other factors kick in which are not things that help us be price competitive. The US still has a lot of manufacturing, generally of high tech/complex stuff like aircraft, medical equipment and other industrial goods but consumer level stuff is just too price competitive for production to be done in a high labor cost place like the US or Western Europe.