I was bored, so I went on an instruction dumpster dive. I also hate it when recruiters mislead high school kids...I get they have to "meet mission," but it's distasteful and unethical at times.
NSTC-M 1533.2D, Regulations for Officer Development, links CNRCINST 1533.4E via an NNOA link rather than an official NETC or NSTC link...not sure why the instruction isn't on an official website.
CNRCINST 1533.4E details much more on how ISR works (if it is the current instruction). Per instruction organizations like ANSO, NNOA, and NOMA also have ISR allocations in addition to Navy Recruiting Command.
More importantly:
"6. [...] It is desirable to grant all allocated ISRs; however, this should occur only where the intent of the program is met. Failure to grant all allocated ISRs simply results in the awarding of additional Navy National Four-Year scholarships."
I read that as "don't feel bad for ditching your ISR for USNA, big Navy will reallocate those resources to another scholarship."
Additionally, NAVCRUIT form 1100/32, NROTC 4-Year ISR Contract also states: "I also understand this contract places me under no obligation to accept an NROTC scholarship."
Likely your recruiting region will "lose" the allocation (which might be why the recruiter is making the stakes seem so dire--it's fake), but I wouldn't feel bad for the recruiters. It's your path to take and your future, not theirs.