I think in the OP's situation the FAs should monitor and if practicable, prevent the seat-swapping-into-exit-row-seats.
1) Exit rows are a (slightly) premium product attracting a higher price on nearly every airline these days, so its a similar situation to the fact that you can't seat-swap Y for PE nor Y or PE for J; AND
2) In theory, at an exit row seat you have to pass the FA's pre-take-off "Are you willing and able to undertake those (emergency situation) duties?" test. I think you actually have to positively reply with the answer "Yes" to that question. Seat jumpers obviously haven't. If the answer to the pre-flight question isn't important, why ask it?
This was my thinking exactly. Unlike any other seat in Y, you pay up to $180 extra for the additional legroom. If any unallocated seat is fair game, why not just jump into the the available F or J seat - same principle.
As rooflyer also pointed out, there are certain requirements to sit in the exit row, which is why you cant confirm the seats online, or have your 12 y.o. sitting with you in the exit row. I agree, that if the pre flight requirements arent enforced, why ask.
As I said earlier, I don't have a problem with people changing seats in general.