Where in the QF Lounge access rules does it say "Must be departing from the same terminal as the Lounge is in". My understanding is only requirement is next flight must be QF, EK, JQ or Oneworld and doesn't specify it needing to be Intl/Dom ETC.
If the OP could get to the doors, they should be...