Many OS lounges will not let you in without the card so it is important to carry it. Domestically and for QF lounges no issue but others can be painful in their application of the rules.
Particularly in the US as some lounges want to scan the actual card to verify eligibility to use the lounge plus it may also be a way that lounge bills the other airline (that you are a frequent flyer member of) for your lounge visit.
I guess the same applies to priority checkin/boarding, so the only benefits I might loose would be additional luggage / and potentially seat selection
Additional baggage. We booked an US Air flight via their website for a codeshare flight that was UA metal. Although members of US DM we're just run of the mill members so don't hold elite status with US & those ff no's were in the pnr.
When we checked in with UA at LAS for our flight to LAX we showed our TK *A Gold memberships cards in order to get the free luggage however the CSA had to remove the (no status) US DM ff no's & enter in our TK ff no's so we wouldn't be charged for the checked baggage. Once the bag tags were issued she put our US DM numbers back in.
At the time of booking I'd paid about $9.00 extra pp for the mileage multiplier to get triple miles so I thought the removal of the & subsequent re-entering of the US DM ffn may have stuffed that up but the points still credited correctly.
Priority Boarding. If you are travelling in a premium cabin eg J/F you would get more priority than if travelling in whY with no status however whether or not you receive PB when an elite ffn is absent from that boarding pass will be up to the individual who is doing the boarding for the airline on the day you're travelling.
For instance when flying whY on AA & crediting to QF (QF ffn in pnr & therefore on boarding pass) if you're a QF WP you'll have the wording "Priority AAccess" stamped on the face of your boarding pass & therefore be eligible to use the PB lane and board early.
If you were to credit to a program where you didn't hold elite status & your boarding pass did not have "Priority AAccess" but "group 4" on it you may get knocked back for early boarding & be told to wait until your group is boarding. Showing an elite card with another airline may not work, however sometimes QF WP cards get confused with AA Plat or AA EXP cards because they look the same so you may get let through. Definitely a YMMV scenario.