Ask at the UA desk on check in if they can check your bag all the way to SYD. If the flights are all on the one ticket it shouldn't be a problem. QF have however said they are not going to play friendly with non-OW carriers if not all on the one PNR (effectively ticket), so you may be out of luck. (I guess it depends on how UA view the situation)
US airports don't have outbound customs, therefore all you would need to do is go through security. Passport checks are done on boarding.
FWIW, with a 12 hour layover in LAX, I'd look at leaving the airport area. LAX is hardly a shining example of an excellent airport, and even with lounge access it will get tedious very quickly. Lots of people rent cars and go for a bit of an explore, that's probably what I'd do.
Just keep in mind the car rental places at LAX are actually a little bit away from the airport, so you'll need to give yourself time to catch the bus, that said the buses are free and provided by the car rental places with their branding / logos all over them.