I had a very similar habit of misplacing passports until I also got a travel wallet. A lot bulkier and harder to forget, but still need to force my self to put the passport back in the wallet and not in my back pocket after clearing customs! Now also making sure the wallet and passport are in my bag before take-off.Yes, I think it's time to invest in one of these!
I can take comfort in knowing that I'm not the only person to have ever done this. While I was at the airport police office, I noticed that they had a list of around a dozen passports that had been handed in to them and had not yet been collected.
Edit: I have now ordered one of the Bellroy travel wallets.