Personally, it's all about being comfortable. On long haul flights you may end up sleeping in those clothes depending on the availability of PJs, so I prefer to be as comfortable as possible. I also always have a warm jacket as planes, airports, and hotels are sometimes freezing even in summer...