Yes sometimes you will be told to take it off. But not every time. And every time you don't get told to take it off is time and inconvenience saved.
My old belt had a small buckle and I never had to remove it flying domestically, nor at most airports overseas. It did still cause issues at...