I use factory ammo while CC in the city, running errands, ect. While hunting, I usually have handloads in my sidearms.
I don't think the bad guy is going to wait for me to change over from handloads to factory loads; I will use what I have available.