Another option for removing rust stains from quartz countertops is using a baking soda paste, which Little says is non-abrasive and can safely be applied to quartz.