Worries that long-term U.S. growth will fade could matter more for the currency than the mechanical impact of tariffs.