Could the impact of a sizable asteroid or Comet change the Earth's distance from the sun?

No asteroid or comet has enough m to change Earth's orbit in any appreciable way. For example, asteroid 433 Eros (the largest asteroid that crosses Earth's orbit) has a m of about 1 billionth that of Earth. That means that If it were to hit the Earth head-on (which it could never actually do, because Eros and Earth orbit in the same direction), Earth would lose about 1 billionth of its momentum. That would move the Earth only 1 km closer to the Sun.