You are right – and that is why there are those who hold to a view called “Historical Preterism.” They believe that all end-time prophecy was fulfilled around AD 70. The problem is that the questions the disciples asked were lumped together and the stones being removed was only one question. The main question was when Jesus would return. This cannot be answered by Historical Preterism – unless you and I missed Christ’s return!
So, the choices are that Jesus was separately answering two separate questions, or that Jesus was answering the disciples main question of what will be the sign of your coming – the stones being thrown down was relatively unimportant. This second choice is what I believe.