In short, there are many reasons for not turning to physics for the last word on time. First, physics is itself in something of an impasse, with its two most powerful theories in conflict. As Barry Dainton has put it:
We know that our current fundamental physical theories are imperfect: quantum theory and general relativity have yet to be fully reconciled. It may well be that the theory that emerges from this eventual marriage will have very different implications for the nature of space and time than those of currently accepted theories, so it would be very short-sighted to take current scientific theories to be the last word on space and time in our universe.
Since general relativity treats physical quantities such as velocity and position as having determinate values, which quantum mechanics cannot accommodate, and quantum mechanics allows interaction between particles at faster-than-light speeds not permitted by general relativity, this is not only shortsighted but also contrary to the spirit of science.
Furthermore, it is not only unscientific but also unphilosophical to assume that any findings and theories from objective, quantitative science will settle the nature of time once and for all or that what is lost in physics of our experience and of what makes our world intelligible was well lost because illusory. To say this is not to reject science — how could any sane person deny that it is the greatest collective cognitive achievement of humanity? — but to assign it to its proper place and to rescue time from the jaws of physics and from the dropped jaws of philosophers so awed by physics as to hand over metaphysical inquiry to physicists.
more here.