Albert Einstein, 1950 (Scientific American):
"[re: the general principle of relativity] ... without this ... it would be practically impossible for anybody to hit on the gravitational equations, not even by using the principle of special relativity ...
... This is why all attempts to obtain a deeper knowledge of the foundations of physics seem doomed to me unless the basic concepts are in accordance with general relativity from the beginning. This situation makes it difficult for us to use our empirical knowledge, however comprehensive, in looking for the fundamental concepts and relations of physics, and it forces us to apply free speculation to a much greater extent than is presently assumed by most physicists.
I do not see any reason to assume that the heuristic significance of the principle of general relativity is restricted to gravitation and that the rest of physics can be dealt with separately on the basis of special relativity, with the hope that later on the whole may be fitted consistently into a general relativistic scheme. I do not think that such an attitude, although historically understandable, can be objectively justified. The comparative smallness of what we know today as gravitational effects is not a conclusive reason for ignoring the principle of relativity in theoretical investigations of a fundamental character. In other words, I do not believe that it is justifiable to ask: what would physics look like without gravitation? "
Some people really don't like the implications of what Einstein seemed to be saying here about his own theories. Some online physics people in the past have insisted that Einstein couldn't possibly have said such a thing, but I've checked paperback reprints and the original 1950 SciAm publication and there it is, in black and white. That's what he said.
Einstein's point is entirely logical, and supported by the historical record. If we look at the background to the development of relativity theory, notably the Newtonian Catastrophe, we find that the version of general relativity that we ended up with, with special relativity providing an underlying flat-spacetime layer, was not just not inevitable, it actually seemed to depend on a rather unlikely-looking chain of historical accidents. Probablistically, it really shouldn't have happened like this.
We should have understood that gravity slowed time a century before Einstein eventually noticed, shortly after John Michell had predicted gravitational shifts way back in 1783. With the benefit of that missing piece of information, Gauss and Riemann and Clifford and friends ought to have been able to complete their curved-space projects (allowing curvature in four dimensions rather than just three) and should have been able to produce a general theory of relativity in the Nineteenth Century, before special relativity had been thought of. Einstein's special theory was partly a reaction against the aether models that dominated in the absence of a proper curvature-based description, and when Einstein went on to try derive his more ambitious general theory a few years later, he naturally wanted it to incorporate his earlier theory.
But special relativity assumed inertia without gravity, and energy-concentration without curvature. Its founding geometrical principles are fundamentally incompatible with key results that arise from the general principle of relativity. Like Einstein said, historically understandable, but not objectively justifiable with the benefit of hindsight.