Back in 1916, Einstein was still working to the assumption that the universe should be neat and tidy, and since he was now using a more mathematical approach, this meant "infinite and unchanging".
If you were solving the equations of general relativity, and getting solutions in which the universe appeared to be unstable, then you could throw those away. Chaos was bad. Order was good. Stability was good. Static solutions were better than dynamic ones.
Since it seems that gravitational mass is always positive, gravitational effects are cumulative, and over a large enough region, the combined background curvature should be enough to curve space right back on itself. The combined attraction also ought to be trying to make the universe contract, so we've appreciated for a while that unless there was some other effect in play, the universe should either be expanding and slowing, or collapsing in on itself (see: Erasmus Darwin, 1791).
Einstein wanted his universe to be pretty much flat at very large scales, so he got rid of the effects caused by cumulative curvature by adding an additional squiggle to the equations: an invented long-range repulsive effect whose purpose was to counteract the cumulative long-range effects of gravitation, allowing a tidy, constant, unchanging, static universe. If the rest of the equation generated long-range curvature effects and evolution over time, the upper-case Greek letter Lambda (Λ) represented the necessary compensating effect that might exactly cancel these effects.
Einstein referred to this as the Cosmological Constant.
Unfortunately, Einstein had made his model too tidy. A few years later, Edwin Hubble successfully measured a distance-dependent trend in the spectral shifts of light from a range of galaxies (Hubble shift), and we realised that the complicating large-scale effects that Einstein thought he'd eliminated with his Cosmological Constant seemed to be physically real. After taking some time to think the matter over, Einstein agreed that a Riemann-type solution (without Lambda) gave a cleaner and more natural implementation of General Relativity. He later described his early decision to invent the Constant to force large-scale flatness onto GR as "The biggest blunder of my career".
End of story.
However, the subject seemed to kick off again in the 1990's when a lot of headlines started appearing in in the popular science press (and in scientific papers) to do with the idea of dark energy, and the idea that the universe seemed to be expanding faster than GR1915 predicted – these articles usually declared that "Einstein's Cosmological Constant" was back, and had excited-sounding researchers competing to see who could give the best quote about Einstein having been "right all along".
This wasn't really true: Einstein's Cosmological Constant had been a mathematically-derived thing that only had one allowable value, and whose justification was to set the strengths of a range of effects in the model (large-scale curvature, distance-dependent redshifts, change in size over time) to zero. It had been there for purely logical reasons, in the context of a static universe, because a static universe seemed to need it. It existed to explain an assumed physical equilibrium that turned out not to exist, in a universe that wasn't ours. It was derived from bad assumptions, but at least it was derived.
The modern counterpart was almost the opposite. The antigravitational "dark energy" cosmological constant applied to an expanding universe that seemed to be expanding too fast for GR1915, and the effect initially had no fundamental logical, mathematical, geometrical or theoretical basis. It was, essentially, a parameter describing the extent to which the result of our GR predictions "missed" the actual data.
More recently, some researchers have tried to put the dark energy idea onto a more "theoretical" footing by arguing that perhaps the constant might not have a fixed arbitrary value, but might be a measure of the universe's expansion. That'd make the "modern" CC less fudgey, but it'd also mean that, as well as the thing not being Einstein's, it wouldn't be a constant, either.
So why did we initially get all those news stories announcing things like: "Eighty years later, it turns out that Einstein may have been right ... So he was smarter than he gave himself credit for." [*] ?
Putting it brutally, it was about PR. Attaching Einstein's name gave a false sense of historical provenance and a false sense of respectability. It let researchers use Einstein's name as a shield to deflect awkward questions about the apparent arbitrariness of their new expansion effect, and it turned a fairly boring and slightly negative story about GR failing to agree with the evidence into a snappy human-interest story about the throes of the scientific process coming out right in the end, and Einstein being right, and GR being right.
The "Einstein's Cosmological Constant returns: Einstein was right after all!" stories generated a lot of news headlines, and let researchers give interviews to magazines and appear on the telly and improve their departments' media profiles. Suddenly there were a lot of editors and journalists wanting quotes on the cosmological constant, because they wanted to print the same reader-grabbing "Einsteiney" headline, but didn't want to put their name on the claim, as reporters, because it was dodgy. So they rang round the universities and found a bunch of cosmologists happy to give the right quote if it meant getting their name in a magazine or getting onto the telly.
The story was junk. It was researchers collectively gaming the news media, and manufacturing and repeating a story that they knew would work, in order to get more media exposure. And unfortunately, that's the sort of behaviour that makes the general public more inclined to distrust scientists.
Why String Theory is Still Not Even Wrong - John Horgan recently sent me some questions, and has put them and my answers up at his Scientific American site, under the title Why String Theory is Still...
2 hours ago