Obviously, none of you have ever worked on software.
I have. But then, I agree with most of what you said.
The important point is that when you're writing new software, it's not analogous to making a movie. It's more like inventing the VCR. You must create something entirely new. There is no established proecedure, and everything must work.
The design process is much more difficult than any other type of creative endevour because it is totally unforgiving. In other arenas, if you make a mistake, you simply have something not quite as good as it otherwise would have been. But when designing mechanisms, most mistakes will result in your whole design not working at all. These mistakes must be fixed, and there is no way, literally no way at all, to predict how long this will take, especially early on in the process. Not only that, but unlike a film or other such project, the Mongolian Hordes Technique simply doesn't work. (Read "The Mythical Man-Month: Essays on Software Engineering", if you'd like to know more.)
All the same, long before their delay announcement, Valve should have known if stuff worked or not. The later the stage of the project, the easier it is to give estimates.