Probably since Ada Lovelace's time, there have been attempts to find better, more reproducible ways of creating software. Software engineering is a field that is so immature, so young in the world that it hardly deserves the term "engineering". Every programming language solves the same problems in a myriad of different ways. In fact, most programming languages solve the same problem in multiple ways. While this can be a source of strength (Perl is fscking great and I will broach no argument on this subject), it can also be a source of confusion, fear, anger and general all around crappy code. Because software engineering is more at the level of craftsmanship than, say, civil engineering, it is not unusual for projects to come in over budget or late. There have been a whole host of methodologies meant to reign in this chaos, to put some certainty around the process of making computers into something other than paperweights. The methods ascendant today seem to be largely based around the "agile" concepts, and the leader of those seems to be Scrum. But is Scrum already doomed?
Agile is a set of principles designed by people who wrote software for a living. It even has a manifesto, because you cannot get anything taken seriously in the computer works if it doesn't come with some piece of overwrought labeling designed to make people forget that they aren't, actually, in a struggle for the soul of the world. The manifesto, however, does do a nice job of encapsulating the ideas that dive the agile methodologies:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.
These are not bad ideas, by any stretch of the imagination, but the devil, as always, is in the details. In practice, these principles have led to distinct flavors of agile methodologies. The first are focused on the practice of writing good code. They focus on the steps developers should take to ensure they produce the best possible code they can. The other branch is focused on the process of running a project. It speaks to how people should organize project teams in order to deliver code. Scrum is focused entirely on project management, assuming that the practice of software engineering will be implemented within the guidance of a Scrum project.
That, of course, is the first problem with Scrum. Since it is a project management methodology, it is primarily sold to and focused on the people most involved in project management: project managers and other management layers. By leaving as an exercise for the reader the best practices of software engineering, Scrum acolytes generally end up giving short shrift to the importance of software engineering practices. If you create a methodology, then people following that methodology -- especially if they are not developers -- will discount those things that the methodology does not emphasize. In the case of Scrum, then, we are left with a process built around an assumption that is not often true: that the team will employ agile engineering practices. No project methodology can save a project from poorly written code. Scrum is no exception to that rule. Bad code will ruin a project, and Scrum does little to prevent bad code. That however, is only one of the problems with Scrum.
As you can see from the principles laid out above, agile methodologies are not designed around documentation or detailed project justification. They are meant to be free-flowing, reacting to the current situation rather than adhering to the grand plan. This tendency, of course, does not align with the normal operating procedure of many, many kinds of people who can be involved in software projects. This is not a concern when the agile methodology is focused on the practice of building software. In those situations, the effect agile methodology has on how other aspects of the business do their business can be managed and minimized. This is not true about project management methodologies.
Project management methodologies, such as Scrum, are focused on how the project is run. That means that finance people who want detailed ROI calculations and business analysts who want to dictate detailed requirements for every aspect of the entire project and user experience designers who want to create a unified, holistic interface must all bend to the demands of the project methodology. In practice,of course, very few companies put the needs of the development organization at the center of the way it does things. In practice, then, these conflicting ways of working mean compromise, and compromise means that one or more of the agile principles is ignored. And that means that much of the power, and thus effectiveness, of the agile principles are lost, increasing the chances of project failure. By focusing on the project, rather than the process, Scrum opens itself up to damaging changes by the rest of the business.
The acceptance of Scrum in the business world is, ironically, the final nail in its coffin. It is a common joke among developers that Scrum has risen above all the other agile methodologies because it was the first to come up with a certification process. Once you could get a certificate, then you had an easy story to tell upper management when it came time to justify your project methodology choice. But certification has proven to be a double-edged sword, as with almost every other kind of technical certification. For some, it is a validation of their experience, training and aptitude. For many, many others, however, it is a chap piece of paper handed out after a class that no one fails. There is now, then, a mass of people who apply Scrum not out of an understanding of the value of the agile principles but rather because a book or an instructor told them to do X and then Y and then Z. Instead of professionals applying principles, you have robots applying recipes. A recipe may work well enough in some circumstances, but it is rigid and inflexible, making mistakes out of a lack of understanding when the rules should be broken. Straight jackets are not conductive to creativity or flexibility.
Scrum is born from good ideas and good intentions. But it focuses on the wrong aspects of building software and its own success contains within it the reasons it cannot be successful.