I wrote a very long email for a client bid recently, and it got me thinking a great deal about how agile software and Drupal work together. I know this is a technology tirade, which I'm against, but I was doing a lot of deep thinking on the subject and this client got the (assumedly) benefit. It's actually somewhat undiscussed, but Drupal's object-based (module-based, whatever) approach maps directly into Agile software development quite nicely.
So I wrote down a few thoughts for managers of projects that might provide some ideas about what corporate managers face, and what kinds of concerns members of the corporate Drupal community are facing on a day-to-day basis. It's somewhat interesting to me, and as usual, I'm very interested in your feedback.
Best,
Jonathan Lambert
Dear Startup & Corporate Managers:
This is long. You should still read it. If you’re confused about Agile approaches to software engineering, and what it means for you and your business, this will help to clarify. This puts this whole, "buy social media sites" business into perspective, and should provide some ongoing guidance for Drupal based implementations.
At core, the problem most corporate managers have is reconciling corporate culture and an understanding of "how development is done," against the tools that actually provide the functionality required to build next-generation Internet applications.
Many corporations are more concerned with day-to-day risk management (and often rightly so) than innovation, or at least would rank risk reduction above "doing something cool with the technology." This is a fundamental conflict of values when using open-source software to build corporate backed projects.
The software methodology that's been designed and developed and created to deal with the uncertainty of software engineering in modern Web 2.0 applications is Agile software engineering, consisting of a number of different approaches with different names and ideas such as XP, Scrum, etc. With the Agile software approach, which Drupal projects easily map into, sometimes the biggest problem is mapping the terminology of corporations and agile software projects.
This conflict of terminology can lead to incredibly expensive problems for projects. For example, how do you contain risk in a complex software project, while retaining the innovation required to iteratively respond to changing project requirements? So the problem is how to go about reconciling corporate needs with Web 2.0 software methodologies that provide innovation, and have a productive discussion about how business and development can work together.
The first step is not to talk about software
As you know, we work with Drupal. We also work with Java, .Net, and a host of other technologies – whatever properly supports the project, though we really love Drupal. It's not about a particular technology, it's about an approach to the way you design and build systems based on how technical choices support your core business goals.
Every project needs to be innovative to some degree even if your goals are quite modest. However every project also requires a balanced
budget, and a good, hard look at the ongoing relationship between your organization, the software and tools that get built and already exist, and the consultancy that builds it. You have to make sure it drives your core.
That’s why we use Drupal, Hibernate, Spring, and other Open Source tools. We think Open Source is bringing about better technology opportunities for companies. Open source enables you to try out different business approaches, and actually evaluate software for its merits.
While it is possible to get closed source software for evaluation, it's entirely different when you're actually able to try out different acts as a company and see which one is going to give you the best bottom-line business results. This is a fundamental shift in software, and one of the main drivers of the Web 2.0 innovation. You're now able to evaluate your technology decisions from the inside in an ongoing relationship with your product, and you can continually reevaluate the value proposition that software provides throughout the period of time you have a relationship with that software.
So, if technology is no longer at the core of decision-making, if the overall value proposition is now more akin to a question of what will we use for right now, and the far-reaching impact of that decision is more a question of organizational support and priority, what is the criteria by which you should be making an evaluation to select a software vendor, software solution, and software development tools?
The solution is to move beyond rhetoric, and start looking at the approach to how software is built. It does not matter whether you use Ruby on Rails, Drupal, Java, or another software set. The real question is the process by which you approach the development of that software.
The technology choice isn't as important as it used to be. It's how you work with that software that drives software decision-making, and the type of organizational approach you have with the software you implement.
See what you want to build is social networking site...
So you want to build your giant, successful company into a social networking site. Congratulations, you must've noticed that social networking provides a fundamental shift in the balance of power between people, vendors, and the organizations that bridge the gap.
The old model of a software company delivering the vision of an organization is a giant, monolithic project is slowly being replaced with a new model where the organization, technology, product management, and vendors work together to build exactly what users want. For years, it's been well known that websites that deliver exactly what users want tend to be more successful. A great deal of time and effort has been spent creating website testing techniques, analytics, as well as testing methodologies to learn how to reconcile user needs with business requirements.
The old model was about doing all of this testing upfront, and incurring a great deal of costs in the process. You would start with an idea, and then would take that idea and you would sit down and write up a requirements document that would outline exactly what it is that that idea needs in order to come to light. Once this document had been produced, you would take this document and allowed into the world to find the consultancy that was able to translate your vision of business requirements into technical requirements, and of course wireframes. If you were progressive you would spend a great deal of your time testing initial versions of your site in a lab for usability purposes. This would provide you with some early feedback about what works and what does not work on your idea, which you can then incorporate into your wireframes and which would provide you valuable feedback about what is and what is not working in your application.
A great deal of time and energy would go into this upfront planning process, and you would probably dump a great deal of money into it while you're working at it. The idea is that if you plan the work up front, you would reduce the implementation time, the business risk and the cost of building the software with expensive software engineering firms.
The problem with this approach is that as you factor the project, and work within over time, you find that 30 to 40, sometimes even 50% of the quality planning that went into the project is actually incorrect, and you end up with a project that is somewhat of a variation on the original plan. As you progress through software engineering you learn things. You learn what works and what does not. You learn the things that provide value to users, and you find the things that seemed like a good idea at the Project planning phase are no longer able to be implemented with exactly the original vision, or you find some kind of major architecture problem that makes it impossible to achieve your original vision.
This means that you're actually going to end up with some variation of the original vision, but because of the fact that you're working with fixed bids, you're actually going to end up with some kind of extra costs incurred if you actually execute the original project schema. Or you'll end up with a project which does not fulfill all of the original requirements, or it takes a great deal of time to complete, and many times you end up with a budget overruns which is 50% or more. There are horror stories of projects that run over 10 to 20,000%, and many software engineers can give you specific examples of projects which are filled with this approach.
The new model is about collaboration, but it requires trust.
Social networks are process, not a product, and by nature are cocreated with their users. Trust is required because upfront planning is not entirely possible, and if you're going to work with software projects which fail in their execution more than 50% of the time, you end up having to build change into the process. To do so requires a great deal of change in the way software parts are built, and most importantly in a relationship that vendors and their client have.
The fundamental value proposition of agile methodologies is to shift the planning into an ongoing planning that's done on a day-to-day basis in a structured fashion. To do so requires a number of things to be in alignment:
So you want to turn your property into a social networking site and social network inside her process not a product, and you need to be able to close the feedback loop between development, the business, and most importantly the users.
The good news is there is a high correlation between agile development methodologies and successful social networking builds, as demonstrated by countless examples. So the risk will probably have great deal of reward.
Waterfall versus Agile
Most project start off with a waterfall approach, which is to say that there started with a plan to turn it into a wireframe, which turns into milestones, which turns into a final product, which then gets fixed because it's broken. There is nothing wrong with this approach. It's merely more expensive, and actually have a higher risk than agile, but it's also extremely conducive to risk containment for organizations.
However the utilization of waterfall methodologies for software development can end up driving a correspondingly lower likelihood of success for projects that require it, or would be better adapted, can I tell methodologies, such as social networking applications. While you may obtain your immediate objectives, the project tends to be relegated to mediocrity over time because you're not asking your users to participate in its creation and adoption, and this process is at the core of making social networking applications and their like successful.
What agile does is move your development team into a line with your business goals, and closer to your organization, but only if it's done right. If your focus is driving massive online community and incredibly quick and seamless user adoption then you need to start thinking differently. Right here and right now.
If predictable waterfall development with upfront risk mitigation and the big team of offshore developers is on the left, and agile software development with a small, fast, extremely high and team up somewhat more expensive programmers building exactly what users want is on the right, most companies also more in the middle. So in fact a modified approach tends to be one of the best approaches you can use to build software projects of this type, but it also can become a retreat for corporate managers are unwilling to take the risk of true agile software development. Using wireframes as a starting point for agile is actually extremely useful, and allows the agile programmer to go back over the course of the project and make sure that the project goals that they are currently working on using step with the overall project goals. But they are not the be-all end-all of project management, and are not used as a Bible for developers.
In fact, a modified approach tends to be one of the most successful ways to make large projects go without a hitch. The organization is able to communicate in an ongoing basis extremely effectively based on the upfront needs, as long as they are not attached that the outcome of the project is exactly what was in the wireframes, because this is not the point of agile software development, and because user needs cannot intuited upfront by business managers.
So before you actually embark upon your social application development process, you're going to have to resolve whether 91 use open source. And then you know it's a lot to ask of most software managers you're going to have to sit down and try to understand the type of approach it like used to build the product, and will be instrumental on whether your project succeeds or fails.
Business Goals
most companies have clearly defined business goals. Things you'll often find in corporate RFPs:
These goals are incredibly important, but often the route to achieving them is not something that you can map out up front. It turns out that agile software has an impact not only reaching your short-term and long-term business goals, and not only in driving value to your social applications but actually driving long-term value.
Any vendor can provide short-term value, very few (in any industry at any time on any bid) can provide long-term value.
This document outlines a few things we learned about how to drive long-term value was social networking applications and agile methodology.
How do you reconcile waterfall and agile in a corporate mindset?
Let's start with a thought: wireframes and milestones, the core of deliverables for most projects, are the basic building blocks of waterfall development methodology.In fact these tools are fundamentally incompatible with the full convert an agile process, except as a starting point. You cannot use them both at the same time, and realize to benefit from both of them, though you can use one as a starting point for the other -- generally waterfall is a starting point for agile.
The most fun mental feature of agile processes that there are no milestones. It never breaks after it first starts is like that the application should never break after it first starts working. Each feature added during development is complete, tested, and working before it is committed so there is no actual need to have milestones. You can bunch a bunch of features together to make milestones are officially, which is useful for supporting organizational needs like launch dates. You can make marketing extremely happy and create a release, which is a milestone of sorts, but it's not a functional milestone. It's merely done to serve business needs there is no actual need for it were for milestones in agile software.
Waterfall development assumes you are building applications as a whole, has also counter to agile methodologies. You can actually use agile with wireframes, but you actually use the wireframe process, which we call paper framing, to outline the work of a particular element as you work on it, so you're actually doing your project planning as you go along. There is no up front wireframe process, and you avoid the expense of having to do your project planning up front, and you become able to adapt to changes over time as you work. If you do your project planning as you go along, you also gain the benefit of hindsight on the stuff you've already built, and you avoid the pitfalls of having software features developed that bottleneck the application, because you're able to build that feature to support exactly what's possible with the application at that time.
So there is no actual need for wireframes, except as a starting point for gathering features.
Often in waterfall, especially in short-term projects, you end up with a set of requirements that don't allow the application to start functioning until almost the release date. This is because all of the elements of the project have to come together in order for every element of the system to start working together, though this is not always the case. But often, you'll find that features that were laid out in a milestone, say, milestone one, require a great deal of the functionality to build before that milestone is achievable.
With agile software development methodologies, you start by building the minimum amount of features required to ship the first basic feature of your site or project. The moment you have the minimal amount of functionality working for your project.
The moment you have the minimal amount of functionality working you have a home page that can ship. Because you have no need for milestones, you're able to build things in the most logical order. This means you're able to have a working application virtually continuously, and technologies that work with agile, continuous integration and test driven development, are designed to support this process. Note that we have just eliminate the risk that you will not ship the software, and in many cases you have just started the project when you ship a working application.
Back to Drupal is a platform, not a product
Drupal is a social networking platform: it's built around features, not
pages. It speaks many Web 2.0 technologies natively, such as AJAX,
RSS, JSON, etc. All of this is built into the platform.
Because of its Web 2.0 and native functionality, the Drupal often maps extremely easily into agile projects. With agile, you're often working with features. Features are micro units of work, similar to something like a micro-milestone. Each feature has a list of dependencies, responsible development party, input from the business, upfront planning, and probably will require some knowledge about the best approach for implementing it with Drupal.
As such, most projects require original specifications to be broken down from their wireframe components, and mapped into the list of features. Then it requires an expert to build the core functionality of the application for each feature is a best-of-breed approach with Drupal. This enables estimation, a timeframe for completion, as well as provides best practice approach for what Drupal feature should be used to create the application.
Drupal's feature driven approach is the basis of our consultancy, but one of the problems that you confront when working with agile software, is a substantive lack of detail in agile proposals. Most agile project start off by very specifically estimating exactly what needs to be delivered in one day chunks. This looks a lot like a list of feature deliverables, and we've heard from clients that it looks a lot like “generic community components.” It is not, it is a map of everything that must be built, and this is the period of time when wireframes are very useful.
But when you approach a project we would like to use agile with from a, “how much is this going to cost me, how I contain the risk and how I know that I'm one to get exactly what I want at the end of the project,” you're in for some disappointment, and you're likely to come up frustrated and feeling like agile as a process is not worth doing, because you have to give up so much control to be able to use it. For corporate managers are some level of education required to use agile, but jokingly it's often referred to as “drinking the Kool-Aid.”
And since most corporate clients are not happy with the idea of sending a blank check that a specific idea of what gets delivered at the end of the money, but many agile practitioners do is take whatever specifications are delivered into what's called a story estimate. This is essentially a one day at a time level breakdown of the time would take implement those features an agile approach.
This is not always an accurate estimate of the cost of the project, and at best. It's only a ballpark to not time it's going to take to build the software project, because you're chunking out the project in one day increments.
Remember that agile as a platform designed to encompass change, and built with the expectation that features and business requirements will change, based on user input, throughout the entire project. This is a fairly accurate assumption, and the reason we have agile, as waterfall projects tend to break, or increase drastically and cost as change occurs.
Most agile projects provide a preliminary pricing justification, and not hourly development breakdowns, and certainly not overall project cost. Overall project cost is a question of how much money you want to spend, and when you stop. The idea is to build until you have a successful product, which ultimately is the goal of every software project. Many agile projects are done on a value-based basis, because ultimately you're hiring someone for their expertise as a full-time employee, and it's better to be able to control costs than it is to have exacting control over every hour or every dollar.
Even providing estimate is frowned upon by some agile purists. You're supposed to tell the client that the cost is a factor of implementing features, and that they should measure their budgets ongoing based on feature implementation. This works for some clients, but it's a hell of a thing to sell to smart, experienced executives on the idea because their first thought is, "I'm not giving you a blank check."
Agile isn't about blank checks. But let's move on to a little questions and answer section.
Questions and Answers.
What are the top 5 questions companies ask when thinking about agile development project? Well, this is a collection of 10 questions I've been asked by companies considering agile for project development, and I considered to be a fairly good list. -J
Q: Am I not just signing a blank check?
A: You are in fact doing quite the opposite. Agile
puts the business in the control seat, because all of the engineering work
revolves around the business person driving requirements. You gain
massive insight into progress, and avoid the possibility that the
implementation will "go south" without your knowledge, which is
actually a very common problem. You also have a better understanding of
how much each feature costs, as well as what value it drives to the
business. Working with agility makes sure that you're only working on
features that drive value (for the business and for the customer), where as
waterfall development tends to be about driving value just for the business.
The overall cost is usually about the same, but you get exactly what you're
users want and you execute only things that drive your bottom line business
requirements.
Q: How do I know I can still ship on time?
A: You never can not ship once you have a shippable application.
"On time," changes meaning here, to, "what features can I get
into a given time frame?"
Q: What about my existing wireframes?
A: If you have them, they're an asset you should use in your planning, and
to drive your overall development.
Wireframes serve as a starting point and a reference for
ideas going forward, but ultimately the features that get delivered are not
driven by them, just mapped to them. This is why a developer in Agile
will actually go back to wire frames on occasion when speccing complex UI.
Q: Will my project have reduced functionality? How do I know I'm getting
all I'm paying for?
A: Your project will have increased utility to users, which in fact is more
important than the functionality. Most
of the time, eighty percent of features are ignored, and only 20% of features
are actually delivering the value for most applications. But, all those features are built and
shipped, and part of the application because they were part of the
specifications originally designed by the business managers to create the
product.
So, even if it does have reduced functionality, it would
be the functionality that matters to your users and your company. Because
you pay as you go, you don't blow money on things that seemed important when
you started, but are no longer relevant. Our best guess based on our
experiences is that about 30-50% of functionality in most waterfall projects
never see the light of day, and is generally replaced by other functionality,
but the company still pays for it because it's fixed bid, and the company still
has to pay for the changes. It gets expensive, and most software projects
go over budget. In the waterfall world, this is referred to as
"scope creep." This doesn't exist in Agile. In agile, we
jokingly referred to it as a "project."
Q: When does this process end? How do I know when I'm done?
A: You decide on your budget cap and you "buy features" until
you're either happy with the ROI or you decide that it's time to stop spending
money. Thats it. by the way, this is exactly the same process you
follow with waterfall development, but you don't have the luxury of stopping
until the project ships. That's kind of the point. Agile says stop
whenever you want. Waterfall you have to wait and pay until it ships, or
lose the entire investment.
What is agile?
The primary focus of agile methodology is defined in the four main bullet points of the Agile manifesto:
Why would you ultimately choose Agile?
Agile does not mean:
- Your project is going to ship late
- Your project is going to have reduced functionality
- Your business requirements are not going to be addressed
- Your wireframes are not useful as a tool for generating requirements, only that your wireframes come second to your feature-driven requirements
- You cannot deliver milestones, only that your milestones are arbitrary and do not have core value to the process
You choose Agile to reduce risk, expose project complexity, and incrementalize the cost of implementation. The number one reason to choose agile is that you can change direction based on changing requirements, which is the main reason it's used to drive social websites. Building what users want is the main concern of social websites, and what drives adoption and repeat visitors.
User testing, black box testing, and focus groups are no substitute for real users and real world feedback from analytics tests, data mining, surveys, and beta testing, and that's the information that should be used to change your project as soon as it becomes available, not during your next phase of development. That's why Agile is better for social sites.
Remember, you're not building another software applications: it's a social networking platform. It has a high correlation of success when executed with Agile. Therefore, you're more likely to succeed at your ultimate, long-term business goals if Agile is at the heart of your strategy.
The process is the product.
You have no idea what the product actually is, only what you think the product
should be based on some user testing of your baseline assumptions, but you have
not tested your baseline assumptions, and you cannot account for or model new
changes from the original product you envisioned. Stop trying to dictate
to users the product they should be using. Find a starting point that
works for your business needs and work from there - this is Agile development.
Risk Mitigation
Shipping this application as a complete application without following an Agile
development model is extremely risky. Instead you need to start by
replicating the existing features of your site and minimal extra features, then
shipping the Beta. Then introduce, one feature at a time, the features
that are of the highest business value and lowest time to implement. This
enables a transition time - a period of time where the business can adapt to
these new capabilities, while also not alienating your users, as they're able
to come along with you on your Beta and help guide development in a way that
gets them invested in your success. After release, this ramp continues,
and will please your advertisers, as each new functionality release will drive
additional visitors to the site, increasing the repeat user-base, and actually
raising the overall level of traffic. Further, you will actually spend
much less money getting this to market, and almost eliminate the possibility
that your project will not ship.
One of the reasons agile tends to be somewhat expensive in proposals and during
upfront planning, other than the fact is blocked out on a day-to-day basis, is
that it tends to look much further ahead than other types of development. Because of the fact you're simply looking at business
goals, and not actually having to do specification for each and everything you
come up with, you tend to be more comprehensive in your planning.
The very act of building a social network is the process of creating
it. You are not building this product for your organization, you are building
this for your organizations users. If you do not involve your users in
the process of building this community, you are blowing the opportunity that
each and every development project represents, because social networks are the
product of the process of their creation. Building them is about catering to
the customer, not about catering to the client's product needs.
So, great, so I sold you on Agile? Now what?
If you want to work with agile, here's the general idea of what it would look like:
· the first thing you do in any project is set up a face-to-face meeting between the business, the development firm, and any other relevant parties. You need to have the core development staff and the core business personnel at the same table for your initial kickoff planning.
· The second thing you do is start by breaking down everything. If you have wireframes, you break a wireframes down into features, and if you don't, you need to come up with a specific list of everything that you can think of at the site needs to do, but not be to focus on the details, because you can change it as you go a long. The most important thing you do is come up with as much detail as possible for the near-term, and lay out as much strategically important information about the long-term vision of the project, in terms of future development, so that the development group can get a mind meld with the business owners.
The next step is to go through all of those features and list them out on 3 x 5 cards. You then identify and sort them by business priority. This ensures that your developers are working on your most critical features first, the features that are required to launch your application in the shortest time possible. So you store your cards so you know Jackie would these be built first based on this rule:
To
complete your understanding, I've got a few a/b graphs we've prepare for you:
Building a social network vs. Building a "product"
|
|
Building a product |
Building a Social service |
|
When is it a product? |
When you say it is |
When THEY say they are a community |
|
Release frequency |
Build it in great big releases |
Release, early, release often, adapt to your users |
|
Release rigidity |
Rigid releases |
Flexible adaptation |
|
When do you involve your users? |
For good product developers, they involve the users all the way along; for normal product developers, they ship first, and ask questions later. |
All the way along through the entire build. Not your ENTIRE community, just a subset of grey-hairs who are highly invested in the community. |
|
How many layers are there? |
Technical - build what you spec. |
Technical, Social, Economic: Build something that works well, that supports the social structures, and that has adequate economic incentives built into the system to get people to contribute material and content and time and energy to the community; |
Build a social network with the two methods: a comparison.
|
|
Waterfall |
Agile |
|
What drives the development? |
Wireframes drive the project |
Wireframes are used to infer the feature set which is then prioritized weekly and daily and which REALLY drives the project to completion; wireframes are also used for future development features. |
|
Ideal team |
Large offshore team who have done these large social network major site builds and who can duplicate the wireframes quickly. (i.e. NOT Workhabit) |
WorkHabit or similar team working in an agile methodology |
|
What is the release cycle |
Big bang (all functionality at once); |
Replicate current site, then begin adding social functionality one feature at a time in collaboration with your community members |
|
Agility |
Not as much; costly to refactor on the fly |
Inherently flexible. |
|
Development structure |
Milestones |
A long list of features in priority order |
|
How do you budget? |
You get a fixed price bid and hold your developer to it |
You decide on your budget cap and you "buy features" until you are either happy with the ROI or you decide that it is time to stop spending money |
|
How involved is YOUR team? |
Intermittent involvement from key team members at specified "check in" and "milestone" dates. |
DAILY - near full-time involvement from contract signature to "Marketing launch" |
|
Features shipped vs. "Launched" |
A shipped "version" is a "launch release" |
With iterative development, you create marketing "release dates" that are simply the time that you bunch the features up into a logical release grouping and "announce" them (even if they have already been out for a while.) |
|
Benefits |
fixed spending cap; fixed development time; predictable expectations for the outcome. |
you can adapt to change faster; be more flexible if business requirements surface that aren't in the plan; can get it to market faster and develop faster, thereby increasing potential return; more fitted to user demands than arbitrary "design demands"; you can "ship at any point" once you have started. |
|
Risks |
fixed staffing; no ability to flex the site with the user's feedback; don't know what the user feedback is until you release the larger block; changes cost a LOT more. |
business isn't educated enough on agile to buy into this model; requires EXTREMELY high end developers - not a hand-off team in India at low cost; in other words, need "architect level" people for the team and they cost a lot more; will incur a slightly higher cost in development on testing; |
|
Summary |
You build what you want to build regardless of what your users want; |
You build what your market and users want you to build, and you use agility to do it. |
The bottom line for managers, they need to make a choice between agile
waterfall development approaches, and you should make that choice as early on
as possible, because it's going to drive not only your technology process, but
also the team you use for execution. So
it's up to you, do you go agile or waterfall?
Making that choice can make or break your project, and the most
important thing you can bear in mind is that it's all about the people that
build your product.
There are other reasons to use a different approach, and we're not advocating agile for every situation. For example, many professional services Drupal-based projects require milestone driven development, for risk mitigation, ease of client management, and many other good reasons.
Many times, a hybrid approach works best.
Ultimately, it's important to consider each project on it's own merits and make a clear and informed decision about which approach to take.
jonathan's blog
Comments
Nice article
This is definitely a keeper in my book. You having put the Agile method in a really clear context with Drupal, which is something I have been trying to formulate for a while. -Greg
Thanks!
Thanks dude, please let me know if you think of any improvements.
This is something people need to know when working with Drupal, especially as we start to move it forward with test driven development and continuous integration (install profiles!), and other "agile" technology platforms.
+1!
Having worked on both
Having worked on both waterfall and Agile software projects I tend to agree with most of what you say. I prefer to use Agile methods; however as you noted a hybrid is often required in order to suit the business.
I disagree with your assertion that there are no milestones in Agile processes. A milestone is simply a date that something needs to be completed by. All Agile methodologies have the notion of iterations - these are your milestones. You can have as many or as few as you like. The waterfall based projects I've worked on have had interim milestones that had to be demonstrated to the customer - it's just that they were at 6 month intervals not the 1 month intervals I would prefer.
Cost and schedule management will always be a contentious issue. I don't think either waterfall nor Agile methodologies make this any easier. Waterfall projects are often easier to sell to company management because they appear to have a fixed cost, fixed requirements, and fixed duration. In practice though this is not the case. Agile methods on the other hand tend to be harder to sell because they tend to work on the principal that "we will deliver business value to you at the end of each iteration and you can tell us when to stop". Most upper managers (dept heads and the like) I've worked for have had a real problem with the Agile approach because they can't see when the project will be finished nor how much it will cost. In the end, I chose to use a hybrid method to deliver the project. We implemented the system using Agile methods, but reported as though it were a waterfall project. As project manager I benefited from being able to track progress more accurately and get immediate feedback from our customers. My managers got the benefit of receiving realistic progress reports against cost and schedule.
There are a lot of benefits in using Agile methods; but without highly disciplined team members Agile methods have the tendency to go off the rails just as quick as waterfall projects. If we lose sight of what the customer wants, Agile projects become a lot like waterfall projects. (In this case, the customer is not just the people who will actually use the system, but also the managers who oversee the project).
Nice response...
That's a great response.
I posted this blog post because I thought this would make a really great start to a whitepaper for managers on Drupal-based projects trying to figure out project management methodologies. Most seem to need a "primer" to come up to speed, as a lot of this is super new, and even more new when it applies to social media.
Your comments regarding milestones are correct. I only meant that milestones are generally either set up as "task targets" so that "X many stories will be done on this date" or "date targets", or "we will check in by X date and see where we are." Actually providing a X stories will be done on X dates in agile is EXTREMELY difficult, as you kind of need flexibility on what's included or when the date is set. It's common sense that you would do this, but it accounts for most of the "failure" in waterfall projects (failing to hit a date or failing to get in promised functionality). This tension is what managers like to manage on projects, even through it's pretty much a bs metric. It does look good to the board.
Nice comments. Let me know if you think of any others.
Jonathan
Other variables in the choice
A couple of other factors to take into account when choosing between Agile and Waterfall models:
Fantastic article, jonathan.
Exactly
I think you nailed something important here. This is exactly the problem I've run into over and over again.
The solution seems to be to provide a super clear modus for the shop ("this is the way we work and here are the benefits"), but it's a different list for CIOs, CTOs, CFOs, etc... depends a LOT on who the economic buyer is.
All-in-all, great comment.
great information to introduce a new and formidable concept
I must admit I was intimidated to begin reading when I saw the little blip on my scrollbar indicating that this was a monster of an article, but I'm really glad I read through it all. The terms Waterfall and Agile were unfamiliar to me, but the concepts have been something much on my mind in recent projects that I've worked on.
I worked in the corporate web development environment years ago, before the idea of social networking sites even really existed. Those projects all had a very waterfall-like approach, obviously. In the years since, I've approached projects with the waterfall mentality and run into the problems that you mention that are related to the waterfall approach (projects bloated with features that the users don't use, launch dates long in the future making them difficult to predict, "scope creep" which I just thought of as necessary project adaptation). This caused me to question that "old" methodology and I've been using a much more interactive, multi-phase approach since then. Some of what I've been doing sounds a lot like the Agile methodology, but not as refined. I've definitely had issues trying to adapt my business to this new methodology. I had tons of trouble figuring out how to reassure clients that this was the best way to go (fortunately, I had a good relationship with some of them that enabled them to trust me with this) and figuring out how to structure billing was/is a huge problem as well.
You are totally right that more Drupal developers need to understand the concept of Agile development, and how to implement it with their clients. Drupal enables developers to build incredibly powerful websites, but without a solid methodology for approaching these complex projects, the resulting sites won't serve their users or buisnesses needs as well as they could. Without realizing it, I've been trying to learn this, but the changes are sometimes dramatic and it takes time to adapt to the new methods. Do you have any suggested readings, especially in terms of how to analyze the success of individual features that have been implemented?
Anyway, thanks for a great article. It has reassured me that the adaptive process that felt right has it's merits, especially in the context of creating sites with extensive user interaction (social networking). I'll enjoy learning more about Agile and how I can implement it within my future Drupal projects to build more successful sites for my clients.
Agile on develpment/staging servers?
Just some more thoughts:
I'm thinking the Agile approach makes sense even on projects with a large base set of minimal launch features. When working with clients, I have found it useful to pull them into the development process early and have them interact with the development or staging servers.
Preliminary User Testing
Although these internal company users don't truly represent the target audience for the site, they can certainly give feedback as each feature is rolled out.
Admins learn in stages
Let's face it ... it can be intimidating to clients to have to learn how to shape their business around a website with complex and unfamiliar social networking features. Introducing clients to the administrative features early lets them learn how to accomplish tasks over a longer period of time instead of it all being slammed in their face when the site is ready for launch to the public.
This learning process, with the proper guidance will have clients quickly feeling more confident with the progress of the project and more empowered to be able to make their website work in the future.
Clarify what the client is getting
In the waterfall model, the client would see very little of this and would certainly not get to interact with the website early on, so it would be difficult for them to get a real feel for the type of product that they would be getting. Design mockups only say so much.
It can be difficult dealing with reactions from client who see a development version of a website which might be in skeletal form (no design/theming done), but the feedback I get from this process has helped save me a lot of time having to remap websites as the clients vision of what they really want changes throughout the process.
It sounds like you work with clients in a similar manner - having them play with early versions of the site you are building and getting feedback that effect the direction of development. Is that so?
Yes, yes, and yes...
Yea, we do.
Basically, what we do is try to get SUPER CLEAR on what the business end-result is, then work backwards to implement the technology. Basically start with the benefits (variable by project) which can include things like: increased user adoption, decreased support costs, brand improvement.
But the real driver underneath most projects is actually fiduciary, so finding out how to nail that is often the first question.
It's really interesting that when you approach things from that level, from really trying to understand the project from the client's point of view, you often only get to the Agile / Waterfall question WAAAAY down the path.
Open development processes are really important, but there is a risk with social sites with clients that don't get this fundamental point: what a client wants is NOT the same as what will drive use for users. Getting to beta is the only way to get that data.
Waterfall model to Agile Software Development
In waterfall model Writing specs in a fast changing environment and delivering huge updates just does not work. By the time the ink is on paper, it is out of date.
Agile method's greatest strength is actually about early delivery of working results. The end of the first iteration (usually somewhere between one and four weeks) results in a little bit of software that actually works. Here is comparison of Agile Methodology waterfall Model
Waterfall model to Agile Software Development
Post new comment