We create solutions in our heads, and this can happen quickly. Bringing these solutions into the real world takes time.
At some point, some people get worried about the amount of time, so we begin to manage our time. And along the way we'll perceive stumbling blocks that seem to block the flow of our energy on the way to implementing a workable solution.
Create, Divide, and Conquer is a methodology for how to carry the energy from perception of the problem, and creating solutions in thought, through developing those thoughts into plans, through realizing those plans in the real world, and finally, to testing and evaluating our implementation. Then we go around again, until we are satisfied. Also, at each step of the way, we might dive in and apply the same steps to a smaller sub-problem or step.
Create, Divide, and Conquer (CDC) is a methodology for solving problems. It is specifically geared towards computer programming problems, but CDC can be applied to other domains.
The basic workflow is: Create, Divide, Implement, Evaluate, Iterate.
This workflow breaks down into these steps:
1.) Describe the Problem, including ideals, forces, resources, limitations, constraints, and previous solutions.
2.) Brainstorm for paths from the problem space to the solution space, and record these ideas.
3.) Examine the ideas from brainstorming, and synthesize one or more potential solutions.
4.) Divide up the work of building the solutions using the idea of a system of components.
5.) Determine the highest-risk components. Prototype the high-risk components in order to prove that they will work, or to reject them on a rational basis.
6.) Return to the solution space, and re-map paths through the space to possible solutions, using the remaining viable components.
7.) Prototype or stub-out components and interfaces.
8.) Test component prototypes over their interfaces.
9.) Implement components, and systems which integrate the components, in phased releases.
10.) Test and evaluate: testing Functionality, Integration, and User Acceptance.
11.) Iterate and Recurse:
In the following chapters, I will walk through these steps in detail, giving examples from computer system design and programming, building construction, and engineering.
I developed CDC from my experiences in residential building architecture, design and construction; construction management; mechanical engineering; software programming and systems architecture; music composition; and music production. Some phases of these stints I did by myself, others I did working on teams, still others I did managing teams. I found that the methodology worked regardless of the team size. Surprisingly, I found that CDC worked for me when I tackled large projects by myself. Like the guy in the old westerns who just changes hats to be the train station clerk, the town clerk, and the sherriff, I just changed hats and worked on one component at a time, following the process of CDC.
In my career, even solo projects still required input and feedback from other people, so brainstorming skills and communication skills were required on every project. I view the paths from problem to solution as flows of energy -- human energy, human creativity, human interaction. CDC helps me foster an environment where people want to come play and help move the project from problem to solution.
The size of the group of people working on your project will affect your project. As stated in "The Mythical Man-Month" by Frederick P. Brooks, Jr., adding more people to your project does not linearly reduce the end-time of the project, because of increases in the time it takes to communicate, manage, and coordinate; and because of loss of focus.
Not all people can do all things, and not every project should involve multiple people. I believe that individuals, utilizing their individual skills, in concert with other people for collaboration and mentoring, produce their best work when given a large measure of autonomy. CDC is specifically geared to helping those who want to work alone, in small teams, or in collaboration with other individuals or teams.
For me, there are two human prerequisites to the CDC methodology. 1) The individual should have challenge, autonomy and meaning. 2) All people on a group project should be willing to support each member of the group, and should have knowledge that by fulfilling other group members' needs, they fulfill their own.
CDC proposes componentizing the project, as do many other methodologies. This leads to componentizing the artifacts of the project. Perhaps not every problem we wish to solve will benefit from componentization. In software, componentization yields benefits to the life-cycle of the project, especially compared in the maintenance phase to monolithic systems. But componentization helps directly with problem solving and implementation, because each component can be tackled by a small number of people -- ideally the terminal component is one which can be designed, implemented, and maintained by one person in a short period of time. By assembling teams of people who are responsible for these components, and who help to integrate them into the next system up, we can assemble very large solutions that have craftsmanship and humanity.
First, someone notices an improvement on reality that they think will satisfy some need.
So the first step is description of the problem.
(I think describing the world as a set of problems is not optimal, but "description of the problem" is a well-known first step, so let's think of "problem" as shorthand for "situation in reality that I'd like to change.")
At this point, resist the urge to describe HOW you would solve the problem.
Describe the Problem
Evaluate other solutions
It could be, at this point, that you have an extensive list of solutions, and a clear grasp of the problem. If you've stated the problem correctly thus far, you may already be done. If the problem is already solved, or nearly solved, or you change your mind about the problem, you really are done. Or you may be nearly done because the solution lacks only the implementation phase. This may seem like a punt, since the implementation phase is where all the "work" seems to be. But this is not true. If the problem is fully described, and the solutions are well laid out, then the implementation can procede smoothly, and perhaps almost immediately; if not, then realizing the solution could still be impossible.
An important part of stating the problem is stating the resources that you have and any needs that you have. Resources include: people, time, place, money, things.
Time should be stated as part of the problem statement if time is of the essence.
Other resources should be stated as part of describing the problem as well.
If you own a piece of land, and the problem is that you'd like a house on that land, then your resources become part of the problem statement.
Did you need the house done-done in two years, or would it be acceptable to have the house done in three years, if you could occupy part of it in one year? Does the construction cost represent a maximum you are willing to spend on the project, or does it represent decisions you've made solving the problem already? Be willing to eliminate statements from the problem in favor of a wider solution.
Why is this problem statement preferable to the first? Well, it is if you are the kind of person whose needs match this problem statement better.
You can see that there are many solutions available to this problem, and each balances certain trade-offs against the problem statement. A pitfall is to embed preconceived notions into the problem statement. This pitfall limits the solutions that you will be able to weigh.
Problem solving is all about trade-offs. Once you are in the thick of problem solving and design, you will push each decision to its limits to optimize the solution. These limits are clearly spelled out in the problem statement of a complete solution. You may not know all the limits, but as you design and implement, you'll come back to the problem statement and update it with newly discovered or measured limits. These limits are stated as we have already: resources, constraints, forces.
A complete problem statement includes existing reality, needs, desires, limitations, constraints, and resources.
So we are on the first step of the Create, Divide, and Conquer methodology, and already we need to brainstorm -- just to get the problem statement. You may have noticed that brainstorming is a major step in CDC. But we haven't gotten to the brainstorming step yet! CDC is fractal in nature. When you subdivide a task, you may find that that task requires the CDC methodology itself. CDC works by scaling up and down. Scaling up happens through iteration and composition, scaling down happens by recursion and subdivision. So in our first step (define the problem), we must apply the steps: define the problem, examine resources, brainstorm, propose solutions, weed out impractical solutions, divide up the solutions, implement solutions, test and evaluate, iterate and recurse.
When you have practiced this methodology, and you may already use it, some of the steps will come to you very quickly. You begin to see that each statement carries assumptions and limitations and possibilities, and these thoughts enable you then to see various possible futures. The trick is to let these futures come to you but don't let them rule you or your solution space. Until then, simply practice the steps and work methodically, but never so methodically as to hamper your flow or ideas.
So let us say we have our problem statement, including resources and constraints. Our problem statement doesn't have to be perfect. We'll come back to it again and again as we implement solution iterations. We can test our problem statement by reading it, sharing it, thinking about it, evaluating it.
If it looks like a complete statement of the problem, then we look at it and decide if we still want to solve the problem. Many problems, once correctly stated, cease to be problems. They may already be solved, they may become dreams, or goals, or we may decide that we don't want to solve that problem after all, now that we know what it truly is.
Let us say, then, that we wish to solve this problem. Now we are ready to brainstorm.
CDC has four guidelines about brainstorming: Flow, Allow, Capture, Recognize.
Spelled out, they are:
Flow
The first thing in brainstorming is to get into a good head-space. For everyone this will be different. You might meditate, do breathing exercise, socialize, do physical movement, altered states, whatever works for you. You want to be in a space where everything seems possible. I find brainstorming to be fun, exciting, hilarious, spontaneous, silly, earth-shattering, deep, and devious. I have found that when people are really groovin on brainstorming, they get real joy, mirth, and prankiness, and great ideas come up. It is critical to maintain this space if you can achieve it. Which brings us to guideline number two: Never Say Never.
Allow
There will be plenty of opportunity in the next steps after brainstorming to eliminate ideas that won't work. For now, we don't know what won't work, because we haven't tested all the ideas. Our brains are very good at associative, creative thought. But we must pamper our brains for them to deliver. Creativity is both vaunted and scorned in our puritanical society. Sometimes people who are creative are lifted up as "creative types," almost to let everyone else off the hook for being "uncreative." Sometimes people who are creative are ridiculed. In fact, humans are creative. We all need to give each other permission to be creative and let our light shine. The easiest single step to take in this direction is Never Say Never. The most effective and the most difficult is to apply this truly to ourselves.
Never Say Never means that during a brainstorming session, you take care of yourself and your team-mates by not letting a disparaging word leave your mouth. This helps you and everyone else feel comfortable coming up with the creative ideas. Any new idea represents change, and people naturally fear change. Fear of change makes us comfortable with conformity. Fear of rejection makes us quiet and conservative. There will be times when solving problems that a conservative nature will be a help. But during brainstorming, everything must be possible, and everyone must feel safe to share the impossible possibility.
Forms of "Never" include:
You get the point. Anything that can be perceived by the speaker as criticism will help to shut down the speaker. We want to encourage ourselves and others to speak freely and creatively. You can't create when others are destroying. Also, our most vocal critic is usually ourselves. When you hear your inner critic, thank it for its valuable input, and ask it to return later. Right now you are creating, and creating means making something out of nothing, which is itself impossible, so quibbling about reality or how others will see you is not useful in this moment.
On the flip-side, riffing is a good technique for giving affirmation and keeping new ideas coming. Riffing is taking someone's idea and massaging it a bit so that it is still the same, yet different, or extended somehow, and new. Musicians riff off each other, comedians riff off everything that has already been said, and brainstormers can riff in the same way -- funny but caring. In the moment of coming up with something funny, or enjoying someone else's thought, sparks show up.
For example, when trying to name something, toss out names, then riff on the names rapid-fire, while someone jots down what is said. Never say "No." Just add new ideas, no matter how silly, no matter how trite. The brilliant idea will come out in time. It might emerge later, after the brainstorming session, but the brainstorming will help it to emerge.
Capture and Recognize
Everything that someone says should be captured as a statement. There are lots of ways to do this.
Of the 4 steps in brainstorming (Flow, Allow, Capture, Recognize), step 3 (Capture) helps steps 1, 2, and 4. Capturing thoughts as they happen makes space for more thoughts, more flow (step 1); it affirms and allows (step 2); and individuals receive instant recognition that their thought was worthy of being recorded (step 4).
It is not necessary to record who said what, although you may if that helps people to feel recognition. I think that brainstorming sessions are group events, and all those participating should be recognized for contributing to the space that possibilities emerge from. I find that recognizing people in the moment of their inspiration satisfies their short-term need for recognition, and that recognizing people in the list of people who brainstormed the solutions satisfies their long-term needs for recognition. It helps to work this out beforehand with participants so that everyone knows their needs will be taken care of.
Recording in writing the ideas that are tossed out also generates a space of possibilities, and, seen together, these possibilities create more possibilities. Having the lists or maps visible to everyone lets people see the ideas that are building up, and stimulates associative thoughts. Since some people are more auditory, or you may be working over an audio channel, you may find it useful to have the scribe repeat the thought when it is written.
These ideas are then available for later evaluation, when we take the next step. The ideas finally chosen for implementation should be on this list. (Although, be open to more sessions when we iterate.)
Finally, the captured ideas allow people to see what wasn't tried, when they are trying to understand the project, or when they are fishing for project ideas later to enhance or fix the project.
Some ways to record ideas
In a large group, someone can volunteer to record the ideas on a whiteboard in a simple list. Enough words should be written to completely convey the idea, but keep it succinct -- somewhere between a word and a soundbite.
If someone has an epiphany, complete with a 3D mental model, then they should take some time to record that, probably in their own notebook. The group may wish to take time to allow people to record these kinds of things. In general, if someone is hot on an idea, then hearing other ideas will only slow them down and maybe make them lose the thought. If the flow is going well, then one person explaining their epiphany might actually impede the group flow. If so, wait a bit, for a lull, then offer this more complex idea. The group can experiment with how to deal with timeouts. With my brainstorming partners, we use a protocol where all I have to say is "Ooooo! One second!" and then I start writing, and the other people similarly write or keep staring into space. After a minute or two, my idea fleshed out, I'll say "Okay, I think I got it!" Then I may share the idea, or listen to someone else's idea, and share mine later. But at that point the conversation is opened up again, and other people's ideas were not stifled or criticized -- in fact many times people find this gives them a good balance of space for ideas, and fun and recognition when the sharing is going on. Or, you may want to offer your idea right away as a soundbite. Often, others in the group are going to the same place as you, and knowing that you are working a particular path will stimulate them. If an idea like this gains traction in the group, then focus the brainstorming on this tack, but not by shutting anyone else's flow off. In general, it doesn't help having a leader stand up and shout "Okay, people, IDEAS!" Brainstorming should be collaborative, open, and free, with fun and recognition. In this environment, people will come out to play. And when people play, they are creative. For this step, we are looking for creative energy, not critical thinking. Time for that soon enough.
Another technique that works well is drawing or mapmaking. Lists tend to be a bit left-brained, while drawings tend to plug into our spacial, visual brain. Both can be good for associative thinking -- in an extensive brainstorming session, I'll end up with lists, maps, flowcharts, and outlines. Maps can be maps in the cartographer's sense, but I'm thinking of maps and graphs in the mathematician's sense: points connected by paths, where the points and the paths can contain information/attributes.
Outlines or hierarchies can be useful for organizing thoughts. I tend not to build outlines past one or two levels deep when brainstorming, because breaking things into collections is abstract and connection oriented, but also cements the abstraction that drives the hierarchy as the only thing that governs an idea's placement in the solution space. In this mental zone, we want to use abstraction and collection to the extent that associations come up, but not to exclude ideas by pigeon-holing ideas or objects. So I make hierarchies or outlines right in my lists or maps, and I keep them minimal.
I find that mind maps work very well for many different kinds of thinkers. These are simply network diagrams -- nodes and links -- known to mathematicians as directed and undirected graphs, or maps. You draw your thoughts in a picture, or short set of words, then connect them to other thoughts using lines. The lines may just connect the thoughts, or may be a process or thought themselves. You can name the process or connection by writing along the lines. You could also write information next to the line if the line/connection has an important attribute. It doesn't really matter. We are not doing formal UML diagrams here - the point is to capture what people are thinking in a way that is rapid, associative, and open. Each thought might connect to multiple thoughts, and connecting lines show that clearly without dropping into implementation details.
At some point, the new ideas might wane. After allowing some space and time, revisit the ideas proposed so far, and get an intuitive sense of what has been recorded, while holding open the space of possibility.
Be sure to give yourself time for this step. It might involve taking a break, eating, sleeping, or talking about something else. But a very key step in this process is to give yourself time to really click through the vast number of possibilities. Our brains, it turns out, are very good at this kind of thinking -- this back-burner, subconscious, sleeper-thread kind of thinking. Many solutions come upon waking, or change of venue. I find that most quick solutions come to me when I get up out of my chair and head off for a quick bite. I'm no more than half a block out from my door when all of a sudden, everything is clear and I don't know why I didn't see it before. This pattern, one of working doggedly against a problem, and then giving up, only to find the solution present itself later, has happened to me so often, that I now actually plan for it. When I sense that I'm stuck, I make sure I have the information I need, then I go for a walk. I've been slammed against a thought-wall for an hour, but the solution comes in the first 2 minutes of the walk.
The kind of thinking we do when we are actively, intensely problem solving is exactly the wrong headspace in which to come up with the creative solutions to our problems. I like to do what I call "Stock and Process." I stock my brain with all aspects of the problem statement. I research every previous solution, look at every current solution proposal from brainstorming, then I relax, and forget I'm working on the problem. Later, my subconscious comes up with suggestions, showing me that it has been working steadily on while I sleep, shower, or eat my breakfast. This is a different kind of thinking than "which train is going to be faster getting uptown," or "what is the third root of 27." This kind of thinking takes time, and should be given that time, through stocking the brain with all the facts it will need, and then relaxation. This relaxation gives you the space, the freedom to think. It's about trusting our subconscious, about trusting our intuition. Later, we'll evaluate, critique, and investigate, but for now, we're looking for new ideas.
Another thought about frustrating thought-patterns. Sometimes we are frustrated because there seems to be no workable solution. But in some instances, what we are actually frustrated with is our inability to see the problem. This is why defining the problem clearly, in writing usually, will help us to map out the problem space. You can recognize this frustration mode when your thoughts seem to point all back to each other in a giant loop, or spiral, or spider-web. When you sense this kind of thinking, be sure to have a map or graph of the thoughts on paper. You may spot the way out immediately, or you may not. But seeing them all there stocks your brain with the pathways through the problem space, and helps your brain by giving it a physical model of that space, which helps it construct a mental model of the space. Confusion and frustration come about when the mental model is folded and circuitous. We don't know how mental models actually work, but building a physical model, such as a mind-map, can help the brain build a clearer mental model.
When it seems that there might be some workable solutions in the space, it is time for the next step - evaluation of possibilities.
At this point, engage the critical thinking portion of the brain, and look for what-ifs, and why-nots, and follow the logical extrapolation of implementing the ideas from brainstorming. Identify leads that seem to be easy. Identify leads that seem to be hard. Sort things by how much you know about them. We are looking for things that can be identified as necessary and easy. These things should be marked and maybe fleshed out just a little, to ensure that they are doable. Then we look for things that might work, but have an indeterminate implementation, either in time, resources, or practicality. Then we look for a complete solution in the workable and maybe-workable space. We then increase the space to include maybe-not-so-workable ideas, until we have some proposed solutions using our remaining, viable ideas.
So now we have a set of possible solutions, with no obvious Gotchas. Now we Divide.
If you have a team, then let the people help brainstorm how to divide up the work. If you are working alone, then the work divisions should make sense for the units of work you can work on at one time, which ones you need to research more, which ones you need help with, etc.
The work should be divided according to the following:
Capabilities should really mean: things that a person can do, or could do, given the right resources, time, learning, and mentoring.
By letting people in the team brainstorm on carving up the work you will:
For software, "units that communicate with each other" means that quality components can be designed using a divide and conquer strategy so long as 1) the authors can communicate easily with each other about the interfaces, and 2) the components can communicate completely, repeatably, and quickly over the interfaces.
A unit of work is a conceptual thing. When you divide up a large project, the project is a unit of work. The parts you divide it into become units of work. Ideally, you want to divide until the units match the people who will be working on them. For CDC, the terminal unit of work is something that one person can do while maintaining focus. At the smallest time interval, a unit of work might be coding one function. More typical in the object-oriented programming world would be to have a unit of work be an object class. Coding that class might take a month to get right, but you'd probably want to have iterations that were shorter than that, for example between a day and a week. The shorter a time frame and the smaller a unit of work, the more likely it is that that piece will be testable and will fit in with the other units of work. Too small of a unit of work can lead to things like too much time spent doing setup or housekeeping, or possibly breaking the flow. So an experienced team will learn what size components to break things into that works for them. I'd say start very small, work on the riskiest pieces first, keep communicating, and keep evaluating the work and how the solution is responding to the results of evaluating the components.
CDC assumes that in the "Divide" step of the methodology, individuals will work on components (the smaller work-units), and that individuals, working within a component, should be given greater leeway and more creative control. This is viable because the individual will have some responsibility for documenting how the component is constructed (for later people who need to maintain it) and will have responsibility for integrating the component into the larger system. In the integration phase, there are some standards that the component must adhere to. That is what makes CDC work -- the interfaces of the components are designed before they are implemented.
Before we choose a solution, though, we may need to evaluate some of the ideas. This is where prototyping comes in. We critically evaluate the ideas with our partners, looking for holes, pitfalls, and brick walls, by talking about them, drawing, checking specs, checking interfaces, to see if any of the ideas are impractical or incongruent. Anything that can't be proved with this type of Gedanken (thought-experiment), should be prototyped. This means building a working model that demonstrates only a subset of the functionality. In software, we build prototypes that have very different code than production code, except that the user interface looks and behaves correctly. These are used for user-acceptance testing. We also build test harnesses that use an existing piece of software to see if that piece will conform to our specifications from the problem statement. And, one more example from software is testing an existing piece of software for integration.
After building and testing the prototype, we would then know if the idea of using this existing software is not a crazy idea, and we might also get some idea of the effort involved in using this piece. This last kind of prototype is the most time-consuming step so far, but it can yield amazing results, because 1) it allows us to re-use existing software, but more subtly, 2) it allows for crazy ideas to emerge from brainstorming as not-so-crazy. Not-so-crazy ideas that actually work are sometimes called brilliant ideas.
Let's look at another software example.
Running thought-experiments and building quick-and-dirty prototypes can answer these questions. These are all standard considerations in the software world, but thinking about all the pieces of your solution in this way lets you evaluate every piece while staying in the realm of possibility.
So we get to a set of possible solutions, each with known and unknown components. Now we have to make some divide and conquer decisions. Look at the effort required to investigate the crazier ideas, and the not-so-crazy ideas. Decide upon the solutions that contain those ideas you are willing to test, then set out to test those ideas rapidly and minimally. Test only the questionable or problematic areas first. You are looking for the riskiest components that are also critical or difficult to implement in terms of effort(time) or available skills. You want to eliminate the risk. So you either prove that this risky component is do-able and has a big payoff, or you want to reject it because it doesn't work, or because you have gotten a better idea of how much effort this idea will take to realize and that amount of effort is not justified by the payoff. Then return to the possible solution set and eliminate any solutions that depend on the rejected impractical ideas. But be careful! Impractical ideas mean ideas that you have actually demonstrated won't work given your constraints. Also, be sure your constraints are part of the problem statement, not some constraint you came up with when looking at the idea. If you come up with a new constraint, it should either be rejected, or included back in the problem statement.
After this step, it might start to seem like we've been iterating. We have. We must keep sight of all the steps as we go through the early ones, and we must also go back and make sure the design is updated. We will be revising the problem statement, the proposed paths throught the solution space, and the viable component list.
What we've got at this point is a list of components to build and hook together.
This is where we do the work that looks like work: building the form and function. Whenever we get to a point in the implementation where things are coming together, we Test and Evaluate. But not real integration testing. At this point we are doing Unit Testing, to see that our building blocks conform to the test harnesses and specifications we set up. The implementation phase can also include creating test harnesses. In computer programming, this means testing calls over our interfaces, and testing our building blocks for proper outputs given the inputs decided on in the Divide phase.
Phased Release is almost a methodology in itself. It dictates that users of the final product get to see and play with the components as they are being built and wired together. Phased Release works best when the interim products demonstrate functionality that will be present in the finished solution. Some early releases will simply be documents from the Create and Divide steps. These documents may be too technical for all interested stake-holders, but at the very least, peer-review of these documents can be considered an early release. But it is important to get prototypes and designs out in front of end-users soon. To do this successfully, prototypes should behave like the end product and let the user/tester evaluate the progress and report on any problems. With Phased Release, the prototype is replaced by the actual product, but limited by its state of completion. Still, at each release, the actual product may be evaluated, and design problems may be caught early. Phase Release also gives all stake-holders a sense of the progress.
When building components, a few guidelines will help. Interfaces are how components communicate everything to the outside world. No other implementation details should leave the component except over the interfaces. Think of this as respecting encapsulation, and don't allow any "back doors." An encapsulated component will be swappable with another component that has the same interface but different implementation. An encapsulated component may do magic on the inside, but from the outside, is more predictable, and is less likely to reach out and affect other systems haphazardly.
The types of testing we do in this step: Functionality, Integration, and User Acceptance. Functionality testing tests the components to see if they do the things that the Divide step relegated to components. Integration testing tests the interaction of these components by exercising the components through their interfaces. User Acceptance testing tests whether the integrated components work as a system to solve the problem defined in the problem statement, as interpreted by a user of the final product.
Science is a methodology. Science is about repeatability. Science does not mean that you can't be creative, or intuitive, or even wacky. In fact, most "scientific discoveries" are "lucky." But lucky in the sense that Louis Pasteur meant -- you prepare, you record, you try. And then one day, either through brute force or by some unconscious process, you happen across something new and interesting. You know that it is interesting because you are prepared. Either because you have been looking for it specifically, or because your especial training has readied you for this event somehow. In science, we allow our creative, intuitive selves to drive, to lead, and we record our steps in a log, and maybe even apply a methodology so that we don't go in circles too much. And then when we find the thing that will help us on our solution path, we know how we got there and how we can repeat it. Here is the "Scientific Method":
Each step is a human, social activity. Practicing science means that we make a record of what we do using this methodology so that we can repeat the results and so that others can repeat them, too.
The Scientific Method is applied throughout CDC. Observation can be found in the problem statement, Hypothesis in the brainstorming and divide steps, and here, Experimentation appears in CDC as the testing step. Evaluation then corresponds to CDC's penultimate step, where we consider our experimental results against our problem statement. CDC then dictates that we iterate and recurse until we are satisfied in the evaluation step.
To iterate means to say or do again and again. Software design methodologies usually specify some form of iteration. CDC is iterative in that steps should be iterated over until the problem is solved. Iteration tends to find gaps, and tends to refine the solution. CDC also allows for short-circuit iterations. If you are stuck on a step, you may find another way through the steps that may involve skipping a step, starting at step 1, or just repeating the last few steps. The steps are useful, and powerful, but not canonical. The idea is to keep the energy moving from problem to solution space.
To recurse is to come about to the same place in the path again. In software, recursion has a very specific meaning, which is that a function is entered again from within the function itself. Bottles of shampoo often have instructions that, if read a certain way, could imply infinite recursion: "Wash, rinse, repeat." Within a recipe or instruction, a step tells you to repeat the entire instruction. Software uses recursion to navigate tree-structured data.
What differentiates CDC is its fractal nature. When you subdivide a task, that task requires the CDC methodology itself. CDC uses the word recursion to imply that the structure of the problem space can be viewed as a tree. CDC declares that we solve problems with one methodology which breaks the problem space up into multiple, smaller problems, each of which can be solved by diving in and treating each smaller problem with the same methodology, as though each smaller problem were a fresh, brand new, stand-alone problem. So we recurse into the smaller problem, and then pop back up once we have made enough progress on the smaller problem such that work on the larger, encompassing problem can proceed.
Everything may flow from the plan, but in problem solving, the plan is never complete until the problem is solved.
In manufacturing, the plan can be complete, at least for the current design version. I had an automobile bodyshop teacher who said that you do each step to its fullest, then move on to the next step. I found this to be excellent advice when building something. Cars, like houses, are built in layers. You weld, then grind, then hammer, then sand, then fill and sand, then prime, then sand, then paint. You can't do any welding on the body's sheet metal after you do the final sanding, and you can't do any metal grinding after you have done the final painting. Because welding causes heat distortion, and requires grinding, both of which ruin a nice sanding job, and any work will obviously destroy a perfect paint job. Furthermore, any imperfections in sanding the metal will become apparent when you spray on the shiny topcoats -- even deep scratches in the metal that you have filled with primer and sanded smooth will retain more paint solvent from the final coat of paint, and during drying, that solvent will bubble through and show up as scratches in the finish. So you must complete each step, and complete it to the highest level of quality possible for that step, then proceed to the next step.
However, in problems solving, I don't find this to be the best methodology. Many methodologies advocate iteration over versions (Waterfall, Spiral, Iterative, XP, etc.) It is because you cannot take each step to its fullest if you don't know the effect of that step on the solution or on later steps. Or worse, you can't take a step to its fullest if you don't understand the step or what follows it completely. But we can't get lost in "analysis-paralysis," else the project will never proceed. In problem solving, we must venture into the unknown during many of the steps.
Plans are one way to coordinate work in parallel. Plans are also useful for getting "buy-in" or approval for solution paths. CDC advocates getting buy-in via iterative releases and early evaluation. But planning is still a very useful part of any structured workflow.
Sometimes, early design decisions (i.e. in the solution space) must be taken as constraints in the later problem space. For example, a building's foundation is literally set in stone, so changing the outline of the building later is problematic. But some elements of the later design can easily be modified. Adding windows and doors does not affect the foundation, while going up another storey does. So the foundation, originally created as a solution to the problem of supporting a building as designed, becomes a constraint to later problems -- such as discovering that the view from the upper storey would be ever so much better were it one storey taller, or, worse, if the whole building were moved ten feet to the south.
Planning is one way to work through the problem. In planning, the ideas are worked through using drawings, models and prototypes, so that the solution can be visualized and understood, even before the thing is built.
Many social situations place constraints on changing plans once they are approved. These constraints are placed to prevent changes that could affect safety, acceptance, and so on, due to improperly scrutinized changes.
However, in building a house, the strictures placed on the process by the "planning department" make it expensive to change your mind or creatively alter the plan. The city's planning department should really be called the "final plan approval department," because for every change you submit, they may charge an additional fee from hundreds to thousands of dollars, and hold up approval for anywhere between two weeks to six months. House building is so routine that the people at the planning and zoning departments seem to feel that the building problem has been solved, Le Corbusier be damned. However, even though building design and construction are well known, and could be said to be a form of manufacturing, custom home construction and remodeling are anything but standard and predictable. It is because when building a home, the owners and builders are really problem solving. It has been said that constructing a single family dwelling comprises 100,000 decisions. Many of these decisions have been made before by the architects and builders, but each building is a new situation with new forces: the owners and their desires, the architect's desire to work out new ideas, and builder's desire for challenging, interesting work.
When designing a new car, auto manufacturers first build a "concept car." This is a one-of-a-kind car, built by hand, that will be evaluated by a kind of "User Acceptance" testing -- the car will be trotted out at car shows and rallies, and enthusiasts and critics will give feedback. If a concept car gets rave reviews, the manufacturer will then try to make a version of the car that retains the excitement of the design, but is affordable to mass-produce. (Alas, most of the time, the design trade-offs in this next iteration kill the beauty and excitement of the concept.) But even before a concept car is built, there are a number of prototypes built. The first prototypes are scale models, built from paper, clay, or plastic. Then full-scale models, also built from clay or plastic so that the shape of the body can be added to or shaved away until the designers are happy with the form. From this form, exact measurements are taken and artisans hand-beat metal sheets until they become the flowing curvatures of the body. It is an outrageously painstaking and expensive step to form a car's body by hand, so the designers work out the shapes in the clay prototype, when the form is more easily modified. (Nowadays software modeling is also used.) Building a concept car is in the realm of problem solving -- the current and next steps are not completely known, and the final design is not known, so iteration is used to find the path through the problem space, while affording maximum creative oportunity.
Similarly, when building a house, it is impossible to design the house optimally without truly understanding the site. Water drainage, light from the sun, views, wind, neighbors, soil stability -- these forces must be understood before the house is finished. Time constraints force most buildings to be constructed before the site is understood. One option is remodeling, which is a very expensive way to do iterative design. Another option is to build iteratively, but that is usually only available to owner-builders or those whose budgets can allow the architect to be on-site and making changes during construction. The great cathedrals and castles have often taken this approach, whether originally planned or not. Cathedrals had construction schedules that were so long that design changes crept in as the patronage changed, or the architects and master builders died or got replaced. But other kinds of change happen organically -- windows are added or changed as the structure gets higher, and the views can finally be understood.
Architects have many tools available for prototyping: scale drawings, perspective drawings, computer-generated renderings and fly-throughs, animated solar studies, and scale models. Of course, experience helps a lot. The visual effect of large masses on our perception of space and proportion is such that buildings cannot be understood until after they are built. So a good architect uses knowledge of buildings that have been built before that have similar massing to guide the design. In this sense, everything that is built is a prototype for everything that is built after it. This is why CDC emphasizes understanding the problem first and understanding case studies of previous solutions.
CDC is useful, then, for solving problems that are more in the realm of the unknown. Software today lives mostly in this realm. Once we've built a working version, we will move on to manufacturing and maintenance. Before then, use CDC to move your reality from problem space to solution space.
Thank you for taking the time to read Create, Divide, and Conquer. I hope that you find it interesting, useful, and hopefully, transformational in some small way. I would be grateful for your thoughts, comments, and experiences after reading about CDC. [Contact me by eMail]
I would like to thank Wendi Olson for being an early reader, booster, and brainstormer on this material, and to thank Nate Rushfinn for being an early reader, and for being there very early for me, teaching me his discoveries on many of these methods.
Laramie Crocker
Berkeley, CA
Early version written in Brownsville, VT
Document version: 2010-04-12
(posted at 6:24 AM, after a long editing session, and a walk, and incorporating a few more ideas.)
Copyright © 2010 Laramie Crocker