In the spirit of the New Year, we thought we would share our list of top things that you should consider putting on your technology’s roadmap for 2009.   

1. Develop the ability to rollback: If you can make only one change to your product and process in 2009 and you don’t currently have the ability to rollback, this should be at the top of your list.  Being able to push code changes and then pull them back from production in the event of a problem will save you more customers and more effort than any other single item.
2. Break changes into smaller pieces: There is almost never a need to redesign the entire site or service at once.  Break it into parts and take it one piece at a time.  This will be lower risk and give you an opportunity to learn along the way.
3. Remove SPOFs:  Commit to removing all single points of failure in your architecture.  Single servers, firewalls, load balancers, power supplies, etc should all be listed and tackled one at a time until they are all eliminated
4. Remove synchronous calls:  Having one service call another service in a synchronous manner causes a multiplicative effect of failure.  Five synchronous calls on servers with five 9’s availability (99.999% uptime) leads to a maximum of 99.995% for the system.  Eliminate synchronous calls wherever possible and create fault-isolative architectures to help you identify problems quickly.
5. Incent a culture of excellence:  Hire the right people and hold them to high standards. Set aggressive yet achievable goals and motivate them with your vision. Be a leader.  
6. Develop a disaster recovery plan: Disasters happen, no one expects an entire data center to be down but things like that happen.  Plan on it and start making changes today to keep your services up and running in the event of a disaster.
7. Develop quality into the product from the start:  Don’t expect QA to ensure quality is built into the product.  We’ll post more about this in the New Year but quality starts much, much earlier in the product development life cycle.
8. Split your application or database:  Start this year thinking about how to split your application and database.   We recommend our cube model for both because working on all three axes gives you unlimited scalability in both your app and database.
9. Start Logging:  As we discussed in a recent post, start logging your application data but follow our three key guidelines - 1) logging must not impede the performance of the application 2) use a common framework and 3) look at the data
10. Celebrate your success:  Take time now and throughout the year to look back and congratulate your team and yourself.  If you want to foster creativity on your team, celebrating victories is a great way to keep the energy up on your team.  If you are getting ready to present your 2009 goals to your team, we recommend that you start by focusing on the amazing accomplishments that you have had in 2008.

Wishing you and your teams a great New Year!   

-The AKF Team

This is the second on our two part post on the Principles of War and our interpretation of them relative to the business world.  The 9 US Principles of War (derived from von Clausewitz’s essay “Principles of War” and his book “On War”) are Objective, Offensive, Mass, Economy of Force, Maneuver, Unity of Command, Security, Surprise and Simplicity.  We will discuss the last four of these in this post.

Unity of Command.  The US Armed Forces definition is that for each objective, there should be a single owner or commander and that the forces necessary to achieve that objective should be placed under the authority of that commander.  In the business world, this does not mean that you should slice your technology, client services and product teams into separate groups under each general manager or objective owner.  Rather, it means that the person responsible for achieving some business objective should have the authority to direct the resources necessary to achieving that goal.  These resources could be set up in project teams that respond to the needs of the objective owner, or they could be “dotted lined” to the individual.  The key here is that for any objective there should be a clearly defined and empowered owner of the objective.  

Security.  Security enhances freedom of action by reducing vulnerability to hostile acts, influence or surprise.  While you may jump immediate to “information and technology security” implemented as policies within firewalls and such, we believe this has a much broader meaning.  Portions of this principle speak to your actions and efforts to get early warning of competitive threats – not just the threats afforded by hackers and the like.  What are you doing in an ethical fashion to find out how your competitors are responding to your actions?  How do you monitor the strategies and products of your competitors?  How well do you know your competition?

Surprise.  Strike the enemy at a time or place or in a manner for which he (or she) is unprepared.  This principle is the hardest to achieve within the business world, but can have incredible results when it is in fact achieved.  Surprise is achieved when a struggling computer company releases a portable personal music device such as Apple did with the iPod.  Sony, the creator of the portable music device phenomenon was taken completely by surprise and had previously never even considered Apple a competitor.  Surprise, therefore, does not have to be a principle you apply to those you currently consider to be your competitors – it can be applied to markets tangent to the ones in which you currently operate.  Surprise can also manifest itself as a change in approach such as Nintendo’s approach with the Wii.  While Microsoft and Sony focused on more complex graphics and processors, Nintendo took the surprise approach of using less sophisticated graphics and processing power (thereby offering a lower initial price) and focused their approach on a revolutionary game controller (the Wii Nunchuk and motion system). 

Simplicity.  Prepare clean, uncomplicated plans and concise instructions that ensure thorough understanding.   This is perhaps the most easily understood within the business context of all the principles of war.  Simply stated, you need to make sure your plans, orders, and objectives are understood by all parties and are unambiguous.

Summarizing the nine principles from our two posts, then leave us with 9 Principles of Business in a slightly restated fashion:

1. Create Clearly Defined Aggressive But Achievable Goals (Objective)
2. Be First to Market and Aggressive in Your Implementation (Offensive)
3. Align Your Companies Organizations with Your Objectives (Mass)
4. Employ Your Teams and Organizations in Accordance with Their Capabilities (Economy of Force)
5. Maintain Business Flexibility but Do Not Oscillate Constantly (Maneuver)
6. Clearly Define Ownership of Objectives and Empower those Individuals (Unity of Command)
7. Sense and Respond to your Competition (Security)
8. Surprise your competition with your timing or approach (Surprise)
9. Constantly Communicate and Simplify Your Plans (Simplicity)

Many authors have previously described the relationship between business and war and we believe that the most successful businesses approach their operations as would General Douglas MacArthur when he claimed that “In war, there is no substitute for victory”.

Carl von Clausewitz offered several tenets of war in his essay “Principles of War” and later expanded upon those in his book “On War”.  Many armed forces throughout the world have taken portions of these tenets and adopted them for their own use.  This post is the first in a two part series relating the 9 US Armed Forces Principles of War to your everyday business activities, strategy and tactics.  The 9 US Principles of War are Objective, Offensive, Mass, Economy of Force, Manuever, Unity of Command, Security, Surprise and Simplicity.  We will discuss the first 5 in this post and the next 4 in a subsequent post.

Objective.  The US Armed Forces definition is to direct every military operation toward a clearly defined, decisive and attainable objective.  We think this is pretty self explanatory and includes concepts about which we’ve previously blogged such as the need to set aggressive but achievable goals.  The most important aspects of “Objective” as applied to your business are for your goals to be clearly defined, well understood, measurable and attainable.

Offensive.  The military definition is to seize, retain and exploit the initiative.  The business definition here is found by looking at what Offensive implies – specifically that it’s all about time to market and getting the right features, products and services out and adopted first.  Being first offers the best chance at achieving virility within the market, and creating a viral marketplace or product is the military equivalent of seizing the high ground.

Mass.  The military definition is to mass the overwhelming effects of combat power at the decisive place and time.  Mass here in military terms is different from the concentration of forces which may not be desirable.  Combat power refers to all the aspects of military power from infantry and armor, to field artillery and other combat multipliers. The business equivalent is to ensure that your business units are aligned with your greater business objective and that they are contributing to it properly.  Your technology, product, marketing and finance teams should all realize and be contributing to the core objectives necessary to win your business battle.  If you wish to win quickly, they cannot be marching to separate agendas and they should not be fighting with each other.

Economy of Force.  This one can be confusing, but within the military definition is a reference to “No part of the force should be left without a purpose”.  The military definition also hints that every part of the force should be used in the most effective way possible.  Goals and objectives are again part of this, but more importantly you should be able to answer the question of whether you are using the right team for the job at hand.  Not only should you ensure that every organization has a purpose directly relating to your most important initiatives, you need to ensure that they are the best team to have those specific goals and objectives.  Client Services and Customer Support teams might be useful in helping to QA new products but allocating them 100% to such an endeavor is probably not the most leveraged use of their time.  Conversely, forgetting to include Customer Support or Client Services in any product rollout is a failure to employ a very important part of your “combat power” in achieving product success.  While its useful for engineers to understand customer needs and complaints, allowing more than 5 to 10% of their time to be taken up by such activities is a costly endeavor relative to your future product needs.

Maneuver. Place the enemy in a position of disadvantage through the flexible application of combat power.  This one relates to how flexible you are in your product delivery lifecycle, and whether you are set up to respond to your competitors actions in the marketplace.  This IS NOT an argument that you should abandon products in flight and constantly change your strategy.  Constant change in strategy is a clear indication of a management team incapable of defining a winning path and it’s a early indication of likely future failure.  You should be flexible, and changing features or making course corrections a few times a year is appropriate.  Ensuring that your product delivery processes allow you the flexibility to change (with the additional cost that implies) is critical to success.  But constant change is not a strategy – it’s a recipe for disaster.

With the economic downturn in full force, you are probably spending a great deal of time thinking about how to cut cost, reprioritize revenue generating features, or delivering more in 2009 with less resources.  You might think now is not the time to care about “creativity” and “energy” but we think this it is even more important.  Having a team that is fully engaged with all of their creative forces focused on your business is crucial to achieve any of those other objectives.  The way to achieve this is by creating an environment where people know where they stand in terms of performance, get to own deliverables, can openly question decisions or standards, and show each other respect.  

A couple ideas that we have either read about or seen in practice in organizations are team or individual training events, four day work weeks, allocated time to work on personal interests, self selection of features/stories, and mentoring.  Training can take the shape of many different forms including formal classes at universities, external workshops (WARNING: self-promotional plug….such as our Technology Workshop), or internal classes taught to each other by members of the team.  Everyone knows different things, sharing this knowledge is good for both the team as well as the presenter, giving her practice explaining technical items verbally  and ensuring she knows the subject completely.  

Mentoring is another low cost method of helping foster a more open and creative environment.  Pairing junior and senior engineers together provides both parties the opportunity to practice different skills.  Additionally, it helps facilitate what are likely two different groups to begin a dialog.  Mentoring can be extended in many different forms.  Ask the CEO to take a different engineer as a mentee each quarter, meeting with them for lunch or breakfast every second or third week for the quarter.  This is a great way to remind the top executive to appreciate the engineers and gets engineers exposure to the business challenges that the CEO faces daily, a real win-win proposition.

Some of the more radical approaches for developing a creative environment are already well documented by some very popular companies including Google and 37Signals.  If you haven’t read the 37Signals book, we recommend this as a great source of ideas for fostering a creative and unique environment for your team.

That is the question that has caused debate for many years among operations and engineering staffs.  We’ve recently read a couple very well written and well thought out articles on this topic and wanted to offer our ideas on the debate.  The first article is by Todd Hoff from HighScalability.com who advocates in Log Everything All the Time, as the title implies, that everything should be logged for potential use.  Todd has another article describing Facebook’s open source Scribe, Product: Scribe - Facebook’s Scalable Logging System, where he observes the fact that Facebook must agree with his logging approach by virtue of their development of this product.  The other article titled, The Problem With Logging by Jeff Atwood of CodingHorror.com, argues for a more tempered approach.  Jeff summarizes his position as “Start small and simple, logging only the most obvious and critical of errors.”  

Our position is squarely in the camp of log everything but with a few caveats.  These ignore-at-your-application’s-peril cautions are 1) logging must not impede the performance of the application 2) use a common framework and 3) look at the data.  Let’s go through these one at a time.

1) Logging must not impede the performance of the application – As Jeff points out, “logging isn’t free” and we agree with that but we would add that the potential benefit of the data outweighs the resource cost, unless it negatively affects performance.  Get ready for one of our repeating themes:  Do it if the BUSINESS benefit of logging outweighs the cost of logging.  Most web / application servers are not utilized completely because most teams don’t know precisely the performance parameters and resource constraints of their application, especially as it changes with each release.  If you are fortunate enough to be in an organization that really understands the bottlenecks and performance of the application on specific hardware, more than likely there is a single resource that is the bottleneck, i.e. memory, i/o, or CPU.  Your logging service should not put further demand on a constrained resource, all surpluses are fair game.  And what should go without saying is all logging must be done asynchronously.  Losing a log event is acceptable but impacting a transactional event is not.

2) Use a common framework – Chose or build a common framework that is used throughout the application and that includes common definitions.  Just like definitions of Priorities and Severity for bugs are defined, logging definitions must be determined and adhered to.  Code reviews are a way to ensure common usage.  Data being sent to five different files in different formats defeat the purpose of logging, common usage, format, gathering, and analysis is where the payoff is realized.

3) Look at the data – Logging tons of data, and when we says tons think of Scribe that claims to handle 10’s of billions of messages each day, looking at this data is completely overwhelming.  But looking through some mechanism automated or manual is mandatory for the benefit to be gained.  As Todd points out there are products like Hadoop to help process the data into viewable and actionable information.  Jeff makes the point that “the more you log, the less you find”, but our point is that by the time you know you have a problem and need to inject logging you’re too late.  Properly logging and analyzing of the data will identify the problems early and make diagnosis easier.  We think products such as SCAMP application monitoring software are excellent for creating an easy way of seeing inside the application.

As long as you avoid the pitfalls stated above, we feel that logging can be a very beneficial addition to your quality assurance, scalability, availability initiatives.  We highly encourage you to read all the articles cited, both HighScalability and  CodingHorror are on our must-read list of blogs that we subscribe to.  As always let us know what you think.  I’m sure we have not heard that last of this great debate.