Mitch Pronschinske

In just over six years, Apache Maven has become one of the most coveted tools for project build and reporting management. It’s been five years since the release of Maven 2, and now the Maven committers have released the next landmark version of the software. Incredible Improvements in Little Time It didn’t take long for Maven to become one of the most respected and desired tools in computer engineering. However, the product only continues to improve as Maven 3 is now available for those needing this extra computing power. The whole system just gets more and more interesting as time goes on, and that is exactly what people are most interested in when they look through different software options that are open to them. They just want something that they know will add to the amount of work that they can get done. We took some time to speak with the visionary founder of the Maven series of products, and we got some answers directly from him about how these products work and what kind of updates we might expect in the future. Believe it or not, just sitting down with him and getting some of these answers was a big help getting us to a place where we better understand the product. There is so much buzz and excitement about Maven 3 right now, and there should be. However, we wanted to hear directly from the creator of it to see which features we should be most excited about. We are so happy that he took the time to speak with us and review his innovative product. Hear what we learned today directly from the creator. The first beta release of Maven 3, which is now complete after seven public alphas, was released this week. Maven founder and Sonatype CTO Jason van Zyl answered some questions for DZone about Maven 3 earlier this month. Below are the main new feature categories of Maven 3. Drop-in Replacement Users of Maven 1.x may remember the bumpy transition to Maven 2 because of several fundamental changes. The Maven committers remember too, and they’ve put a lot of extra work into providing backward compatibility and making Maven 3 a simple drop-in replacement for Maven 2.x in most cases. van Zyl says this was “very difficult given how much of the internals we’ve changed.” Apart from fixing problems with duplicate dependency and plugin declarations, no changes are needed for your POMs. They’ve made the command line fully compatible between 2 and 3. Polyglot Maven Polyglot Maven is not a part of Maven 3 per se, but it is a tool from van Zyl’s company, Sonatype, that can be integrated with Maven 3 via an extension point. The extension points are a new feature in Maven 3 that support tools such as Tycho, Polyglot Maven, and Maven Shell. As you’ve probably guessed from the name, Polyglot Maven supports dynamic languages and is trying to provide first-class POM-mapped DSL (Domain Specific Language) support for Groovy, Scala, Clojure, Ruby, Xtext, and YAML. Polyglot Maven currently supports YAML. This is a welcome feature for developers who find the original XML format annoying. If you don’t, no big deal. Van Zyl also says it’s important for these DSLs to have repository interoperability and tooling to leverage M2Eclipse. M2Eclipse Maven 3 has changes related to embedding that make it work a lot better inside of M2Eclipse (the first Maven integration plugin for Eclipse). Maven 3 is now capable of a 200 to 300% performance boost while running in this plugin environment built specifically for Maven and the Eclipse IDE. M2Eclipse will provide some extra XML metadata in the Maven POM, which M2Eclipse only recognizes. This is one feature that enables high build performance. M2Eclipse also downloads all sources automatically and has a single-click new project creation feature for any of your dependencies. Maven Shell The Maven Shell is another extension point. It is Maven embedded in a long-lived shell process that caches parsed POMs, avoids start-up costs when invoking Maven repeatedly, supports Maven Archetype integration, provides Nexus integration, includes a built-in help system, and on Mac OS X, provides Growl support. Van Zyl says typical cases will see a 50% reduction in build times. Version 1.0 of the Maven Shell integrates the make-like reactor mode that builds only the modified modules. Support for project workflow, Hudson, Tycho, and Polyglot Maven are also present. Other improvements Developers working in multi-module or multi-pom projects won’t have to specify the parent version in every sub-module in Maven 3. Instead, you can add version-less parent elements. Maven 3 will also be able to see which POMs supplied which artifacts. In M2Eclipse, you can then deselect a certain contribution and select others. This is made possible through Maven 3’s decoupling of execution plans and execution. Maven 3 also includes extension points (mentioned above), which allow developers to hook up to different extension points instead of subclassing a plugin to alter the plugin’s behavior. You might, for example, have an extension point to alter how web.xml is processed through the WAR plugin. The source code in Maven 3 uses Google Guice for dependency injection and Peaberry to add OSGi capabilities to Guice. The whole dependency resolution is refactored into a standalone product by Sonatype called Mercury, for which Maven 3 is a client. Believe it or not, the Maven 3 codebase ended up being 1/3rd smaller than Maven 2. Maven 3.1 Looking toward the next release, Maven 3.1 will include a security manager with the settings.xml implementation as the default. Sonatype is planning an implementation that interacts with Nexus. Maven 3.1 will also introduce POM mixins, which make the configuration more maintainable and portable. Mixins will help solve the problem in Maven 2.0, where sharing configuration could only be done via inheritance. POM mixins are a type of POM composition that allows parameterized POM fragments to be injected into the current POM with a simple reference. References: "What's New in Maven 3" - Eingestellt von Reikje, DZone interview with Jason van Zyl, and "EclipseMagazine Interview with Jason van Zyl on the Maven Ecosystem"

This article was originally published on 3/19/15

Java is changing the world around us in major ways. Here is how.

A New Programming Language Has Grabbed the Spotlight Programmers universally stand up and take notice when a new programming language takes hold. Sometimes, those programmers are eager to give it a try and see what it is all about. In other cases, they may pan the whole thing as not being the right fit for what they need out of it. Either way, there is no question that programmers will at least take a look at what this new language is about and try to determine on their own if it has real value for them or not. Right now, that programming language is F#. Microsoft bills this as a programming language that may be used to create "robust, and succient" code. They say that it makes things easier on programmers and that the number of applications where this language may prove useful is nearly endless. Indeed, it does seem like there are many upsides to using this particular programming language. F# is a relatively new language for Microsoft's .NET framework that is quickly gaining momentum in the financial and scientific software development communities. IntelliFactory believes that it can forge a new path for F# in the mainstream web application space. In less than two weeks, the company will debut its WebSharper platform for writing web apps in F# that are easily compiled into JavaScript for the client-side. F# is a three-paradigm language that uses imperative, functional, and object-oriented programming (class-based). New programming design approaches that are not easily expressed by just objects are possible with F#. The functional programming capabilities can also lead to some mind-bending code. F# is a strongly typed language that uses a type interface so that data types don't need to be explicitly declared. It includes a functional programming component that supports eager evaluation. The language also uses pattern matching to resolve names into values. In ASP.NET development, F# also offers productivity advantages over VB and C#. F# is different because it is statically checked and type-safe. It addresses weaknesses in ASP.NET development like untyped values, complex form construction, and using strings for IDs and method names that connect markup with code-behind (class-files). Writing a web application in F# on the WebSharper platform can be less time consuming if a developer is not great at writing web apps in JavaScript. Through WebSharper, developers can write a web app using a large subset of F# and .NET core libraries and then just let WebSharper map the code to JavaScript. WebSharper can integrate with ASP.NET applications, but it is different from the standard approach because it builds applications from miniature web pages called "pagelets". The pagelets correspond to functions on the client-side and they are automatically translated into JavaScript. WebSharper supports a wide range of JavaScript libraries, making it easy for developers to optimize their code in whatever way they choose. jQuery, qooxdoo, Flapjax, and Yahoo UI are all supported by WebSharper. The leap from F# to JavaScript is manageable because both are functional languages that support lambda expressions and closures. Another unique part of WebSharper is a "formlet". A formlet is a special pagelet that provides form functionality. Formlets in Web Sharper run and validate on the client, submitting their result to either a client- or a server-side callback. jQuery auto-complete WebSharper will be released on February 8th. Pricing has yet to be determined, but customers will be required to pay for support. WebSharper comes fully integrated into Visual Studio. Want to get ahead of the curve before F# really takes off? DZone's Essential F# refcard can jumpstart your exploration into Microsoft's newest language. The card was authored by Chance Coble and Ted Neward. What People Love About F# F# is still relatively new as far as programming languages are concerned, and this means that there is still a lot to learn about it. That said, there are people from all over who are clamoring for it already. A few of the reasons why they love it include: It is Interactive This coding language responds well to virtually any challenge that you throw at it. It is interactive in the sense that it updates to the latest developments and challenges that you throw its way. It will move along with you as you work through the various steps required to get it working exactly how you want it to. For most people, this is a top benefit. They don't have time or energy to slow down and try to get their programming language to catch up to whatever it is that they are doing. Fortunately, they don't have to worry about this as much because they can simply use the F# system to take giant leaps and bounds in the right direction. It is User-Friendly There are many pieces of programming code that get complex and in the weeds with the programming side of things that they are not efficient tools to work with. Instead, it seems like those programming languages are practicing a bit of gate-keeping in the sense that they don't allow non-programmers to make much progress with them. The last thing in the world that true developers want is for people to feel like they are unable to do their programming. The beauty of programming is that you never know who might create the next big thing. However, this can only happen if the tools they use make sense to them. Fortunately, the F# programming language is adapted for people who are not necessarily programmers themselves. A Flurry of New Apps May Come From WebSharper WebShaper is making it easier for anyone to get involved with creating their own apps for the Internet via their platform. You don't have to have any extensive knowledge of coding or app development in order to get involved. In fact, the entire platform is designed with beginning users in mind. The goal is to make things as simple and straightforward for them as possible. A great way to make this all happen is to use WebSharper to start creating today. People say that it is: Easy to get on and get started Great for beginners Encouraging people with new and interesting ideas In short, the WebSharper platform is exactly what people are looking for when they are first getting started in their app development journey. It gives them the space to be flexible and interesting, but it is also not overly complex. Think of it as the training wheels for app development. You can make a lot of mistakes with training wheels and still be just fine. You are still learning at this time, so it just makes sense that you would want to try out your ideas in an environment where it is safe to do so. WebSharper is that space. If you are still uncertain about how to get started or if app development is right for you, poke around the platform for a while and see what other people are up to. What you may be surprised to find is that there are many people just like yourself who have some basic questions about how things work and what steps they should take next. This should give you some confidence that you are among people at a similar skill level as yourself.

HornetQ 2.0 broke records and defeated top-ranked messaging services in benchmark tests. Why wasn't it widely adopted? Software vendors make all kinds of claims about their products, but what developers care about is proof. When testing a new product, it's important to see how it stacks up against its competition. For years, researchers at UT Darmstadt have compared the performance of message-oriented middleware servers based on Java Messaging Service (JMS). In 2010, the SPECjms2007 benchmark record was smashed by HornetQ, an open-source enterprise messaging system from JBoss. At the time, SPECjms2007 was the industry-standard benchmark for JMS systems. While the benchmark has since been retired, it provided a framework for detailed performance analysis as well as workload and performance metrics for the products being analyzed. The framework considered all components that were a part of the application environment, including hardware, JMS server software, JVM software, database software if used for message persistence, and the system network. It used horizontal and vertical topologies as scalability networks. During the test, HornetQ 2.0 was directly compared to Apache ActiveMQ 5.3, another JMS system that was known for its speed at the time. To ensure consistency, both messaging systems were tested on an IBM x3850 system utilizing identical OS setups. Not only did HornetQ completely decimate the records set by ActiveMQ, but it managed to smash all SPECjms2007 benchmarks at the time. While HornetQ 2.0 didn't have the reputation of ActiveMQ, it had proven that it was faster than every other message broker on the scene. In fact, HornetQ 2.0's scores were a staggering 307% higher than any previous results published by SPECjms2007. Shortly after the tests, DZone interviewed Tim Fox, who was HornetQ's project lead. During the interview, he discussed the main factors that contributed to HornetQ's impressive performance. HornetQ utilized Netty, a low‑level Java NIO library written by the author of Apache MINA. The JMS system also had a secret weapon, which Fox explained: "This system is a very high‑performance journal, which is basically written in 99‑percent Java. But we also have a small layer of native code that's accessed by JNI. And what this does is, when you're running on Linux, it detects that you're running on Linux, and then it automatically enables a bit of native code, which allows us to pass into Linux asynchronous file IO. And basically, what this does is it allows us to get much faster persistence than would be possible in pure Java." Netty Architecture After HornetQ's record-smashing benchmark test, the messaging system released several updates. Later that year, HornetQ released its 2.1.0 update, which refined the original release. It was followed up by several more updates added new features, like Vert.x integration, and strove to improve upon the original JMS system. Competition is a vital aspect of innovation, and when HornetQ broke records, it brought plenty of new competitors to the scene. These competitors included RabbitMQ, a messaging broker written in Erlang, and SonicMQ, a message queuing system that's now known as Aurea Messenger. In spite of HornetQ's incredible speed, however, it failed to surpass its biggest competitor. Near the end of 2014, the codebase for HornetQ was donated to the Apache ActiveMQ community. Today, it exists as an Active MQ subproject called Artemis. At one point in time, HornetQ was faster than any other available enterprise messaging system. It offered a full enterprise feature set that could easily stand side-by-side against ActiveMQ and even offered some features that ActiveMQ was lacking. Since HornetQ was not built around JMS, it was also protocoled agnostic, which was another major advantage People that used HornetQ had plenty of praise for its features. Its management API made it easy to work with, and its NIO optimizations were appealing to many users. HornetQ also had detailed documentation, making it easy for users to learn what they needed to know. It’s clear that HornetQ 2.0 had many advantages, so why did it fail to find a wide audience? It's possible that the JMS struggled because ActiveMQ was already widely adopted at the time of its release. ActiveMQ fell short of Hornet 2.0 in those benchmark tests, but it still had a high adoption rate and name recognition. Even though HornetQ got plenty of attention when it broke benchmark records, future updates didn’t receive the same level of coverage. Because of this, many potential users may not have been aware of the new features that HornetQ added. Many people were drawn to ActiveMQ because of its reputation as a stable messaging system. Users were unwilling to take risks on a lesser-known system like HornetQ and continued to stick with ActiveMQ, which had already proven that it was reliable. Some people were unwilling to trust sensitive data to a system that they didn’t know much about. Beyond that, open-source JMS providers are highly reliant on the surrounding community. With an active community of users, it's easier to troubleshoot issues and find solutions to problems. Since ActiveMQ had more community interest than HornetQ, it eventually fell behind its competitor. Today, ActiveMQ serves as a speedy communication bridge that allows for messaging between two or more applications, even if they're written in a different programming language. It's still known for its speed, but it's also known for its reliability. The ActiveMQ message broker allows for communication between system components even when one of the components is down. ActiveMQ was able to correct the issues that lead to its defeat in those benchmark tests, adding new features and improved functionality. It can be integrated with the Apache Camel Framework, making it possible to read and write messages between the two systems. New features have been added with every iteration of ActiveMQ, which have helped to improve its speed and its flexibility. These features include STOMP and UDP support, mirrored queues, and broker clustering. It's used by many major companies, including SquareTrade, Cox Communications, and ClearScale. Even though HornetQ is no longer updated, it lives on in ActiveMQ, the JMS it defeated in its record-breaking benchmark test. You can see its influence in other JMS providers as well, like ActiveMQ's main competitor, Kafka. HornetQ may not be in use today, but that doesn't mean it should be forgotten. Modern developers can still learn from the way HornetQ enhanced Java with just a few lines of native code. Artemis, the ActiveMQ subproject that’s specifically based on HornetQ’s code, is intended to be a next-generation ActiveMQ broker. It currently serves as a successor to the ActiveMQ classic. According to Apache’s Artemis roadmap, it could eventually serve as the next significant version of ActiveMQ. Currently, Artemis offers protocol support for Stomp, OpenWire, MQTT, and more. It offers features like replicated message store and message scheduling. In the future, Apache intends to add additional features to enhance Artemis, such as pluggable dispatch policies and configurable IO exception handling. Even though HornetQ is no longer updated, it lives on in ActiveMQ, the JMS it defeated in its record-breaking benchmark test. HornetQ may not be in use today, but that doesn’t mean it should be forgotten. Artemis may eventually become the next-generation messaging broker that many people hoped HornetQ would some be.

The increasing complexity of the software development task has been answered by an increasing number of tools for managing that development. Here's our top ALM solutions.

Necessary tools, environment, and access to learning can inspire developer teams and productivity. Discover top motivators for developers and IT professionals.

In this post, learn how to simply install Joomla on one of Microsoft's SQL-based backends with the setup utilities provided here.

Some have probably been expecting it for a long time, and this week it finally happened. Apache Harmony, an open source cleanroom implementation of Java was moved to the Apache Attic, where inactive projects are sent. The project management committee voted 20 to 2 in favor of discontinuing the project. One of the votes against moving Harmony to the Attic was PMC chair Tim Ellison, who thought it was too early to deactivate Harmony. But Harmony was probably already dead and buried once it's primary corporate sponsor, IBM, switched its support to OpenJDK last year. Android has not gotten invovled in the Harmony project recently because of their ongoing lawsuit with Oracle. Developers may still use the code while it resides, inactive, in the Apache Attic. Here were some comments from last year when most predicted the death of Harmony: "Well, pragmatically I would prefer one great open source JVM, rather than multiple average ones. So as long as OpenJDK is still GPL, I see no reason to cry over this. Reality is that I don't know of a single project going into production using Harmony. " --Jacek Furmankiewicz "Google has single handedly turned around the disaster that was J2ME. Assuming they'll eventually sort out the patent mess (and given the stakes, they will) that removes from the equation all the partners that had very little to bring to the table when it comes to mobile Java. Meanwhile, Oracle and IBM need to demonstrate through actual technical innovation that they are still relevant in the Java world. Last time I checked, the enterprise Java world was dominated by things like Spring (under the Apache license) rather than any JCP efforts. Oracle bought an empty shell. Filing patent related lawsuits left and right is probably not going to be very helpful since that tends to scare away customers. So, I'm hoping that this will end pretty quickly. Once it does, all parties can get back to moving the agenda forward on the run-time, language, and APIs. There is a lot of stuff that needs to start happening there and if Oracle won't do it, others will do it for them. In a nutshell, that's why Google is shipping Harmony rather than CDC. I'm pretty sure Google would have preferred to stay in the Sun community a few years ago if only Sun was not being so unreasonable." --Jilles van Gurp "For me, the question is: what to we, as coder, expect from Java? I don't think I will ever use a self patched SDK/openSDK in any production; I even doubt I would ever work in a project which would like todo that. Oracle might be the bad boy here, but -man!- they know techology. I strongly believe, that the SDK will be less stagnant in performance/features and lots of those 10 year old problems in Bugzilla will finally be tackled. Sun let the "open" part of Java start smelling and people started to invest significant time in non-Java languages like Scala and new ways of dealing with partitioning of services aka OSGi containers. Since JVM 1.5 they were not really able to focus this community power to anything bigger then some lame syntactic sugar and a DOA flash clone. Harmony is a nice place to play around with an open JVM, but I think this job moves more over to the more general LLVM. And I don't think that I want to bet my (professional) future on the fact that Google has to step always in when the rest of the industry has just a bad haircut day. IBMs move is logical. Whatever Harmony is or was, the impact was already limited. You simply can't build such infrastructure without more people building it. One company alone wouldn't push Apache or Tomcat, and any serious openJDK shouldn't do either." --Igor Laera Let the conversation now continue.

the approval of the blueprint container specification by the osgi alliance enterprise expert group (eeg) inspired members of the eeg to start an open source project centered around implementing the blueprint spec and other technologies for osgi applications. in september the apache aries project was born in the apache incubator. the purpose of the apache aries incubator is to create a new community of people interested in building enterprise osgi technology geared toward the application programming model. for an introduction to the history and the purpose of the aries project, dzone interviewed ian robinson, a distinguished websphere engineer and a member of the osgi eeg. robinson is at the frontrunner for the apache aries project and has begun using its technology for ibm's websphere application server. dzone asked robinson about the factors inspired the aries project. robinson said, "from a standards direction, the work of the osgi alliance eeg was to define a set of specifications that would form part of an enterprise profile for osgi." he says the eeg has approved several specs for technologies that allow osgi applications to consume existing java ee technologies like jta, jpa, jndi, etc. "the purpose of the eeg was not to try and define competing specifications but to take what exists already in the java enterprise space and define how those technologies become consumable for applications running in an osgi framework," robinson said. robinson also observed some point efforts starting up inside existing apache projects that didn't have an enterprise osgi home to host them. one example was an implementation of the blueprint container spec, which started out inside apache geronimo , an open source java ee application server. robinson said that developing a blueprint implementation in geronimo made sense since the app server could use the it, but that didn't provide much visibility of the blueprint work outside of the geronimo project. robinson and his collaborators thought it would be a better idea to start a new incubator project who's primary focus was enterprise osgi, form a community around it, and then gather a set of osgi technologies in that new project so that other projects like geronimo, felix karaf, and servicemix could use that technology in their own runtime environments. apache aries is not an effort to build a new enterprise application server or a new application integration runtime. robinson says the purpose of the project is to build components like the blueprint container that can be used by those enterprise application servers. apache geronimo is currently working on consuming the aries blueprint container and apache felix karaf , which is the kernel of an enterprise integration runtime, is already consuming the aries blueprint container. in its three month existence, the apache aries incubator has already been successful in building a sizable community. including robinson, there are currently 43 committers distributed across a wide variety of companies. robinson says at the end of an incubation period, an incubator is considered a success and a top level project if it builds a vibrant community, and aries is well on its way with companies like red hat, progress, ibm, and sap represented. right now, no timeframe has been determined for when the aries project intends to graduate from incubation. robinson says the community will decide when they've done enough work to become a top level project. dzone asked robinson the most important question for any apache project: 'how did the project get its name?' robinson explains: "we started thinking in ibm about the aries project way back in early april when the blueprint work started in apache geronimo. i mentioned that geronimo is a consumer of blueprint, but not the obvious project to develop it - we thought back then that what we needed was a new incubator for the blueprint container and other enterprise osgi technologies. aries is the star sign for that time of the year - simple as that." hence the logo for apache aries is the ram. ibm's websphere application server v7 already uses some of the technology in the apache aries project for its open alpha , which helps deploy enterprise applications as osgi bundles. to get involved with the apache aries project, you can visit their "getting inolved" page.

Learn about the benefits of a mature project, the pitfalls, and how to qualitatively and quantitatively analyze a project's maturity level in the correct context.

Find out what the most popular Java / JVM language web frameworks are in 2015.

Welcome to the Developer Personality Quiz! After some inspiration from Alistair Doulin's Programmer Personality Test, I've created my own more extensive personality test for developers. Below are links to the short and full versions of the quiz. The short version is 25 yes/no questions (approx. 4 min.) and the full version has 62 (approx. 9 min.). Developer Personality Quiz - Short Version Developer Personality Quiz - Full Version Hope you have fun with this and share it with your friends and colleagues!

See an extensive comparison chart below to understand which type of mobile app is the right one to build in your organization. This article was originally published in DZone's 2014 Guide to Mobile Development.

Hyphenation is always a big question, and the fact that it can vary for the same word causes significant confusion. This article will give you some tips and common usages.

Even among the large IT/development publications, I see inconsistencies in the use of the word front-end. Is it hyphenated or not?

API Management solutions must combine an API Portal (for signing up developers) with an API Gateway (to link back to the enterprise). But where do these come from, and what is the relationship with SOA? To answer these questions, first let's look at a bit of history: In the 2000's, we had the SOA Gateway and the SOA Registry, working hand-in-hand. This was "SOA Governance". The SOA Registry (with a Repository) was intended to be the "central store of truth" for information about Web Services. It was often the public face of SOA Governance, the part which people could see. Usually the services in the registry took the form of heavyweight SOAP services, defined by WSDLs. The problem was that developers were often forced to register their SOAP services in the registry, rather than feeling that it was something beneficial to them. Browsing the registry was also a chore, involving the use of UDDI, also a heavyweight protocol (in fact, it was built on SOAP). Fast-forward to the current decade, and we find that the SOA Registry has been replaced by the API Portal. An API portal is also the "central store of truth", but now it includes REST APIs definitions (usually expressed using a Swagger-type format) as well as SOAP services. The API Portal is designed to be useful and helpful to developers who wish to build apps, rather than feeling like a chore to use. The lesson of SOA was that an attitude of "If we build it, they will come" (or "If we put it in the SOA Registry, people will use it") does not work. You have to make it into a pleasant experience for developers. API portals work for the very reason that SOA registries did not work: usability. Just like the SOA Gateway worked with the SOA Registry, so the API Gateway works hand-in-hand with the API Portal. Together, the combination of the API Portal with the API Gateway constitutes "API Management". The API Portal is for developers to sign up to use APIs, receive API Keys and quotas, and the API Gateway operates at runtime, managing the API Key usage and enforcing the API usage quotas. The API Gateway also performs the very important task of bridging from the technologies used by API clients (REST, OAuth) to the technologies used in the enterprise (Kerberos, SAML, or proprietary identity tokens such as CA SiteMinder smsession tokens). For more on this bridging, check out my webinar with Jason Cardinal from Identica tomorrow on "Bridging APIs to Enterprise Infrastructure". Gartner defines the combination of SOA Governance and API Management as "Application Services Governance". I'm proud to say that Axway (which acquired Vordel in 2012) is recognized by Gartner as a Leader in the category of Application Services Governance. We've seen an evolution of technologies (SOAP to REST) and approach (the UDDI registry to the web-based API Portal) in the journey from SOA Governance to API Management. From 30,000 feet, SOA Governance and API Management might look similar, but the new approach of API Management has already outshone SOA. The API Gateway and API Portal are key to this.

You might be asking yourself, 'why do I need a MongoDB-like experience for DynamoDB when there are already full-MongoDB cloud services like MMS, MongoLab, MongoHQ and MongoDirector? One developer believes there is a need and has set up an experimental project called DyngoDB. It provides the MongoDB-style interface in front of Amazon's DynamoDB and their CloudSearch service. Apparently, in the developer's case, he only wants the MongoDB interface but prefers the DynamoDB storage engine. We'll have to see if other developers also have this specific set of preferences.

This is a classic DevOps presentation from a former Velocity conference that I thought people could take another look at or see this for the first time. It's definitely worth it. Velocity 09: John Allspaw, "10+ Deploys Per Day: Dev and Ops Cooperation at Flickr" John Allspaw (Flickr/Yahoo!) and Paul Hammond (Flickr), "10+ Deploys Per Day: Dev and Ops Cooperation at Flickr"

This tutorial runs through a method for building a Java web service in Eclipse using Apache Tomcat and Apache Axis. The process takes under ten minutes.

RabbitMQ Simulator from Alvaro Videla on Vimeo. A demo of the new version of the RabbitMQ simulator with support for several exchange types, importing and exporting configuration and more. he goal of this app is to use it as a teaching tool for tutorials, presentations & more. Also it could serve as a topology designer for messaging applications since it supports exporting the design to RabbitMQ. Video Sections: - Intro - Direct Exchange - Fanout Exchange - Topic Exchange - Import/Export Configuration - Advanced Mode

In this post I'll document the simple configuration needed by ActiveMQ to configure the JDBC persistence adapter and setting up MySQL as the persistent storage. With this type of configuration you can also configure a Master/Slave broker setup by having more than one broker connect to the same database instance. List of Binaries used for this Example mysql-5.5.20-osx10.6-x86_64 mysql-connector-java-5.1.18-bin.jar apache-activemq-5.5.1-fuse-01-13 Configuring MySQL Download and install MySQL. Note for OS X users: The dmg provides a simple install that contains a Startup Item package which will configure MySQL to start automatically after each reboot as well as a Preference Pane plugin which will get added to the Settings panel to allow you to start/stop and configure autostart of MySQL. Once you have you have MySQL installed and properly configured, you will need to start the MySQL Monitor to create a user and database. macbookpro-251a:bin jsherman$ ./mysql -u root Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 29 Server version: 5.5.20 MySQL Community Server (GPL) Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> Then create the database for ActiveMQ mysql> CREATE DATABASE activemq; Then create a user and grant them privileges for the database mysql> CREATE USER 'activemq'@'%'localhost' IDENTIFIED BY 'activemq'; mysql> GRANT ALL ON activemq.* TO 'activemq'@'localhost'; mysql> exit Now log back into the MySQL Monitor and access the activemq database with the activemq user to make sure everything is okay macbookpro-251a:bin jsherman$ ./mysql -u activemq -p activemq Enter password: Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 28 Server version: 5.5.20 MySQL Community Server (GPL) Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql>exit ActiveMQ Broker Configuration Download the latest FuseSource distribution of ActiveMQ. In the broker's configuration file, activemq.xml, add the following persistence adapter to configure a JDBC connection to MySQL. Then, just after the ending broker element () add the following bean Copy the MySQL diver to the ActiveMQ lib directory, mysql-connector-java-5.1.18-bin.jar was used in this example. Now start your broker, you should see the following output if running from the console INFO | Using Persistence Adapter: JDBCPersistenceAdapter(org.apache.commons.dbcp.BasicDataSource@303bc1a1) INFO | Database adapter driver override recognized for : [mysql-ab_jdbc_driver] - adapter: class org.apache.activemq.store.jdbc.adapter.MySqlJDBCAdapter INFO | Database lock driver override not found for : [mysql-ab_jdbc_driver]. Will use default implementation. INFO | Attempting to acquire the exclusive lock to become the Master broker INFO | Becoming the master on dataSource: org.apache.commons.dbcp.BasicDataSource@303bc1a1 INFO | ActiveMQ 5.5.1-fuse-01-13 JMS Message Broker (jdbcBroker1) is starting Now you can check your database in MySQL and see that ActiveMQ has created the required tables. mysql> USE activemq; SHOW TABLES; +--------------------+ | Tables_in_activemq | +--------------------+ | ACTIVEMQ_ACKS | | ACTIVEMQ_LOCK | | activemq_msgs | +--------------------+ 3 rows in set (0.00 sec) mysql> If you configure multiple brokers to use this same database instance in the jdbcPersistenceAdapter element then these brokers will attempt to acquire a lock, if they are unable to get a database lock the will wait until the lock becomes available. This can be seen by starting a second broker using the above JDBC persistence configuration. INFO | PListStore:activemq-data/jdbcBroker/tmp_storage started INFO | Using Persistence Adapter: JDBCPersistenceAdapter(org.apache.commons.dbcp.BasicDataSource@78979f67) INFO | Database adapter driver override recognized for : [mysql-ab_jdbc_driver] - adapter: class org.apache.activemq.store.jdbc.adapter.MySqlJDBCAdapter As you can see the second broker did not fully initialize as it is waiting to acquire the database lock. If the master broker is killed, then you see the slave will acquire the database lock and becomes the new master. INFO | Database lock driver override not found for : [mysql-ab_jdbc_driver]. Will use default implementation. INFO | Attempting to acquire the exclusive lock to become the Master broker INFO | Becoming the master on dataSource: org.apache.commons.dbcp.BasicDataSource@2e19fc25 INFO | ActiveMQ 5.5.1-fuse-01-13 JMS Message Broker (jdbcBroker2) is starting INFO | For help or more information please see: http://activemq.apache.org/ INFO | Listening for connections at: tcp://macbookpro-251a.home:61617 INFO | Connector openwire Started INFO | ActiveMQ JMS Message Broker (jdbcBroker2, ID:macbookpro-251a.home-53193-1328656157052-0:1) started Summary As you can see, it is fairly simple and straight forward to configure a robust highly-available messaging system using ActiveMQ with database persistence.

A generally useful question posed by Charming asks how many queues one should use in RabbitMQ for the maximum message passing throughput/performance. I thought I'd distill the answers by Brian Kelly and RobotEyes here for anyone who's worried about their performance with RabbitMQ. And I'll bet that some of these pointers are relevant to other message queues as well: From the RabbitMQ blog: RabbitMQ's queues are fastest when they're empty. When a queue is empty, and it has consumers ready to receive messages, then as soon as a message is received by the queue, it goes straight out to the consumer. In the case of a persistent message in a durable queue, yes, it will also go to disk, but that's done in an asynchronous manner and is buffered heavily. The main point is that very little book-keeping needs to be done, very few data structures are modified, and very little additional memory needs allocating. From the rabbitmq-discuss mailing group: Use a larger prefetch count. Small values hurt performance. A topic exchange is slower than a direct or a fanout exchange. Make sure queues stay short. Longer queues impose more processing overhead. If you care about latency and message rates then use smaller messages. Use an efficient format (e.g. avoid XML) or compress the payload. Experiment with HiPE, which helps performance. Avoid transactions and persistence. Also avoid publishing in immediate or mandatory mode. Avoid HA. Clustering can also impact performance. You will achieve better throughput on a multi-core system if you have multiple queues and consumers. Use at least v2.8.1, which introduces flow control. Make sure the memory and disk space alarms never trigger. Virtualisation can impose a small performance penalty. Tune your OS and network stack. Make sure you provide more than enough RAM. Provide fast cores and RAM. Hope that gives you some ideas for improving your queueing performance.

Some interesting insider info was released this week about the Facebook deployment process. I've compiled a TL;DR version here in case you don't want to read the whole piece, but I recommend you do. The information comes straight from Chuck Rossi, the head of Facebook's release engineering team, who was interviewed by Ars Technica's Ryan Paul: An important aspect of Facebook's development culture is the idea that developers are fully responsible for how their code behaves in production. This philosophy mirrors the "DevOps" movement, which encourages lowering the wall between software development and IT operations. If any of the code in a Facebook update causes problems in production, the developer who wrote it is on the hook for making sure that the issue gets resolved as quickly as possible. --Ryan Paul, Ars Technica "Exclusive: a behind-the-scenes look at Facebook release engineering" I think we've got more evidence here that "DevOps" is just a name for what many super-successful companies have already been doing. On to the TL;DR! For deployment Facebook uses Uses custom BitTorrent P2P system Site updates take 15-30 minutes JS, CSS, and Graphics are hosted on distributed CDNs Deployment steps Update goes to an "a2" tier which rolls update out to a small, random collection of users. Check-in procedure on company IRC where all developers who submitted code for the update have confirm that they are ready to respond if there is a problem when the update goes out. Head of release engineering issues command in terminal to begin deployment Watch web-based monitoring dashboards as the update rolls out. A small amount of servers will fail during most deployments, but it usually doesn't cause any issues. That's because any of Facebook's servers can handle any sort of page request, so they don't have to worry about serialization and migration of user session states. The servers can keep handling incoming page requests during their software updates. No-downtime, as we all know. Frequency of deployments: One minor update on most business days One major update on a weekly basis, usually Tuesdays "Release early and often." The mantra rings true at Google, Facebook, and many other companies that we look up to as the best in the business. Make sure you check out the whole article for the full story on Facebook's release engineering.

Looking for a "DevOps Manifesto" to help guide your own organizational transiton? Well, there's no 'manifesto' out there like we have for Agile and SOA, but I think if you look around, you'll find that this commuity does have some well-defined, agreed-upon goals. In Kris Buytaert's presentation: "Devops, the future is here, it's just not evenly distributed yet" he gives a great 10 point summary of organizational goals from Edward Deming that coincide with what the DevOps movment is looking to achieve in IT organizations: Adopt the new philosophy. We are in a new economic age. Western management must awaken to the challenge, and must learn their responsibilities, and take on leadership for a change. Cease dependence on inspection to achieve quality. Eliminate the need for massive inspection by building quality into the product in the first place. Constantly improve your system of production and service, to improve quality and productivity, and thus constantly decrease costs. Institute training on the job. Institute leadership. The aim of supervision should be to help people and machines and gadgets to a better job. Drive out fear, so that everyone may work effectively for the company. Break down barriers between departments. People in research, design, sales, and production must work as a team in order to forsee problems of production and usage that may be encountered with the product or service. Eliminate slogans, exhortation, and targets for the work force; asking for zero defects and new levels of productivity. Such exhortations only create adversarial relationships, since the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the work force. *Eliminate management by objective. Eliminate management by numbers and numerical goals. Instead substitute with leadership. *Remove barriers that rob the hourly worker of his/her right to pride of workmanship. The responsibility of supervisors must be changed from sheer numbers to quality. *Remove barriers that rob people in management and in engineering of their right to pride of workmanship. Institute a vigorous program of education and self-improvement. Put everybody in the company to work to accomplish the transformation. The transformation is everybody's job. On a more technical level, Kris says you need the following things to get to a point where your team can deploy at a moment's notice, quickly and safely: Version Control Continuous Integration Built Pipelines Bug Tracking Integrated Testing Automated Deployment Here are all of the slides from his presentation: Devops, the future is here, it's just not evenly distributed yet. View more presentations from Kris Buytaert

Most databases have them: a small handful of big tables that grow at a substantially faster rate than any of their peers (think tweets, clicks, check-ins, etc). Over time, the sheer size of these tables cause queries against them to slow down, and increase the size and time taken for backups. In some cases, not all of this data needs to be “live” so that it can be randomly accessed forever, and can be archived once some criteria has been met. I like to think of this as Eventually Irrelevant (if I may coin a term). This post was authored by Brian Ploetz Many database products support partitioning features, which allow you to arrange data physically based on some criteria (typically date, range, list, or hash-based partitioning). These partitioning features allow you to drop partitions from the live database when appropriate. While MongoDB supports horizontal partitioning via sharding, it does not currently support the notion of dropping shards. Even if it did, the criteria for what documents to drop from a collection would most likely not be based on your shard key, so it’s unlikely you’d want to drop a shard anyways. You’d want to drop data from all shards. Capped Collections and the forthcoming TTL-based Capped Collections feature are not quite what we’re looking for either. Capped Collections have very strict rules (including not being able to shard them), and for both normal and TTL-based Capped Collections, as old documents are aged out they are simply dropped on the floor. What we need is more control over the dropping process, such that we can guarantee that our data has been copied to an archive database/data warehouse before being dropped from the main database. I set out to find the most efficient way of manually archiving documents from a large collection given the tools currently available in MongoDB. For the purposes of this exercise, let’s assume we have a large collection of ad clicks which are associated with ads. We want to archive clicks which are associated with ads which expired more than 3 months ago. MapReduce MongoDB wants you to do bulk operations using MapReduce. Since all of the examples and documentation revolve around aggregation, I suspected I wouldn’t be able to use MapReduce as the backbone of my archiving process. After some experimenting, I found that my intuition was correct. Upon attempting to insert/remove documents within the map or reduce functions, I would quickly run into exception 10293 internal error: locks are not upgradeable errors: Sun Sep 25 21:22:24 [conn5] update warehouse.clicks query: { _id: ObjectId('4d6bf191db0b4b2b1fad5b65') } exception 10293 internal error: locks are not upgradeable: { "opid" : 4248173, "active" : false, "waitingForLock" : false, "op" : "update", "ns" : "?", "query" : { "_id" : { "$oid" : "4d6bf191db0b4b2b1fad5b65" } }, "client" : "0.0.0.0:0", "desc" : "conn" } 0ms Sun Sep 25 21:22:24 [conn5] Assertion: 10293:internal error: locks are not upgradeable: { "opid" : 4248174, "active" : false, "waitingForLock" : false, "op" : "update", "ns" : "?", "query" : { "_id" : { "$oid" : "4d6bf198db0b4b2b20ad5b65" } }, "client" : "0.0.0.0:0", "desc" : "conn" } 0x10008de9b 0x1002064f7 0x1002c1ec6 0x1002c4f3d 0x1002c5f7a 0x1000a8181 0x100147df4 0x1004dc68e 0x1004efc93 0x1004dc71b 0x1004dcb71 0x10049a078 0x100158efa 0x100377760 0x100389d0c 0x10034c204 0x10034d877 0x100180cc4 0x100184649 0x1002b9e89 0 mongod 0x000000010008de9b _ZN5mongo11msgassertedEiPKc + 315 1 mongod 0x00000001002064f7 _ZN5mongo10MongoMutex19_writeLockedAlreadyEv + 263 2 mongod 0x00000001002c1ec6 _ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE + 886 3 mongod 0x00000001002c4f3d _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE + 5661 4 mongod 0x00000001002c5f7a _ZN5mongo14DBDirectClient3sayERNS_7MessageE + 106 5 mongod 0x00000001000a8181 _ZN5mongo12DBClientBase6updateERKSsNS_5QueryENS_7BSONObjEbb + 273 6 mongod 0x0000000100147df4 _ZN5mongo12mongo_updateEP9JSContextP8JSObjectjPlS4_ + 660 7 mongod 0x00000001004dc68e js_Invoke + 3864 8 mongod 0x00000001004efc93 js_Interpret + 71932 9 mongod 0x00000001004dc71b js_Invoke + 4005 10 mongod 0x00000001004dcb71 js_InternalInvoke + 404 11 mongod 0x000000010049a078 JS_CallFunction + 86 12 mongod 0x0000000100158efa _ZN5mongo7SMScope6invokeEyRKNS_7BSONObjEib + 666 13 mongod 0x0000000100377760 _ZN5mongo2mr8JSMapper3mapERKNS_7BSONObjE + 96 14 mongod 0x0000000100389d0c _ZN5mongo2mr16MapReduceCommand3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb + 1740 15 mongod 0x000000010034c204 _ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 628 16 mongod 0x000000010034d877 _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi + 2151 17 mongod 0x0000000100180cc4 _ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi + 52 18 mongod 0x0000000100184649 _ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 10585 19 mongod 0x00000001002b9e89 _ZN5mongo13receivedQueryERNS_6ClientERNS_10DbResponseERNS_7MessageE + 569 Upon killing the client which kicked off the MapReduce process, theses errors would continue to spin out of control, and I’d have to bounce the server. While MapReduce’s divide and conquer approach is ideal for what we’re trying to accomplish, unless the ability to obtain a write lock within a MapReduce function is supported, it is not a viable option for now. db.eval() While a server side script is certainly a viable option for writing an archiving process, it has a fundamental flaw in that it won’t work for sharded collections, which could be a non-starter for some folks. An example archiving process using db.eval() would look something like this: archiveClick = function archiveClick(doc) { if (expiredAdIds.indexOf(doc.ad_id) != -1) { archivedb = db.getMongo().getDB("archive"); archivedb.clicks.save(doc); // before we remove the original document, make sure the archive worked if (archivedb.getLastError() == null) { db.clicks.remove({_id: doc._id}); } else { throw "could not archive document with _id " + doc._id; } } } archiveClicks = function archiveClicks() { threeMonthsAgo = new Date(); threeMonthsAgo.setMonth(threeMonthsAgo.getMonth()-3); expiredAdIds = []; getExpiredAds = function(ad) {expiredAdIds.push(ad._id.str);} db.ads.find({end_date: {$lte: threeMonthsAgo}).forEach(getExpiredAds); db.system.js.save({_id: "expiredAdIds", value: expiredAdIds}); db.clicks.find().forEach(archiveClick); db.system.js.remove({_id: "expiredAdIds"}); } db.system.js.save({_id: "archiveClick", value: archiveClick}); db.system.js.save({_id: "archiveClicks", value: archiveClicks}); db.runCommand({$eval: "archiveClicks()", nolock: true}); Note the nolock: true sent to db.eval(). This ensures the mongod isn’t locked for other operations while this runs. The Holy Grail: Define Custom archive() Functions On Collections In order for our archiving process to work for sharded collections, we will need to add new functions to collection objects in the shell to perform the archiving. Frankly, this feels a lot more natural than db.eval(). I wanted to support two modes of archiving: immediate archiving, and a mark & sweep approach which allows you to queue documents to be archived at one point in time, and perform the actual archiving at a later time (off-peak hours). I’ve made these functions available in my mongodb-archive project on GitHub. Download the archive.js file, and use it like so: load("archive.js"); threeMonthsAgo = new Date(); threeMonthsAgo.setMonth(threeMonthsAgo.getMonth()-3); expiredAdIds = []; getExpiredAds = function(ad) {expiredAdIds.push(ad._id.str);} db.ads.find({end_date: {$lte: threeMonthsAgo}).forEach(getExpiredAds); archiveConnection = new Mongo("localhost:27018"); archiveDB = archiveConnection.getDB("archive"); archiveCollection = archiveDB.getCollection("clicks"); for (var i = 0; i < expiredAdIds.length; i++) { print("archiving clicks for ad_id: " + expiredAdIds[i]); db.clicks.archive({"ad_id": expiredAdIds[i]}, archiveCollection); print(""); } The mark/sweep variant would look like: db.clicks.queueForArchive({"ad_id": expiredAdIds[i]}); // ....some time later..... db.clicks.archiveQueued(archiveCollection); The biggest factors on the performance of the archiving process are really no different than the things that impact the performance of your MongoDB database overall: working set size, indexes, disk speed, and lock contention. The more indexes there are on the source collection, the longer the remove() will take. The more indexes there are on the archive collection, the longer the save() will take. The more of the data that you wish to archive is resident in memory, the faster the process will be. On my MacBook Pro (2.3GHz i7, 8GB RAM) with a 5400 RPM SATA drive and a clicks collection containing over 11 million documents, I was able to archive ~500 documents per second with the immediate archiving approach. Example output from mongostat when this was running: Archive DB: insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 1 0 0 3 0 208m 2.85g 15m 3.8 0 0|0 0|0 1k 1k 2 17:13:02 0 0 45 0 0 46 0 208m 2.85g 15m 3.4 0 0|0 0|0 61k 6k 2 17:13:03 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 8 0 0 9 0 208m 2.85g 15m 0.1 0 0|0 0|0 10k 2k 2 17:13:04 0 0 47 0 0 48 0 208m 2.85g 15m 3.4 0 0|0 0|0 66k 7k 2 17:13:05 0 0 0 0 0 1 1 208m 2.85g 15m 0 0 0|0 0|0 62b 1k 2 17:13:06 0 0 18 0 0 19 0 208m 2.85g 15m 2.6 0 0|0 0|0 24k 3k 2 17:13:07 0 0 79 0 0 80 0 208m 2.85g 16m 1.1 0 0|0 0|0 111k 11k 2 17:13:08 0 0 131 0 0 132 0 208m 2.85g 16m 8.8 0 0|0 0|0 181k 17k 2 17:13:09 0 0 216 0 0 217 0 208m 2.85g 16m 3.8 0 0|0 0|0 291k 27k 2 17:13:10 0 0 425 0 0 426 0 208m 2.85g 17m 10.5 0 0|0 0|0 600k 53k 2 17:13:11 0 0 490 0 0 490 0 208m 2.85g 19m 7.3 0 0|0 0|1 658k 61k 2 17:13:12 0 0 630 0 0 632 0 208m 2.85g 20m 11.3 0 0|0 0|0 844k 78k 2 17:13:13 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 818 0 0 818 0 208m 2.85g 22m 13.6 0 0|0 0|0 1m 101k 2 17:13:14 0 0 686 0 0 688 0 208m 2.85g 21m 9.6 0 0|0 0|0 932k 85k 2 17:13:15 0 0 0 0 0 1 0 208m 2.85g 22m 0 0 0|0 0|0 62b 1k 2 17:13:16 0 0 178 0 0 179 0 208m 2.85g 22m 1.2 0 0|0 0|0 252k 23k 2 17:13:17 0 0 974 0 0 975 0 208m 2.85g 23m 12.3 0 0|0 0|0 1m 121k 2 17:13:18 0 0 920 0 0 921 0 208m 2.85g 26m 10.1 0 0|0 0|0 1m 114k 2 17:13:19 0 0 810 0 0 811 0 208m 2.85g 26m 8.1 0 0|0 0|0 1m 100k 2 17:13:20 0 0 612 0 0 613 0 208m 2.85g 27m 8.1 0 0|0 0|0 798k 76k 2 17:13:21 0 0 0 0 0 1 0 208m 2.85g 27m 0 0 0|0 0|0 62b 1k 2 17:13:22 0 0 0 0 0 1 0 208m 2.85g 27m 0 0 0|0 0|0 62b 1k 2 17:13:23 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 30 0 0 31 0 208m 2.85g 27m 1.1 0 0|0 0|0 46k 5k 2 17:13:24 0 0 852 0 0 853 0 208m 2.85g 27m 14.8 0 0|0 0|0 1m 106k 2 17:13:25 0 0 768 0 0 769 0 208m 2.85g 29m 9.5 0 0|0 0|0 1m 95k 2 17:13:26 0 0 867 0 0 868 0 208m 2.85g 32m 13.4 0 0|0 0|0 1m 107k 2 17:13:27 0 0 705 0 0 706 0 208m 2.85g 31m 10.8 0 0|0 0|0 936k 88k 2 17:13:28 0 0 331 0 0 332 0 208m 2.85g 32m 3.9 0 0|0 0|0 446k 42k 2 17:13:29 0 0 0 0 0 1 0 208m 2.85g 32m 0 0 0|0 0|0 62b 1k 2 17:13:30 0 0 551 0 0 552 0 208m 2.85g 32m 6.1 0 0|0 0|0 739k 69k 2 17:13:31 0 0 728 0 0 729 0 208m 2.85g 35m 12.2 0 0|0 0|0 949k 90k 2 17:13:32 0 0 991 0 0 992 0 208m 2.85g 35m 11.7 0 0|0 0|0 1m 123k 2 17:13:33 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 935 0 0 935 0 208m 2.85g 38m 10.9 0 0|0 0|1 1m 116k 2 17:13:34 0 0 431 0 0 433 0 208m 2.85g 39m 5.9 0 0|0 0|0 563k 54k 2 17:13:35 0 0 0 0 0 1 0 208m 2.85g 39m 0 0 0|0 0|0 62b 1k 2 17:13:36 0 0 518 0 0 519 0 208m 2.85g 40m 5 0 0|0 0|0 693k 65k 2 17:13:37 0 0 904 0 0 905 0 208m 2.85g 40m 8.9 0 0|0 0|0 1m 112k 2 17:13:38 0 0 958 0 0 959 0 208m 2.85g 41m 14.2 0 0|0 0|0 1m 119k 2 17:13:39 0 0 899 0 0 900 0 208m 2.85g 44m 11.1 0 0|0 0|0 1m 111k 2 17:13:40 0 0 401 0 0 402 0 208m 2.85g 42m 5.9 0 0|0 0|0 540k 50k 2 17:13:41 0 0 856 0 0 857 0 208m 2.85g 44m 9.4 0 0|0 0|0 1m 106k 2 17:13:42 0 0 232 0 0 233 0 208m 2.85g 45m 2.9 0 0|0 0|0 311k 29k 1 17:13:43 Source DB: insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 2 0 12 1 13 0 56g 115g 2.82g 52.2 0 0|0 0|1 1k 822k 2 17:13:02 0 0 0 42 0 43 0 56g 115g 2.65g 88.1 0 0|0 0|1 5k 4k 2 17:13:03 0 0 0 14 0 15 1 56g 115g 1.88g 101 0 0|0 0|1 1k 2k 2 17:13:04 0 0 0 33 1 35 0 56g 115g 1.89g 53.8 16.6 0|0 1|0 4k 4k 2 17:13:05 0 0 0 0 0 1 0 56g 115g 1.9g 0 0 0|0 1|0 62b 1k 2 17:13:06 0 0 0 34 0 35 0 56g 115g 1.9g 47.3 0 0|0 0|0 4k 4m 2 17:13:07 0 0 0 91 0 92 0 56g 115g 1.89g 86.8 0 0|0 0|0 12k 8k 2 17:13:08 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 149 0 150 0 56g 115g 1.9g 82.5 0 0|0 0|0 20k 13k 2 17:13:09 0 0 0 287 0 287 0 56g 115g 1.9g 74.3 0 0|0 0|1 38k 25k 2 17:13:10 0 0 0 387 0 389 0 56g 115g 1.91g 64.6 0 0|0 0|0 52k 33k 2 17:13:11 0 0 0 580 0 581 0 56g 115g 1.93g 59 0 0|0 0|0 78k 49k 2 17:13:12 0 0 0 685 0 686 0 56g 115g 1.93g 47.9 0 0|0 0|0 93k 58k 2 17:13:13 0 0 0 729 0 730 0 56g 115g 1.95g 42.6 0 0|0 0|0 99k 61k 2 17:13:14 0 0 0 551 1 552 0 56g 115g 1.97g 34.5 0 0|0 1|0 74k 47k 2 17:13:15 0 0 0 0 0 1 0 56g 115g 1.98g 0 0 0|0 1|0 62b 1k 2 17:13:16 0 0 0 368 0 368 0 56g 115g 1.96g 20 0 0|0 0|1 50k 4m 2 17:13:17 0 0 0 1032 0 1034 0 56g 115g 2g 35.9 0 0|0 0|0 140k 87k 2 17:13:18 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 834 0 835 0 56g 115g 2.01g 42.3 0 0|0 0|0 113k 70k 2 17:13:19 0 0 0 922 0 922 0 56g 115g 2.02g 38.9 0 0|0 0|1 125k 77k 2 17:13:20 0 0 0 338 1 340 0 56g 115g 2.03g 13 0 0|0 1|0 46k 29k 2 17:13:21 0 0 0 0 0 1 0 56g 115g 2.04g 0 0 0|0 1|0 62b 1k 2 17:13:22 0 0 0 0 0 1 0 56g 115g 2.04g 0 0 0|0 1|0 62b 1k 2 17:13:23 0 0 0 185 0 186 0 56g 115g 2.02g 24.7 0 0|0 0|0 25k 4m 2 17:13:24 0 0 0 920 0 920 0 56g 115g 2.02g 29.9 0 0|0 0|1 125k 77k 2 17:13:25 0 0 0 741 0 742 0 56g 115g 2.04g 49 0 0|0 0|1 100k 62k 2 17:13:26 0 0 0 886 0 887 0 56g 115g 2.05g 33 0 0|0 0|1 120k 74k 2 17:13:27 0 0 0 683 0 684 0 56g 115g 2.08g 57.3 0 0|0 0|1 92k 58k 2 17:13:28 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 138 1 140 0 56g 115g 2.06g 14.5 0 0|0 1|0 18k 12k 2 17:13:29 0 0 0 12 0 12 0 56g 115g 2.07g 0.2 0 0|0 0|1 1k 4m 2 17:13:30 0 0 0 761 0 763 0 56g 115g 2.08g 46.5 0 0|0 0|0 103k 64k 2 17:13:31 0 0 0 762 0 762 0 56g 115g 2.09g 47 0 0|0 0|1 103k 64k 2 17:13:32 0 0 0 957 0 959 0 56g 115g 2.1g 35.3 0 0|0 0|0 130k 80k 2 17:13:33 0 0 0 905 0 905 0 56g 115g 2.13g 38.5 0 0|1 0|1 123k 76k 2 17:13:34 0 0 0 239 1 241 0 56g 115g 2.12g 14.9 0 0|0 1|0 32k 21k 2 17:13:35 0 0 0 0 0 1 0 56g 115g 2.13g 0 0 0|0 1|0 62b 1k 2 17:13:36 0 0 0 780 0 781 0 56g 115g 2.13g 42.8 0 0|0 0|0 106k 4m 2 17:13:37 0 0 0 805 0 806 0 56g 115g 2.14g 39.8 0 0|0 0|0 109k 68k 2 17:13:38 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 988 0 989 0 56g 115g 2.16g 34.8 0 0|0 0|0 134k 83k 2 17:13:39 0 0 0 953 0 953 0 56g 115g 2.17g 39.9 0 0|0 0|1 129k 80k 2 17:13:40 0 0 0 375 1 376 0 56g 115g 2.18g 13.9 0 0|1 0|1 51k 1m 2 17:13:41 0 0 0 867 0 869 0 56g 115g 2.19g 41.7 0 0|0 0|0 118k 73k 1 17:13:42 Total: MongoDB shell version: 2.0.1 connecting to: localhost:27017/prodcopy archiving clicks for ad_id: 4d6c12497b90420373000056 archiving documents... archived 19045 documents in 40701ms. For the queued approach with this same set up, I was able to queue ~1200 documents per second, and the archiving process performed at about the same ~500 documents per second. Example output from mongostat while this was running: Archive DB: insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:44 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:45 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:46 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:47 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:48 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:49 0 0 248 0 0 249 0 208m 2.85g 17m 6 0 0|0 0|0 283k 31k 2 17:16:50 0 0 780 0 0 781 0 208m 2.85g 18m 8.2 0 0|0 0|0 883k 97k 2 17:16:51 0 0 1256 0 0 1257 0 208m 2.85g 22m 13.7 0 0|0 0|0 1m 155k 2 17:16:52 0 0 1084 0 0 1085 0 208m 2.85g 21m 10 0 0|0 0|0 1m 134k 2 17:16:53 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 1108 0 0 1109 0 208m 2.85g 25m 9.8 0 0|0 0|0 1m 137k 2 17:16:54 0 0 1108 0 0 1109 0 208m 2.85g 24m 9.8 0 0|0 0|0 1m 137k 2 17:16:55 0 0 1110 0 0 1111 0 208m 2.85g 27m 11 0 0|0 0|0 1m 137k 2 17:16:56 0 0 1166 0 0 1167 0 208m 2.85g 31m 11.3 0 0|0 0|0 1m 144k 2 17:16:57 0 0 1246 0 0 1247 0 208m 2.85g 31m 10.6 0 0|0 0|0 1m 154k 2 17:16:58 0 0 1094 0 0 1095 0 208m 2.85g 35m 9.6 0 0|0 0|0 1m 135k 2 17:16:59 0 0 957 0 0 958 0 208m 2.85g 33m 9.6 0 0|0 0|0 1m 119k 2 17:17:00 0 0 721 0 0 722 0 464m 3.35g 38m 8.9 0 0|0 0|0 1m 90k 2 17:17:01 0 0 243 0 0 243 0 464m 3.35g 34m 9.7 0 0|0 0|0 445k 31k 2 17:17:02 0 0 921 0 0 923 0 464m 3.35g 37m 30.1 0 0|0 0|0 1m 114k 2 17:17:03 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 668 0 0 669 0 464m 3.35g 38m 9.6 0 0|0 0|0 962k 83k 2 17:17:04 0 0 995 0 0 995 0 464m 3.35g 41m 21.2 0 0|0 0|1 1m 123k 2 17:17:05 0 0 467 0 0 468 0 464m 3.35g 27m 12 0 0|1 0|1 679k 58k 2 17:17:06 0 0 1 0 0 2 1 464m 3.35g 17m 85.5 0 0|1 0|1 1k 1k 2 17:17:07 0 0 337 0 0 338 0 464m 3.35g 19m 36.2 0 0|0 0|1 499k 42k 2 17:17:08 0 0 917 0 0 919 0 464m 3.35g 21m 20.4 0 0|0 0|0 1m 114k 2 17:17:09 0 0 1022 0 0 1022 0 464m 3.35g 25m 22.7 0 0|0 0|1 1m 126k 2 17:17:10 0 0 970 0 0 971 0 464m 3.35g 24m 27.4 0 0|0 0|1 1m 120k 2 17:17:11 0 0 166 0 0 168 0 464m 3.35g 25m 1.2 0 0|0 0|0 251k 21k 2 17:17:12 0 0 862 0 0 863 0 464m 3.35g 28m 23.1 0 0|0 0|0 1m 107k 2 17:17:13 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 888 0 0 889 0 464m 3.35g 28m 28.8 0 0|0 0|0 1m 110k 2 17:17:14 0 0 540 0 0 541 0 464m 3.35g 30m 12.2 0 0|0 0|0 802k 67k 2 17:17:15 0 0 879 0 0 880 0 464m 3.35g 31m 22.3 0 0|0 0|0 1m 109k 2 17:17:16 0 0 473 0 0 474 0 464m 3.35g 36m 17.2 0 0|0 0|0 1m 59k 1 17:17:17 Source DB: insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 0 0 1 0 56g 115g 1.86g 0 0 0|0 0|0 62b 1k 1 17:16:29 0 0 0 0 0 1 0 56g 115g 1.86g 0 0 0|0 0|0 62b 1k 1 17:16:30 1 1 1 0 1 3 0 56g 115g 1.87g 51.1 0 0|0 0|1 475b 697k 2 17:16:31 0 0 0 0 0 1 0 56g 115g 349m 129 0 0|0 0|1 62b 1k 2 17:16:32 0 0 0 0 0 1 0 56g 115g 367m 101 0 0|0 0|1 62b 1k 2 17:16:33 0 0 0 0 0 1 0 56g 115g 384m 73.4 0 0|0 0|1 62b 1k 2 17:16:34 0 0 0 0 0 1 0 56g 115g 428m 112 0 0|0 0|1 62b 1k 2 17:16:35 0 0 0 0 0 1 0 56g 115g 436m 94.8 0 0|0 0|1 62b 1k 2 17:16:36 0 0 0 0 0 1 0 56g 115g 450m 108 0 0|0 0|1 62b 1k 2 17:16:37 0 0 0 0 0 1 0 56g 115g 461m 110 0 0|0 0|1 62b 1k 2 17:16:38 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 0 0 1 0 56g 115g 480m 95.5 0 0|0 0|1 62b 1k 2 17:16:39 0 0 0 0 0 1 0 56g 115g 494m 73.5 0 0|0 0|1 62b 1k 2 17:16:40 0 0 0 0 0 1 0 56g 115g 531m 120 0 0|0 0|1 62b 1k 2 17:16:41 0 0 0 0 0 1 0 56g 115g 541m 108 0 0|0 0|1 62b 1k 2 17:16:42 0 0 0 0 0 1 0 56g 115g 564m 74.6 0 0|0 0|1 62b 1k 2 17:16:43 0 0 0 0 0 1 0 56g 115g 579m 127 0 0|0 0|1 62b 1k 2 17:16:44 0 0 0 0 0 1 0 56g 115g 605m 100 0 0|0 0|1 62b 1k 2 17:16:45 0 0 0 0 0 1 0 56g 115g 631m 98.5 0 0|0 0|1 62b 1k 2 17:16:46 0 0 0 0 0 1 0 56g 115g 657m 89.1 0 0|0 0|1 62b 1k 2 17:16:47 0 0 0 0 0 1 0 56g 115g 662m 87.8 0 0|0 0|1 62b 1k 2 17:16:48 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 0 0 1 0 56g 115g 680m 93.5 0 0|0 0|1 62b 1k 2 17:16:49 0 1 0 530 1 531 0 56g 115g 767m 81.3 0 0|0 0|1 72k 4m 2 17:16:50 0 0 0 884 0 886 0 56g 115g 721m 48 0 0|0 0|0 120k 74k 2 17:16:51 0 0 0 1079 0 1080 0 56g 115g 698m 33.6 0 0|0 0|0 146k 90k 2 17:16:52 0 0 0 1294 0 1294 0 56g 115g 720m 23.8 0 0|1 0|1 175k 108k 2 17:16:53 0 0 0 1118 1 1120 0 56g 115g 740m 31.9 0 0|0 0|0 152k 4m 2 17:16:54 0 0 0 1111 0 1112 0 56g 115g 718m 33.3 0 0|0 0|0 151k 93k 2 17:16:55 0 0 0 1097 0 1098 0 56g 115g 706m 33.1 0 0|0 0|0 149k 92k 2 17:16:56 0 0 0 1097 0 1098 0 56g 115g 701m 33.2 0 0|0 0|0 149k 92k 2 17:16:57 0 0 0 1091 1 1092 0 56g 115g 696m 35.1 0 0|0 0|0 148k 4m 2 17:16:58 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 1149 0 1150 0 56g 115g 718m 21.3 0 0|0 0|0 156k 96k 2 17:16:59 0 0 0 1091 0 1091 0 56g 115g 707m 31.2 0 0|0 0|1 148k 91k 2 17:17:00 0 0 0 468 0 470 0 56g 115g 714m 9.3 0 0|0 0|0 63k 40k 2 17:17:01 0 0 0 332 0 333 0 56g 115g 694m 27.3 0 0|0 0|0 45k 28k 2 17:17:02 0 0 0 1013 1 1014 0 56g 115g 713m 17.8 0 0|0 0|0 137k 4m 2 17:17:03 0 0 0 551 0 552 0 56g 115g 691m 39.9 0 0|0 0|0 74k 47k 2 17:17:04 0 0 0 1140 0 1141 0 56g 115g 709m 19.7 0 0|0 0|0 155k 95k 2 17:17:05 0 0 0 126 0 127 0 56g 115g 709m 2.1 0 0|0 0|0 17k 11k 2 17:17:06 0 0 0 92 0 92 0 56g 115g 710m 1.6 0 0|0 0|1 12k 8k 2 17:17:07 0 0 0 605 1 607 0 56g 115g 700m 29.1 0 0|0 0|0 82k 4m 2 17:17:08 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 814 0 814 0 56g 115g 699m 17.3 0 0|0 0|1 110k 68k 2 17:17:09 0 0 0 1036 0 1037 0 56g 115g 708m 18.4 0 0|0 0|0 140k 87k 2 17:17:10 0 0 0 820 0 821 0 56g 115g 715m 14.4 0 0|1 0|1 111k 69k 2 17:17:11 0 0 0 298 0 300 0 56g 115g 683m 24.2 0 0|0 0|0 40k 26k 2 17:17:12 0 0 0 907 1 908 0 56g 115g 699m 20.4 0 0|0 0|0 123k 4m 2 17:17:13 0 0 0 946 0 947 0 56g 115g 707m 16.4 0 0|0 0|0 128k 79k 2 17:17:14 0 0 0 374 0 375 0 56g 115g 677m 36.7 0 0|0 0|0 50k 32k 2 17:17:15 0 0 0 905 1 905 0 56g 115g 689m 16.2 0 0|0 0|0 123k 821k 2 17:17:16 0 0 0 259 0 261 0 56g 115g 690m 5.1 0 0|0 0|0 35k 22k 1 17:17:17 0 0 0 0 0 1 0 56g 115g 690m 0 0 0|0 0|0 62b 1k 1 17:17:18 Total: MongoDB shell version: 2.0.1 connecting to: localhost:27017/prodcopy archiving clicks for ad_id: 4d6c12497b90420373000062 ensuring sparse index on field "arch" marking documents for archive... queued 22227 documents for archive in 19369ms. archiving queued documents to archive.clicks... archived 22227 queued documents to archive.clicks in 26901ms. Upgrading the hard drive to an SSD, I was able to almost triple the performance of the archiving step, and archive ~1300 documents per second. Obviously this is all a giant hack, and it would be nice if MongoDB supported some kind of partitioning feature so we didn’t have to do any of this manually. But until then, this is the best I could get given the tools currently available. Source: http://blog.brianploetz.com/post/12131083486/adventures-in-archiving-with-mongodb

OpenStreetMap is a global project with an aim of collaboratively collecting map data, and today Ken Guest has submitted his PHP package for communitcating with the OSM API to the public and the PEAR PEPr review process: So over the last while, I’ve been working on a PHP package imaginatively named Services_Openstreetmap, for interacting with the openstreetmap API. I initially needed it so I could search for certain POIs and tabulate the results; it’s now also capable of adding data to the openstreetmap database – nodes and other elements can be created, updated and so on. It will even access the details of the user that is being used to modify that data, which is one difference between it and the other single purpose OSM frameworks. --Ken Guest You can view the submission here, and you should definitely take a look at openstreetmap.org if you haven't already. Good news for PHP developers looking to use this project more heavily in their applications.

I’ve been going over a couple of posts by Steve Kilner that question whether Agile methods can be used effectively in software maintenance. It’s a surprising question really. There are a lot of maintenance teams who have had success following Agile methods like Scrum and Extreme Programming (XP) for some time now. We’ve been doing it for almost 5 years, enhancing and maintaining and supporting enterprise systems, and I know that it works. Agile development naturally leads into maintenance – the goal of incremental Agile development is to get working software out to customers as soon as possible, and get customers using it. At some point, when customers are relying on the software to get real business done and need support and help to keep the system running, teams cross from development over to maintenance. But there’s no reason for Agile development teams to fundamentally change the way that they work when this happens. It is harder to introduce Agile practices into a legacy maintenance team – there are a lot of technical requirements and some cultural changes that need to be made. But most maintenance teams have little to lose and lots to gain from borrowing from what Agile development teams are doing. Agile methods are designed to help small teams deal with a lot of change and uncertainty, and to deliver software quickly – all things that are at least as important in maintenance as they are in development. Technical practices in Extreme Programming especially help ensure that the code is always working – which is even more important in maintenance than it is in development, because the code has to work the first time in production. Agile methods have to be adapted to maintenance, but most teams have found it necessary to adapt these methods to fit their situations anyways. Let’s look at what works and what has to be changed to make Agile methods like Scrum and XP work in maintenance. What works well and what doesn’t Planning Game Managing maintenance isn’t the same as managing a development project – even an Agile development project. Although Agile development teams expect to deal with ambiguity and constant change, maintenance teams need to be even more flexible and responsive, to manage conflicts and unpredictable resourcing problems. Work has to be continuously reviewed and prioritized as it comes in – the customer can’t wait for 2 weeks for you to look at a production bug. The team needs a fast path for urgent changes and especially for hot fixes. You have to be prepared for support demands and interruptions. Structure the team so that some people can take care of second-level support, firefighting and emergency bug fixing and the rest of the team can keep moving forward and get something done. Build slack into schedules to allow for last-minute changes and support escalation. You will also have to be more careful in planning out maintenance work, to take into account technical and operational dependencies and constraints and risks. You’re working in the real world now, not the virtual reality of a project. Standups Standups play an important role in Agile projects to help teams come up to speed and bond. But most maintenance teams work fine without standups – since a lot of maintenance work can be done by one person working on their own, team members don’t need to listen to each other each morning talking about what they did yesterday and what they’re going to do – unless the team is working together on major changes. If someone has a question or runs into a problem, they can ask for help without waiting until the next day. Small releases Most changes and fixes that maintenance teams need to make are small, and there is almost always pressure from the business to get the code out as soon as it is ready, so an Agile approach with small and frequent releases makes a lot of sense. If the time boxes are short enough, the customer is less likely to interrupt and re-prioritize work in progress – most businesses can wait a few days or a couple of weeks to get something changed. Time boxing gives teams a way to control and structure their work, an opportunity to batch up related work to reduce development and testing costs, and natural opportunities to add in security controls and reviews and other gates. It also makes maintenance work more like a project, giving the team a chance to set goals and to see something get done. But time boxing comes with overhead – the planning and setup at the start, then deployment and reviews at the end – all of which adds up over time. Maintenance teams need to be ruthless with ceremonies and meetings, pare them down, keep only what’s necessary and what works. It’s even more important in maintenance than in development to remember that the goal is to deliver working code at the end of each time box. If some code is not working, or you’re not sure if it is working, then extend the deadline, back some of the changes out, or pull the plug on this release and start over. Don’t risk a production failure in order to hit an arbitrary deadline. If the team is having problems fitting work into time boxes, then stop and figure out what you’re doing wrong – the team is trying to do too much too fast, or the code is too unstable, or people don’t understand the code enough – and fix it and move on. Reviews and Retrospectives Retrospectives are important in maintenance to keep the team moving forward, to find better ways of working, and to solve problems. But like many practices, regular reviews reach a point of diminishing returns over time – people end up going through the motions. Once the team is setup, reviews don’t need to be done in each iteration unless the team runs into problems. Schedule reviews when you or the team need them. Collect data on how the team is working, on cycle time and bug report/fix ratios, correlate problems in production with changes, and get the team together to review if the numbers move off track. If the team runs into a serious problem like a major production failure, then get to the bottom of it through Root Cause Analysis. Sustainable pace / 40-hour week It’s not always possible to work a 40-hour week in maintenance. There are times when the team will be pushed to make urgent changes, spend late nights firefighting, releasing after hours and testing on weekends. But if this happens too often or goes on too long the team will burn out. It’s critical to establish a sustainable pace over the long term, to treat people fairly and give them a chance to do a good job. Pairing Pairing is hard to do in small teams where people are working on many different things. Pairing does make sense in some cases – people naturally pair-up when trying to debug a nasty problem or walking through a complicated change – but it’s not necessary to force it on people, and there are good reasons not to. Some teams (like mine) rely more on code reviews instead of pairing, or try to get developers to pair when first looking at a problem or change, and at the end again to review the code and tests. The important thing is to ensure that changes get looked at by at least one other person if possible, however this gets done. Collective Code Ownership Because maintenance teams are usually small and have to deal with a lot of different kinds of work, sooner or later different people will end up working on different parts of the code. It’s necessary, and it’s a good thing because people get a chance to learn more about the system and work with different technologies and on different problems. But there’s still a place for specialists in maintenance. You want the people who know the code the best to make emergency fixes or high-risk changes – or at least have them review the changes – because it has to work the first time. And sometimes you have no choice – sometimes there is only one person who understands a framework or language or technical problem well enough to get something done. Coding Guidelines – follow the rules Getting the team to follow coding guidelines is important in maintenance to help ensure the consistency and integrity of the code base over time – and to help ensure software security. Of course teams may have to compromise on coding standards and style conventions, depending on what they have inherited in the code base; and teams that maintain multiple systems will have to follow different guidelines for each system. Metaphor In XP, teams are supposed to share a Metaphor: a simple high-level expression of the system architecture (the system is a production line, or a bill of materials) and common names and patterns that can be used to describe the system. It’s a fuzzy concept at best, a weak substitute for more detailed architecture or design, and it’s not of much practical value in maintenance. Maintenance teams have to work with the architecture and patterns that are already in place in the system. What is important is making sure that the team has a common understanding of these patterns and the basic architecture so that the integrity isn’t lost – if it hasn’t been lost already. Getting the team together and reviewing the architecture, or reverse-engineering it, making sure that they all agree on it and documenting it in a simple way is important especially when taking over maintenance of a new system and when you are planning major changes. Simple Design Agile development teams start with simple designs and try to keep them simple. Maintenance teams have to work with whatever design and architecture that they inherit, which can be overwhelmingly complex, especially in bigger and older systems. But the driving principle should still be to design changes and new features as simple as the existing system lets you – and to simplify the system’s design further whenever you can. Especially when making small changes, simple, just-enough design is good – it means less documentation and less time and less cost. But maintenance teams need to be more risk adverse than development teams – even small mistakes can break compatibility or cause a run-time failure or open a security hole. This means that maintainers can’t be as iterative and free to take chances, and they need to spend more time upfront doing analysis, understanding the existing design and working through dependencies, as well as reviewing and testing their changes for regressions afterwards. Refactoring Refactoring takes on a lot of importance in maintenance. Every time a developer makes a change or fix they should consider how much refactoring work they should do and can do to make the code and design clearer and simpler, and to pay off technical debt. What and how much to refactor depends on what kind of work they are doing (making a well-thought-out isolated change, or doing shotgun surgery, or pushing out an emergency hot fix) and the time and risks involved, how well they understand the code, how good their tools are (development IDEs for Java and .NET at least have good built-in tools that make many refactorings simple and safe) and what kind of safety net they have in place to catch mistakes – automated tests, code reviews, static analysis. Some maintenance teams don’t refactor because they are too afraid of making mistakes. It’s a vicious circle – over time the code will get harder and harder to understand and change, and they will have more reasons to be more afraid. Others claim that a maintenance team is not working correctly if they don’t spend at least 50% of their time refactoring. The real answer is somewhere in between – enough refactoring to make changes and fixes safe. There are cases where extensive refactoring, restructuring or rewriting code is the right thing to do. Some code is too dangerous to change or too full of bugs to leave the way it is – studies show that in most systems, especially big systems, 80% of the bugs can cluster in 20% of the code. Restructuring or rewriting this code can pay off quickly, reducing problems in production, and significantly reducing the time needed to make changes and test them as you go forward. Continuous Testing Testing is even more important and necessary in maintenance than it is in development. And it’s a major part of maintenance costs. Most maintenance teams rely on developers to test their own changes and fixes by hand to make sure that the change worked and that they didn’t break anything as a side effect. Of course this makes testing expensive and inefficient and it limits how much work the team can do. In order to move fast, to make incremental changes and refactoring safe, the team needs a better safety net, by automating unit and functional tests and acceptance tests. It can take a long time to put in test scaffolding and tools and write a good set of automated tests. But even a simple test framework and a small set of core fat tests can pay back quickly in maintenance, because a lot changes (and bugs) tend to be concentrated in the same parts of the code – the same features, framework code and APIs get changed over and over again, and will need to be tested over and over again. You can start small, get these tests running quickly and reliably and get the team to rely on them, fill in the gaps with manual tests and reviews, and then fill out the tests over time. Once you have a basic test framework in place, developers can take advantage of TFD/TDD especially for bug fixes – the fix has to be tested anyways, so why not write the test first and make sure that you fixed what you were supposed to? Continuous Integration To get Continuous Testing to work, you need a Continuous Integration environment. Understanding, automating and streamlining the build and getting the CI server up and running and wiring in tests and static analysis checks and reporting can take a lot of work in an enterprise system, especially if you have to deal with multiple languages and platforms and dependencies between systems. But doing this work is also the foundation for simplifying release and deployment – frequent short releases means that release and deployment has to be made as simple as possible. Onsite Customer / Product Owner Working closely with the customer to make sure that the team is delivering what the customer needs when the customer needs it is as important in maintenance as it is in developing a new system. Getting a talented and committed Customer engaged is hard enough on a high-profile development project – but it’s even harder in maintenance. You may end up with too many customers with conflicting agendas competing for the team’s attention, or nobody who has the time or ability to answer questions and make decisions. Maintenance teams often have to make compromises and help fill in this role on their own. But it doesn’t all fit…. Kilner’s main point of concern isn’t really with Agile methods in maintenance. It’s with incremental design and development in general – that some work doesn’t fit nicely into short time boxes. Short iterations might work ok for bug fixes and small enhancements (they do), but sometimes you need to make bigger changes that have lots of dependencies. He argues that while Agile teams building new systems can stub out incomplete work and keep going in steps, maintenance teams have to get everything working all at once – it’s all or nothing. It’s not easy to see how big changes can be broken down into small steps that can be fit into short time boxes. I agree that this is harder in maintenance because you have to be more careful in understanding and untangling dependencies before you make changes, and you have to be more careful not to break things. The code and design will sometimes fight the kinds of changes that you need to make, because you need to do something that was never anticipated in the original design, or whatever design there was has been lost over time and any kind of change is hard to make. It’s not easy – but teams solve these problems all the time. You can use tools to figure out how much of a dependency mess you have in the code and what kind of changes you need to make to get out of this mess. If you are going to spend “weeks, months, or even years” to make changes to a system, then it makes sense to take time upfront to understand and break down build dependencies and isolate run-time dependencies, and put in test scaffolding and tests to protect the team from making mistakes as they go along. All of this can be done in time boxed steps. Just because you are following time boxes and simple, incremental design doesn’t mean that you start making changes without thinking them through. Read Working With Legacy Code – Michael Feathers walks through how to deal with these problems in detail, in both object oriented and procedural languages. What to do if it takes forever to make a change. How to break dependencies. How to find interception points and pinch points. How to find structure in the design and the code. What tests to write and how to get automated tests to work. Changing data in a production system, especially data shared with other systems, isn’t easy either. You need to plan out API changes and data structure changes as carefully as possible, but you can still make data and database changes in small, structured steps. To make code changes in steps you can use Branching by Abstraction where it makes sense (like making back-end changes) and you can protect customers from changes through Feature Flags and Dark Launching like Facebook and Twitter and Flickr do to continuously roll out changes – although you need to be careful, because if taken too far these practices can make code more fragile and harder to work with. Agile development teams follow incremental design and development to help them discover an optimal solution through trial-and-error. Maintenance teams work this way for a different reason – to manage technical risks by breaking big changes down and making small bets instead of big ones. Working this way means that you have to put in scaffolding (and remember to take it out afterwards) and plan out intermediate steps and review and test everything as you make each change. Sometimes it might feel like you are running in place, that it is taking longer and costing more. But getting there in small steps is much safer, and gives you a lot more control. Teams working on large legacy code bases and old technology platforms will have a harder time taking on these ideas and succeeding with them. But that doesn’t mean that they won’t work. Yes, you can be Agile in maintenance.

I'll start off by saying this article is not intended to be a Cassandra-bashing session, instead it provides an interesting look at one development company's case study to show that Cassandra (although it's fantastic for some) is not for everyone. The company is Nodeta and the application is Flowdock, a free tool (currently in beta) that functions as a web-based team messenger in place of Campfires, Skype Chats, IRCs, etc. Otto Hilska, who talked about the migration, says that "All software developers should be using it… because it better supports their actual workflow.." About a week ago the team finished their transition from the Apache Cassandra NoSQL database to another NoSQL, MongoDB. The switch was made due to stability issues that the developers were having with Cassandra. Hilska explained the details of his company's experience with Cassandra: " All nodes would go into an infinite loop, running GC and trying to compact the data files – occasionally falling off the cluster. We were unable to solve the problem, except that restarting and then compacting a node usually settled it down for a while. Other people had reported similar problems. Last couple of weeks our Cassandra nodes always ate all the resources they were given, slowing down Flowdock. This was not the first time we had run into problems because of our bleeding edge database choice. When upgrading from 0.4 to 0.5, we had to shut down the cluster, only to find out that it hadn’t flushed everything to the disk (even though we explicitly flushed it, as instructed). Thus we ended up having a couple of minutes of discussions lost, and our custom-built indices were miserably out of date and needed to be rebuilt. I think it was 4 AM when we finally got to leave the office."--Otto Hilska Flowdock developers became attracted to a new NoSQL store, MongoDB, because of its recent addition of auto-sharding and replica sets. Hilska wrote the conversion script in a day and it took a week to get Flowdock running purely on MongoDB. Then Nodeta tested it internally for a few weeks before they deployed it to production. However, MongoDB is not without flaws as well. Dots are not allowed in BSON document keys and the document size is limited to 4MB. It's also not as easy to add new nodes as it is with Cassandra. On the other hand, Hilska says that the smart (multikey) indices, complex queries directly from the console, MapReduce, GridFS, and lack of issues make up for these minor flaws. I recently posted a guide for determining the right database solution (Relational or NoSQL) for various use cases. The article has some very good resources and includes situations where MongoDB or Cassandra might be the best choice. Some criticisms of Cassandra emerged a few weeks ago when Twitter announced that it would over to the NoSQL store. By no means has Twitter stopped using Cassandra. They have stated that it's currently being used to store geolocation data and data mining results that feed into things like local trends and @toptweets. The NoSQL's creators at Facebook are also still using Cassandra. Production deployments of MongoDB exist at Foursquare, SourceForge, The New York Times, BoxedIce, GitHub, and SugarCRM.

So you've finished that shiny new website and you want make sure that you and your buddies are in control. Besides the obvious things such as SSL and logging all access, there are a fewest practices for authentication/access that developers recommend. Here are some of the recommendations: Require separate login pages for users and admin using the same DB table. This will prevent XSRF and session-stealing, plus the attacker won't be able to access to admin areas) [Thief Master] Use complex passwords for admin accounts. For example, "uvula{:&:>iuJ", not "12345". Of course, you have to remember it. :) [Developer Art] Introduce an artificial pause between each admin password attempt to prevent brute force attacks. [Lo'oris] Blocking users IP after a number of failed admin login attempts or requiring a CAPTCHA after a failed login (but not the first one, because that's really annoying) will also stop brute force attacks. [Thief Master] If the admin section is in a separate subdirectory, you should consider also adding webserver native authentication to that area (e.g. via .htaccess in Apache). Then an attacker would need both the subdirectory password and the user password. [Thief Master] Consider Second level authentication such as client certificates (e.g. x509 certs), smart cards, cardspace, etc. [JoeGeeky] Restrict access to the admin area. Only allow clients from trusted IPs/Domains. [JoeGeeky] Lock down IPrincipal & Principal-based authorization and make rights immutable and non-enumerable. Also make sure that all authorization assessments are based on the Principal. [JoeGeeky] Set up an email notification system that alerts admins when any rights are upgraded. This will help you catch an attacker that elevates his/her rights. [JoeGeeky] Consider fine-grained rights for admins. Typical Role-Based Security (RBS) approaches are not as safe because some roles will end up with more rights that they need. You should distribute rights based on the exact actions that a admin performs. This could cause a lot of overhead with more diverse admin-types, but it is safer because rights are issued more sparingly. [JoeGeeky] Restrict the creation of further admins and carefully control what admins can do to other admins. It's best to have a locked-down 'super-admin' client. [JoeGeeky] Consider Client Side SSL Certificates or RSA type keyfobs (electronic tokens) for added security. [Daniel Papasian] If you're using using cookies for authentication, use separate cookies for admin and normal pages. One way is to put the admin section on a different domain. [Daniel Papasian] One possibility, if it's practical, is to put the admin site on a private subnet instead of the internet. [John Hartsock] Re-issue auth/session tickets when moving between admin and normal usage contexts of the website. [Richard JP Le Guen] Require equally strong mechanisms (using the above techniques) for basic users so that admins aren't the only ones with highly-secure accounts. [Lo'oris] These tips were gathered in a question by UpTheCreek from StackOverflow.