Monthly Archives: January 2009

Limiting integration tests to a time constraint

4 Replies

I have recently had the need to restrict a test to run within a certain time limit. If the test does not complete within a specific time constraint then the test should fail. The approach that I took was to run the function under test in its own thread and use a CountDownLatch to specify the duration in which the process should complete within. The code for this is as follows.

public class TimeConstrainedIntegrationTest {

    private static final int RUN_ONCE = 1;

    @Test
    public void shouldVerifyGoogleIsAlive() throws Exception {
        CountDownLatch doneSignal = new CountDownLatch(RUN_ONCE);

        new Thread(new VerifyGoogleIsAliveWorker(doneSignal)).start();

        boolean finishedInTime = doneSignal.await(10, TimeUnit.SECONDS);
        assertTrue(finishedInTime);
    }

    private class VerifyGoogleIsAliveWorker implements Runnable {
        private final CountDownLatch doneSignal;

        VerifyGoogleIsAliveWorker(CountDownLatch doneSignal) {
            this.doneSignal = doneSignal;
        }

        public void run() {
            try {
                try {
                    verifyGoogleIsAlive();
                } catch (IOException e) {
                    throw new InterruptedException();
                }

                doneSignal.countDown();
            } catch (InterruptedException ex) {
            }
        }

        private void verifyGoogleIsAlive() throws IOException {
            WebClient webClient = new WebClient();
            Page page = webClient.getPage("http://google.com");
            assertTrue(StringUtils.contains(page.getTitleText(), "Google"));
        }
    }
}

In the code above I assert that Google is alive by checking the title text for http://google.com is in fact Google. Essentially I am asserting expected behaviour, which is what you would typically do. I have encapsulated this within the verifyGoogleIsAlive() method above.

To make sure that the test runs within a specific time, the verifyGoogleIsAlive() method is called within the run method of a class that implements Runnable. In this case I have called the class VerifyGoogleIsAliveWorker. The test case shouldVerifyGoogleIsAlive() should be quite self-explanatory. First a CountDownLatch is instantiated. Then a new VerifyGoogleIsAliveWorker thread is started. The CountDownLatch then awaits for the VerifyGoogleIsAliveWorker thread to run its course and complete.

Of interest though is that the awaits() method on the CountDownLatch allows you to specify the length of time in which the CountDownLatch should wait before ending the thread. If the process doesn’t complete within the specified time, then the awaits() method will return false, and therefore fail the assertion that the test will run within a specific time. If on the other hand the process completes within the allocated time frame then the test will pass.

The above concurrency pattern should be useful in making sure that your tests run within a specific time frame.

Velocity slowing? Then you need JET (Just Enough Testing).

11 Replies

The quickest way to get code into production is not to write tests. [Waits for reader to stop shouting "That is blasphemy!".] Now hear me out, this statement is true based on the assumption that the code will run as expected. However, you cannot know with certainty that your code will run as expected without testing it. Therefore you can write code and deploy it into production quickly by not writing tests, but it does not mean that the code will be defect free. As a result you run the risk of spending an indefinite amount of time fixing defects post release. This takes your time away from developing new features that add more business value.

The opposite is to spend more time upfront writing tests. This results in a longer period of development effort to get code into production, but you have more confidence that your code will run as expected. Which leaves you more time post release to work on new functionality that will add business value, instead of being swamped with fixing defects.

Now you have two contrasting views. The first is that you can quickly deploy code into production by not writing tests, but you have almost no confidence that the code will work as expected. The second is to write tests to improve confidence, but it will take longer to realise any business value. These views provide a sliding scale for software quality, at one end you have no tests and thus almost no quality assurance. At the other end you have unit tests, integration tests, functional tests, and more tests, all of which provide you with a high degree of quality. In a perfect world you would opt for the latter. But we don’t live in a perfect world. Everything costs money. Clients have a finite amount of money to spend on projects that deliver business value, and they want the best bang for their buck.

In the Agile world business value is proportional to the development team’s velocity. The higher the velocity the more business value you’ll eventually get. On a recent project we tested everything, and I mean everything. We had unit tests, integration tests, functional tests, and end-to-end tests. We did TDD, so most of these tests were written before a single line of code was written. Our test coverage was in the 90 percentile range. We had a high degree of confidence that what went into production was bullet proof. This is not to say that there were no defects, there were some defects, but they were all resolved in a timely manner. This kind of leads me to the main point of this post, you can test the hell out of something but it doesn’t guarantee that it will lead to defect free software. In other words you can have 100% test coverage but your software may not deliver what the customer wants. Or you simply didn’t cover scenarios in which your software will be used.

DSC00174aa

On this particular project the iteration manager (Agile project manager) would always state “we should be going faster”. He was right. We had a team of highly capable developers, and some of the things we were doing was not rocket science, but the business domain was quite complex. We were not experts in the business domain, and I believe we compensated by writing a lot more tests than we needed to. Through the power of hindsight I can say this, but on the project the team was geared towards writing tests. In our retrospectives the question of “how can we go faster?” was always raised. There were many useful suggestions, but nobody dared to utter the “T” word. I was always thinking it but didn’t have the balls to say it. Is this a symptom of group think? That as a team or organisation for that matter, we believe that writing tests are good, and so writing a lot more tests must therefore be better. It hardly seems Lean.

The signs that our tests were getting in the way are now glaringly clear. We had a nightly build that ran a large suite of acceptance tests and with a large set of data. Most mornings were spent figuring out why the nightly build failed. Sometimes it was due to an integration server going down, sometimes it was due to a genuine bug being exposed by testing with a large data set. The problem here is that there were a number of false positives that took time to verify and as a result took time away from working on a story.

Another sign was that our pre-commit builds took about 15 minutes. These builds included a decent size set of Selenium based functional tests. Now 15 minutes is a hell of a lot of time to be waiting around for your build to finish. To get an idea of what it was like take this xkcd comic and replace the word “compiling” with “building”. It not only holds you up, but the rest of the team as well. We had this practice of sequential check-ins, in that you have to be holding the “check-in chicken” before you were allowed to check in any code. This queued up check-ins in that you had to wait until the chicken holder had finished their build and checked in, before you were allowed to build and check in your code. It sucked even more when there were two or three other pairs in front of you. Most of the time you would just keep working, but this prevented you from doing small atomic commits, which resulted in larger commits and conflict problems.

Tests do slow you down, and it is pure ignorance to believe otherwise. It wasn’t until my most recent project where my first instinct was to test the hell out of something. The team lead on this project asked “why do you want to write that acceptance tests?”. My initial guarded response was “are you kidding me?”. This led to a massive debate over the value of writing tests. This debate was mostly between the team lead and another developer. I was too absorbed in the concept of not having to write tests for everything. Picture a cloudy sky opening up and a ray of sunshine falling down upon me. That was how I felt when I found another person that challenged the status quo of test the mofo out of everything. The default stance that we take is test everything. Usually this is done with little thought as to whether extensive testing is really required or not. Whether the client asks for it or not they are going to get a thoroughly tested system.

This leads me to the main purpose of this post, to challenge this default stance of test everything thoroughly, no matter how long it takes. It came about through the debate that we had about testing, which led to the team lead putting us in the client’s shoes. If hypothetically you had $100,000 and needed to have your website built, would you be happy that you got one piece of functionality and about five lots of tests for that functionality? Probably not. You might be happier if you instead received three pieces of functionality and a test for each functionality. I am over trivialising this, but you should get the point. Economist would call this disposition of utility gained from having extra tests in exchange for functionality marginal utility.

As in economics I believe the law of diminishing marginal utility also applies to writing tests. In other words there comes a point where the utility of writing tests detracts from the overall goal of delivering business value. This leads me to the JET acronym that I coined which stands for Just Enough Testing. It is loosely based on the concept of JeOS or Just enough Operating System. JeOS allows you to start with a stripped down operating system to run in virtual machines, then add additional services to the OS to suit the environment in which it will be used. I would like to use JeOS as an anology as to how JET should be used. JET is a stripped down testing strategy that you must adopt, then build on this strategy to suit the project that you are on. Simply adopting the stance of test everything is not a strategy that should be employed on every project. The clients’ demands will be different in all cases, and their marginal utility for business value over tests will vary.

Modern Jet Fighter Display

My JET approach is to write unit tests for designing the interactions between the class under test and its collaborators. Integration tests mostly for end-to-end testing, but with the awareness of not going nuts here. Finally having a minimal set of functional tests using Selenium to cover typical workflows through a web application for example.

Adjusting JET to your project is easy. If you have the luxury of a QA for example, then get them to spend more time in writing functional tests so that all regression tests become fully automated. If your site is quite simple on the front end then your minimal set of functional tests should give you enough confidence that your application will work as expected. If however your back end is more complex, then spend a bit more time writing integration tests in that area. JET is not about writing less tests. It is about realising that projects have constraints and you need to work effectively within those constraints, and that means thinking about whether you really need to have that functional test or not, because maybe just having an integration test will give you enough confidence as to how your code will work.

In all honesty JET is just a cool acronym that relates quite nicely to velocity. Whether JET takes off (I couldn’t help myself with that pun) or not is completely left to you to decide. I’m not the only one that thinks it is perfectly ok to compromise when writing tests. Kent Beck also thinks so in his article Where, Oh Where To Test. Kent Beck comes to the conclusion in that article that a pragmatic programmer should decide what to test and how much testing is required based on three criteria: cost; reliability; and stability. From the article: “The testing strategy that delivers the needed confidence at the least cost should win.” In other words you should do just enough testing.

I know there are plenty of test zealots out there. I’ve worked with some of them, and know their counter arguments quite intimately, and they are only sound arguments in a perfect world where a client has an infinite budget for a project. For those people I have provided the pretty pictures of jet aircraft above. For the rest I hope you find the post to be an interesting read.

Building a social networking website

7 Replies

I spent my downtime working over the holidays on a social networking website. I had a lot of fun developing something that I would not do for my day to day job. First of all I was using a technology stack that I liked, and one that is not widely used in ThoughtWorks. I have been building my site using Django with PostgreSQL as my database. After using Sybase on my last project at work, I needed to cleanse myself from having used the crappiest database ever. It was great to be back using good ol’ PostgreSQL.

The main reason for using Django is to use GeoDjango, which makes handling location based data a lot easier. Especially if you plan on using a service like Yahoo’s Fire Eagle, which provides an API for location based social networks. GeoDjango uses PostGIS on top of PostgreSQL to store data in a location friendly format. The domain model and geographic data in GeoDjango makes it easy to create a point (latitude and longitude) on Earth and do things like determine if it is located within a mapped out region or not. This is essentially where I want to go, but I might be limited by my hosting service. Need to work out if I can install all the dependencies for GeoDjango on my shared server at Joyent.

I started with the look and feel of the website, so I have my css styles and branding sorted. I now need to integrate my templates with the user registration functionality. Then start building out the functionality for supporting community events. I will release more details about the site as I progress, so stay tuned!

What credit crunch?

Leave a reply

Finally got around to my first post for the new year. I gave the credit card a good workout during the holidays, which has left me wondering “what credit crunch?”. There were thousands of people out and about buying stuff and “stimulating the economy”. You even have the sales assistants at big department stores saying that they have had the busiest Boxing Day sales ever. I guess the biggest losers out of last year were the bankers, and rightly so. They have been bad boys and girls, and ought to have received sacks of coal from Santa for Christmas.

I also gave my financial planner the flick. The downturn in the economy was a blessing in disguise. Basically it showed me that financial planners are nothing more than glorified sales people. They know very little more about financial markets than you, except for the financial products that they get big kick backs for from funds managers. Investing is not complicated nor time consuming if you take the effort to do your own independent learning on financial markets. I thought I would be too busy to look after my personal finances, so I outsourced it to a financial planner. That turned out to be the worst mistake I’ve made, but I’ve learned from it and moved on. The first rule of investing should be never relinquish control of your finances. Now I am free to invest in what I like without the millionaire’s factory taxing me for doing sweet FA.