How to assert that your SQL does not do full table scans

Thursday, March 25, 2010 3:18:00 PM (Romance Standard Time, UTC+01:00)

In database applications it is often important to ensure that your SQL uses indexes to avoid performance problems. Consider an SQL select statement like this

SELECT text FROM response WHERE questionId = 27 AND participantId = 38

If there is no index on response(questionId, participantId), the database needs to do a full table scan, and if the response table have a lot of rows, this will degrade performance. Such a performance problem might only be observed during testing on large amounts of data, which is not typically something a developer will do before every commit.

It would be useful to be able to detect this kind of problem much earlier, preferably during unit testing. Here I will describe one way to do this using Microsoft SQL Server and .NET:

SQL Server makes a lot of information accessible through performance counters, and one of those performance counters is incremented every time SQL Server does a full table scan. Utilizing that information, we can write a unit test that verifies that indexes are properly in place by asserting that the number of full table scans performed by a piece of code is equal to zero:

using(new AssertSqlFullScanCount(0))
{

var
responses = Response.GetResponses(q, p);
Assert.That(responses, ...);
}

This code uses the following helper class:

public class AssertSqlFullScanCount : IDisposable
{
private
int expected;
private long actual;
private PerformanceCounter fullScans;
public AssertSqlFullScanCount(int expected)
{
this.expected = expected;
this.fullScans = new PerformanceCounter(
"MSSQL$SQLEXPRESS:Access Methods",
"Full Scans/sec");
this.actual = fullScans.RawValue;
}

public
void Dispose()
{
actual = fullScans.RawValue - actual;

Assert.That(actual, Is.EqualTo(expected));
}
}

Do you consider this kind of performance unit testing useful? How would you implement this kind of tests with Oracle

By Lars Thorup

Slow and Brittle: Replacing End-to-End Testing

Friday, August 28, 2009 12:18:38 AM (Romance Standard Time, UTC+01:00)
At Agile 2009, I attended an interesting workshop on the problems of end-to-end testing. Here is the idea that I presented at the workshop:

I believe you need to have some end-to-end testing, because it helps you get fast feedback about end-to-end-related bugs. But we don't need to have a lot of those tests because then most of them test the same end-to-end-related stuff. The problem is that it is so easy to write those tests, especially for legacy code, so people end up having a lot of them. So a solution might be to put a limit on the amount of time that end-to-end testing is allowed to take. And of course we can let the build server enforce that limit by failing the build when the limit is exceeded. When a developer exceeds the limit, some of the end-to-end-tests must then be converted into isolated unit tests. When we only have a few end-to-end tests, neither their slowness nor their brittleness is such a big problem.

By Lars Thorup

Hudson and Sonar

Monday, March 23, 2009 11:13:39 PM (Romance Standard Time, UTC+01:00)

Some interesting new tools in the field of continuous integration and code analytics are getting to speed and are exceeding past the old first-movers.
Two such tools are the build server Hudson and the “Code quality management platform” of Sonar. The first gives you a very quick to install and get running build server, with a very inviting and intuitive user interface and an open platform for third party plugins. Forget the dinosaurish CruiseControl (for Java at least) as the open source build server, Hudson serves you way better.
Your development team will love the excellent dashboard with sunshine and clouds showing how your projects are doing, and the metrics graphs with history of build and test results, build times, tasks etc. In terms of “smartness” it is said to read your Maven pom.xml and ensure that dependent builds are ran in the correct order (still need to try that).



Sonar joins a number of open source code analytics tools and puts it into a nice but at bit over cluttered dashboard. Look here to see a Sonar site of the Apache frameworks. I just want to show one of the features which I find most useful, the coverage cloud:



It identifies the classes probably needing an extra look, because complexity is running high and code coverage are running low. Brilliant way of visualizing code quality.

But remember: one thing is to know the state of your code –  another is to act wisely on it. Code coverage derived from automated test suites does not tell you, if you have assertions in your unit test. Without assertions you can add and delete code without test failing.
The violations of code quality rules and the complexity measures of you components, does not tell you if your design is thought out well, or express the domain your trying to support, only a combined effort of analysis with your customer and experiments can reveal this for you. But on the of path of getting less variation in code and fewer defects – that this getting closer to software nirvana – I will not be without Hudson and Sonar giving me daily updates on how we are doing.
By Jesper Thaning

Add scripting to your system test engine

Tuesday, January 29, 2008 9:42:31 PM (Romance Standard Time, UTC+01:00)
I am currently working on an automated system test engine for a project developing a medical device. The test engine runs on a PC and can connect to the device through some network connection. In the device we have implemented a very simple command line application that allows us to control the device from the test engine. We can send a command like "key Accept" to the device which will act like the user had pressed the Accept-key.

To make automated system testing more manageable we thought about extending this small scripting language with language constructs like loops and subrutines. But we did not want to implement an entire scripting language from scratch. Instead it would be much easier if we could use an existing scripting language, like JavaScript which already contains these language constructs.

It turned out to be quite easy to add scripting to our .NET-based test engine, using the Microsoft Script Control. Now we can write a test scripts in JavaScript like the following:   
    // Test pressing a key 10 times
    var i;
    for(i = 0; i < 10; ++i)
    {
        device.key("a");
    }

The "device"-object exists inside the test engine and is written as a .NET class:

    public class DeviceProxy
    {
        public void key(string key)
        {
        sendCommand("key " + key);
            Console.WriteLine("Pressing '{0}'", key);
        }
    }
An instance of this class is added to the Script Control at startup:
    ScriptControl sc = new ScriptControl();
    DeviceProxy deviceProxy = new DeviceProxy();
    sc.Language = "JScript";
    sc.AddObject("device", deviceProxy, true);
    sc.AddCode(File.ReadAllText(scriptPath));

You also need to please the COM-to-.NET interop by setting ComVisible to
True in AssemblyInfo.cs. Take a look at the sample project.

By Lars Thorup

Is TDD Controversial?

Tuesday, October 16, 2007 8:27:51 PM (Romance Standard Time, UTC+01:00)

If you attended the JAOO 2007 conference, or follow the Test-Driven Development community you may have heard of Jim Coplien and his arguments about why you should not use TDD. I believe that Jim said something like "TDD will deteriorate your design", and it made me ask myself -- Is TDD a controversial topic?

So is it? Not if you ask me. If you rewrite the original Red-Green-Refactor mantra from TDD, into "Spec. the feature" > "Implement it" > "Reduce technical debt", it seems to make quite a lot of sense. In fact, I don't really see any other way you could work and maintain your professional integrity!

So is there any risk of bad design or architecture as a result of using Test-Driven Development? Well, only if you neglect that refactoring is a continuous activity of TDD. If you refactor (small and large refactorings) on a continuous basis, I simply do not see how you could have a design- or architectural meltdown. If anyone experience this, I seriously doubt that TDD is the root cause of their problem.

Check out Roy Osherove's blog entry on the topic here.

By Sune Gynthersen

Test Driven Thread Safety

Friday, September 14, 2007 6:22:46 PM (Romance Standard Time, UTC+01:00)
The other day I needed a thread safe collection class. Since .NET 2.0 lack thread safe generic collection classes and because I wanted a ReaderWriterLock for best throughput I decided to write it myself.
Being a TDD-kind of guy, I wanted to start with some failing tests, and focusing on the thread safety of the class I wanted the tests to fail because of lacking thread safety. In this way I would know for sure when the added thread safety mechanisms was actually working.
But how do you write a test that fails because of lacking thread safety? Well, in this case I could start up some threads, make sure that they used the collection methods simultaneously and then assert that only one thread was allowed to do so at a time.
That turned out to be more easy to think than to code. Lets look at the two problems and their solution:
"Making sure that the threads used the collection methods simultaneously". The collection methods are typically so fast that it is practically impossible to arrange for two threads to try to enter a method simultaneously. However, I decided that for testing purposes, I could slow down the methods and thereby ensure that the first thread was still inside the method when the next thread came around. So inside the Collection.Add()-method I added:
   if (assertEnabled)
{
Thread.Sleep(100);
}

"Asserting that only one thread at a time is allowed to execute a method". A thread cannot know whether other threads are running the same method body. However, inside the method body, we can count the number of threads entering and exiting. And then we can add Assert-statements stating the required restriction on the number of concurrent threads in the method. So again, inside the Collection.Add()-method I added:
   if (assertEnabled)
{
Interlocked.Increment(ref writerCount);
Thread.Sleep(100);
Assert.AreEqual(1, writerCount);
Assert.AreEqual(0, readerCount, "trying to write while reading");
}
// the implementation of the method go here
if (assertEnabled)
{
Interlocked.Decrement(ref writerCount);
}

Having these mechanisms in place I could write a test that failed because of the "trying to write while reading"-exception being thrown.
   [Test]
public void WriterWaitForReader()
{
Thread r1 = CreateReader();
Thread w1 = CreateWriter();
r1.Join();
w1.Join();
Assert.That(exception, Is.Null);
}

Now having a failing test, allowed me to add the actual thread safety mechanism using the ReaderWriterLock in the Collection.Add()-method:
   rwLock.AcquireWriterLock(InfiniteTimeOut);
if (assertEnabled)
{
Interlocked.Increment(ref writerCount);
Thread.Sleep(100);
Assert.AreEqual(1, writerCount);
Assert.AreEqual(0, readerCount, "trying to write while reading");
}
// the implementation of the method go here
if (assertEnabled)
{
Interlocked.Decrement(ref writerCount);
}
rwLock.ReleaseLock();

Bringing me a succeeding test. Then I could add tests for the actual functionality of my collection class and the corresponding implementation.
Depending on the project in which this code is being used I could keep asserEnabled==true in production as well as when running the tests (this requires another condition to avoid enabling the Sleep()). Another choice could be to entirely avoid the overhead of the assertEnabled condition by surrounding it with #if DEBUG ... #endif, so the tests only run in debug mode.
Take a look at the example source code (for Visual Studio 2008). By Lars Thorup

"Stop The Line" Issues

Sunday, September 02, 2007 9:18:21 PM (Romance Standard Time, UTC+01:00)

Creating quality software on a continuous basis is a complicated business.

I think it is a sign of maturity if your team can admit that there are daily problems which are ultimately obstructing flow. While not everyone will appreciate that your team does a complete halt until the obstruction has been removed I think it is a necessity for a team to stay in good shape and continue to deliver frequently.

Just the other day I came to think about what were the most common obstructions of flow we see in software development environments. So here is my preliminary list -- I bet you have seen them all.

1. Build breaks
If you don't have a build, you don't have anything. Nobody is able to do any progress progress as the state of the product is unknown. Developers cannot check-in, as they will not get any response from the continuous integration environment. Testers cannot try out the latest build. Users cannot see the product in action, and the list goes on. A build break is like a blindfold, and you should treat it as such. Don't try to go anywhere until you have your vision back.

2. Test failures
If you have a failing test, you have dysfunctional software. Imaging somebody fitting some cool new styling parts on to a car with a busted engine. Now, who would ever do that? It really is the same with software. Why do any feature work on a software product which has a failing test? I know this can be hard to respect, especially when a deadline is closing in on the team. Deadlines however, does not justify continuing without stopping the line upon test failures.

3. Bug reports
If you have bug reports, you have dysfunctional software. There is no need to do any feature work, until the product is stabilized. Bug reports are even worse than failing automated tests as those bugs can only be reproduced by a human. You should not spend work hours on anything but writing tests for these bugs, and fixing them.

4. Any other blocks keeping the team from a steady flow
If you are experiencing an obstruction which is keeping from moving forward as fast as you could, suggest to the team that you stop doing anymore feature work, until you are able to eliminate the obstruction. One obstacle which I think is quite common is builds that take way to long time. "Yeah, but how can you build 2 million lines of code in less than two hours?" -- Well, I suppose you could start by reorganizing your code and build only the module related to a particular source control check-in. Hopefully you don't have modules consisting of 2 million lines of code!

I'd like to hear about any other obstructions you may have experienced in a software development environment.

By Sune Gynthersen

Automated Test of Swing GUI using Fest

Thursday, August 16, 2007 10:02:34 PM (Romance Standard Time, UTC+01:00)

At my current project I was hired to make some Swing GUI talking to an EJB server. I was new to Swing GUI but I quickly began searching for a test framework fitted for testing them. I came across Fest (Fixtures for Easy Software Testing). Fest is a quite new project, so the API is still subject to some changes (its working towards a “fluent” style), but you can definitely get something out of the current version.

I came from a project where I used Selenium to make automated functional test of web sites, and in fact Fest and Selenium are quite alike in the way test are defined. Its mostly about finding controls like fields and buttons (or links) and then enter text into them or click on them, and off course assert on some attribute of the UI.

When using Selenium you often find yourself setting id attributes on e.g. the links to be able to find them in your test; in Fest you get the same testability of the GUI by setting names on the Swing components. That way you can find them in your test, and avoid basing your test on a specific position in a component hierarchy that are likely to change.


A test in Fest could look like this:

package fest;

import javax.swing.JDialog;
import javax.swing.JTable;

import junit.framework.Assert;
import junit.framework.TestCase;

import org.fest.swing.RobotFixture;
import org.fest.swing.fixture.FrameFixture;

public class FestTest extends TestCase
{
private RobotFixture _robotFixture;
private RecieverFrame _recieverFrame;

private FrameFixture _recieverFrameFixture;
private FrameFixture _senderFrameFixture;

public void setUp()
throws Exception
{
_robotFixture = RobotFixture.robotWithNewAwtHierarchy();
_recieverFrame = new RecieverFrame();
_recieverFrameFixture = new FrameFixture(_robotFixture,
_recieverFrame);
_recieverFrame.pack();

/*
* Make sender frame last, to be able to click on
* buttons as it requires the frame to be in focus
*/
SenderFrame senderFrame = new SenderFrame();
_senderFrameFixture = new FrameFixture(_robotFixture,
senderFrame);
senderFrame.pack();
_robotFixture.showWindow(senderFrame);
}

public void tearDown()
{
_robotFixture.cleanUp();
}

public void testSendMessageToReciever()
throws Exception
{
/*
* Verify that reciever is in a correct initial state
*/
assertEquals("No message", _recieverFrameFixture
.textBox("messageField").text());

_senderFrameFixture.comboBox("forumSelector")
.selectItemWithText("FestForum");
_senderFrameFixture.button("foo").click();

Thread.sleep(500);

_recieverFrameFixture.textBox("messageField")
.requireText("foo");

/*
* Example of finding a Swing component and asserting
* directly on it
*/
JTable messageTable = _robotFixture.finder()
.findByType(JTable.class);
assertEquals("Wrong number of messages", 1,
messageTable.getRowCount());

_senderFrameFixture.button("deleteAllMessages").click();

Thread.sleep(500);

assertEquals("No message", _recieverFrameFixture
.textBox("messageField").text());
assertEquals("Wrong number of messages", 0,
messageTable.getRowCount());

/*
* This is a bit clumsy way to verify that no error
* dialog is present, improvements are possible.
*/
try
{
JDialog errorDialog = _robotFixture.finder()
.findByType(JDialog.class);
Assert
.fail("Expected no dialog present found one '"
+ errorDialog.getTitle() + "'");
}
catch (Exception e)
{
}
}
}

Notice the use of finders to get the components in the Swing GUI, you can choose to find a component by name or class, where the latter makes your test more fragile, I prefer the find-by-name where possible. And it is easy to make your own Matchers to suit your purpose.

A nice feature is that Fest fails if you try to click on a button that is not visible in the frame, verifying that your frame is big enough to contain all your buttons. To improve Fest one could make the assertion failures more informative, they tend to be “Expected false but was true” with little context from the test.

I am using Fest to test multiple Swing frames at a time that communicate together via JMS (Java Messaging) and this can easily be done with Fest. One of the challenges when testing such asynchronous behavior is that your test need to be sleep between invocation and assertion. This is by the way the same challenge you face when testing ajax'ified websites, because your test need to wait until the DOM has been updated by the ajax call.

Fest requires Java5 and can be used in JUnit3 or whatever newer test framework you are using.

Thanks to Alex Ruiz  for putting together Fest.

By Jesper Thaning