Monday, November 28, 2005

Continuous Integration with AsUnit?

We received another excellent question today on the asunit-users list. This one came from Ian Tyrell.

I believe he wanted to know if AsUnit could be coerced to play nicely with other mainstream continuous integration tools. Specifically, he asked, "...what would be needed to take things a step further and automatically run/evaluate the tests. What’s stopped me in the past is the use of LocalConnection between tests and TestUI"...

Well Ian, we haven't really brought AsUnit 2.x to that point yet, but AsUnit 3.x is pretty much ready to go. Fortunately, there is still hope for ActionScript 2.0 developers!

Many thanks to some genius at Macromedia, there is a little-known feature of the Flash Debug Player that allows it to output errors and trace statements to a text file - anywhere on your system.

Following are some articles that extoll the virtues of this feature in depth:

Greg Wygonik
actionscript.org
livedocs.macromedia.com


Once this has been properly configured, you can then write or implement whatever scripts you want to watch changes made to that file, and when some knucklehead (like me) checks in broken code, an email can be sent to the entire team with the failure log and even a list of the most recent version control checkins.

I haven't actually set up a system like this yet, because so far, Ali and I have been able to work pretty much by ourselves over the past couple of years, and if someone checks in broken code - it's usually pretty easy to guess who it was, and - I'm not too hard to find!

As it stands, it should be pretty easy to dig into the sources associated with AsUnit 2.x and make it "trace" the output rather than (or in addition to) sending it to the TextField that it currently sends to.

Ultimately, if you can get the AsUnit UI to send the contents of the TextField to a trace statement instead of the TextField, you should be able to pretty easily set up some service to watch that .log file and provide whatever embarrasing notification you deem necessary...

Please let me know if you get this working, as I would love to update the sources with your changes so that others can also benefit.


Good Luck,


Luke Bayes
www.asunit.org

Friday, November 25, 2005

Testing Asynchronous Functionality...

I encountered an excellent question today on the asunit-users mailing list. AliasRob asked:

>> Just wondering if anyone has had any interesting stories or
>> experiences regarding unit testing asynchronous functions, such as
>> loading a .swf, or waiting for a database or remoting call to return.
>> I'm wondering if there's a particular strategy that's becoming
>> popular.

The testing of asynchronous functionality has been the subject of deliberation by many folks that are smarter and more experienced than I. Following, I'll do my best to describe this problem and it's potential solutions as I have understood them from these other people...

The Problem:
You have a bit of functionality that does not execute in a single thread of execution that you would like to test and verify.


The Information:
There are some fundamental tidbits related to Unit Testing that ought to receive some consideration before we go much further.


  1. There should be no dependency from one test method to another. Test methods should be decoupled.

  2. There should be no dependency from one test case to another. Test cases should be decoupled.

  3. As with applications, test fixtures should attempt, at all times, to avoid code duplication and all other code smells (as outlined in "Refactoring").

  4. A Unit Test should be designed to test the smallest bit of functionality, and nothing else.

  5. A complete application test fixture should execute as quickly as possible in order to ensure that we continue to run it.

  6. The setUp and tearDown methods of the abstract TestCase are overridden and implemented to perform (and clean up from) the complete configuration that needs to be in place for each test method in a test case. These methods will be executed exactly one time before and after (respectively) calls to each method whose declaration matches the expression "public function test*".

  7. One can use Mock objects for a number of reasons, one of which is as a stub or faux representation of an actually-networked resource.

  8. Test cases should never (the emphasis here is mine, most books I have read never say never) require access to the internet in order to successfully execute.



The Solution:
With all that said, the answer to your question is that it depends on exactly what you're testing. If you're attempting to test the backend of a system using it's client, you should stop. The server side of a system should be unit tested independently of any client software. These two halves of an application should not ever be tightly-coupled. The developer or developers responsible for the server should write unit tests that ensure the server is working as expected in whatever language and using whatever technology the server was written in. Likewise, the developers responsible for the client should test its features independently.

How does one test the interface between these two systems?

That's an excellent question. It is done by building a "mock" or "stub" of the server that doesn't actually connect to anything, but instead returns consistent, known results to API queries.

Then, the server developer can build herself a "mock" or "stub" client that behaves exactly the same way in order to test her side of the connection.

Neither of these tests should require network connections.

The biggest issue with this approach occurs if your server has an extensive, large set of complicated APIs. I have often found that the server side of a system should usually have only a small handful of interface methods, and when I have encountered systems that actually did have an extensive interface, it seems that it was usually a code smell where we should have been reading the book, "Enterprise Integration Patterns", instead of over-engineering the connection between our efforts.

In another scenario, you may want to test some piece of functionality like an "xml parser" of sorts or some such other feature that is much easier to test if the data does not have to be managed in ActionScript.

For example, the following is a hideous thing to uncover and later have to manage:


private function getData():XML {
    var str:String = "";
    str += "<Application id='foo'>";
    str += " <child id='bar' />";
    str += "</Application>";

    return new XML(str);
}


For this use case, we have added full support for a truly asynchronous test case. Following is an example concrete TestCase that will execute asynchronously.

The first method called by the TestCase constructor is the "run()" method. In the following example, we override run(), and instead of doing other work in the override, we simply create and request an XML document that resides in our test folders next to the test case in question. In this case, we are using the AsUnit-provided TestCaseXml class which will, upon load completion, call "onXmlLoaded" on whatever object was passed as the second argument of it's constructor.

You'll notice that in that handler, we finally call super.run(). This begins execution of the test case only after a successful load of the xml document. This process can be similarly implemented to load and verify external assets such as swfs or images as well.

import org.yourdomain.data.XmlParser;
import com.asunit.framework.*;

class org.lifebin.data.XmlParserTest extends TestCase {
    private var className:String = "org.lifebin.data.XmlParserTest";
    private var parser:XmlParser;
    private var xmlData:TestCaseXml;
    private var xmlPath:String = "com/yourdomain/data/XmlParserData.xml";

    // This method begins execution of the abstract TestCase
    // We are overriding it here to effectively pause that execution
    // until our xml data is loaded.
    public function run():Void {
        xmlData = new XmlTransport(xmlPath, this);
    }

    public function onXmlLoaded(data:XMLNode):Void {
        super.run();
    }

    public function setUp():Void {
        parser = new XmlParser(xmlData.cloneNode(true));

    }

    public function tearDown():Void {
        delete parser;
    }

    public function testInstantiated():Void {
        assertTrue("XmlParser instantiated", parser instanceof XmlParser);
    }

    public function testParser():Void {
        assertTrue(false); // failing test
    }
}

In closing, I just want to make the point that the asynchronous test case is really only valuable when we want to store "configuration" data in a more easily manageable form. EG, if I have an image parser in ActionScript 3 that will examine each pixel of an image as a ByteArray, it is much easier to load an external, known image than it is to write out some 300k ByteArray in a setUp method. Similarly, it is often easier to test various Xml parsing implementations by actually using a separate xml example document. Keep in mind that the test for these parsers are not intended to also test the "network" or the "server".

One should also note that in this example setUp method, we use the "cloneNode" method of the flash XML object. We also pass in "true" as an argument. This performs a "deep clone" on the loaded XML data structure so that we have a clean copy of the data exactly as it was loaded for each testMethod that is executed.

Thanks,


Luke Bayes
www.asunit.org