Home > java, Subversion > Writing Functional Tests on Groovy on Grails: Experiences from CollabNet Subversion Edge.

Writing Functional Tests on Groovy on Grails: Experiences from CollabNet Subversion Edge.


I first wrote this technical document for the open-source project CollabNet Subversion Edge on how to design and implement Functional Tests using Groovy on Grails. Therefore, this documentation can also be reached at “https://ctf.open.collab.net/sf/wiki/do/viewPage/projects.svnedge/wiki/FunctionalTests

Introduction and Setup

The CollabNet Subversion Edge Functional Tests are based on the Groovy on Grails plugin “Functional Tests“. But, before you get started with them, make sure you have covered the following required steps:

Besides Unit and Integration tests used during development, the source-code under development already contains the functional tests plugin support and some developed classes, as shown in the Eclipse view “Project Explorer”. In the file system, the files are located at CSVN_DEV/test/functional, where the directory CSVN_DEV is where you have previously checked out the source-code. Those set of test cases are the last one being run in our internal Continuous Integration server (Hudson) and it’s usually a good place to find bugs related to the user-facing features during development.

svnedge-functional-tests-view-eclipse.png

Functional Tests Basics

This section covers the basics of the functional tests infranstruction on Subversion Edge and assumes you are already familiar with the Grails Functional Tests plugin documentation. The plugin is already installed in the Subversion Edge development project, as you can use the commands to run a functional test and visualize the test results and report. The test cases are run as RESTful calls to the controllers defined by the application, but it can also use a URL. For instance:

After the execution of an HTTP method wrapper such as “get()” or “post()”, any test case has the access to the response object “this.response” with the HTML code payload. Grails uses this object to execute any of the “assert*” methods documented.

Another important piece of configuration is the CSVN_DEV/grails-app/conf/Config.groovy. Althoug Grails grails uses the closure “environment.test”, SvnEdge uses the general closure “svnedge” during development and test phases and, therefore, values from configuration of that closure are accessable from the test classes.

Functional tests infrastructure

Considering you have your development infrastructure set up, you will find the current implementation of functional tests at the directory “CSVN_DEV/tests/functional”. Notice that the directory structure follows the Java convention for declaring packages, and has already been configured to be included as source-code directories in the current Eclipse configuration artifact “.classpath” on trunk.

csvn-functional-tests-packages.png

We have created the following convention for defining the packages and functional test classes:

  • com.collabnet.svnedge

The package containing major abstract classes to embrace code reuse while aggregating reusable patterns throughout the entire Test infrastructure. The reusable utility methods were extracted during the first iteration of the development of the SvnEdge functional tests. For instance, the access to the configuration keys from “CSVN_DEV/grails-app/conf/Config.groovy” can be easily accessed from the test cases using the method “getConfig()” or just “config”. Similarly, the access to the i18n keys can be performed by calling “getMessage(‘key’)”, as the value of “key” is one of the keys located at the “messages.properties”, which renders strings displayed in the User Interface. Note that the English version of the i18n messages are used in the functional tests. Moreover, the abstract classes have their own intent for the scenarios found on Subversion Edge functionalities:

  1. AdminLoggedInAbstractSvnEdgeFunctionalTests: test class that sets up the test case with the user “admin” already logged in (“/csvn/status/index”).
  2. LoggedOffAbstractSvnEdgeFunctionalTests: test class that starts the application in the index auth page where a user can login (“/csvn/auth/index”).
  • com.collabnet.svnedge.console

The test cases related to the web console, or Subversion Edge itself. Different components must have its own package. For instance, take the name of the controllers to as the name of the component to be tested such as “user” and “repo”, as they should have their own test packages as “com.collabnet.svnedge.console.ui.user” and “com.collabnet.svnedge.console.ui.repo”, respectively. The only classes implemented at this time are the login

  • com.collabnet.svnedge.teamforge

The test cases related to the teamforge integration. As you will see, there are only one abstract class and two functional classes covering the functional tests of the conversion process when the server has repositories to be imported (Full Conversion) and when the server does not have any repository created (Fresh Conversion). The latter case is a bit tricky as the SvnEdge environment defines a fresh conversion when its database does not have any repository defined. In this way, test cases related to repositories need to make sure to “discover” repositories if the intent is to verify the existence of repositories in the file-system.

Running Functional Tests

As described in Grails functional tests “mini bible”, the only thing needed to run a functional test case is the following command under the directory CSVN_DEV:

grails test-app -functional OPTIONAL_CLASS_NAME

The command will start the current version of Grails installed using the functional tests environment. If you don’t provide the optional parameter “OPTIONAL_CLASS_NAME”, grails executes all the functional tests defined. However, since the execution of all current implementation of test classes takes more than 10 minutes, use the complete name of the test class (package name + name of the class – sufix “Tests”). For instance, the following command executes the functional Tests implemented in the class LoginFunctionalTests:

grails test-app -functional com.collabnet.svnedge.console.ui.LoginFunctional

The command selects the test suite class “CSVN_DEV/tests/functional/com/collabnet/svnedge/console/ui/LoginFunctionalTests.groovy” to be executed, as the output of the execution of the test cases identify the environment and the location where the test reports will be saved. The recommendation here is to keep using the Eclipse STS infrastructure to save your commands execution as shown below.

svnedge-eclipse-saved-execution.png

As shown below, the functional tests execution output is the same from executing the tests using the command line or the Eclipse command as shown in the output view. The tests are prepared to be executed and save the output logs and reports in the directory “CSVN_DEV/target/test-reports”.

Welcome to Grails 1.3.4 - http://grails.org/
Licensed under Apache Standard License 2.0
Grails home is set to: /u1/svnedge/replica_admin/grails/grails-1.3.4/

Base Directory: /u1/development/workspaces/collabnet/svnedge-1.3.4/console
Resolving dependencies...
Dependencies resolved in 1565ms.
Running script /u1/svnedge/replica_admin/grails/grails-1.3.4/scripts/TestApp.groovy
Environment set to test
    [mkdir] Created dir: /u1/development/workspaces/collabnet/svnedge-1.3.4/console/target/test-reports/html
    [mkdir] Created dir: /u1/development/workspaces/collabnet/svnedge-1.3.4/console/target/test-reports/plain

Starting functional test phase ...

Once the functional tests execution finishes the execution, the test reports are written and can be accessed using a web browser. The following snippet shows the result of running the test case started above, which shows how long it took Grails to execute the 4 test cases defined in theLoginFunctionalTests test suite, the stats of how many tests passed or failed, as well as the location where the test reports are located along with the final result of PASSED or FAILED. Note that the directory “target/test-reports” is relative to the directory “CSVN_DEV” as described above.

Tests Completed in 12654ms ...
-------------------------------------------------------
Tests passed: 4
Tests failed: 0
-------------------------------------------------------
2010-09-28 12:19:11,334 [main] INFO  /csvn  - Destroying Spring FrameworkServlet 'gsp'
2010-09-28 12:19:11,350 [main] INFO  bootstrap.BootStrap  - Releasing resources from the discovery service.
2010-09-28 12:19:11,350 [main] INFO  bootstrap.BootStrap  - Releasing resources from the Operating System service.
2010-09-28 12:19:11,352 [main] INFO  /csvn  - Destroying Spring FrameworkServlet 'grails'
Server stopped
[junitreport] Processing /u1/development/workspaces/collabnet/svnedge-1.3.4/console/target/test-reports/TESTS-TestSuites.xml
                  to /tmp/null1620273079
[junitreport] Loading stylesheet jar:file:/home/mdesales/.ivy2/cache/org.apache.ant/ant-junit/jars/ant-junit-1.7.1.jar
!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl
[junitreport] Transform time: 2339ms
[junitreport] Deleting: /tmp/null1620273079

Tests PASSED - view reports in target/test-reports
Application context shutting down...
Application context shutdown.

Accessing the Test Results Report

Once the execution terminates, you can have access to the test reports. This is where you will find all the answers to the test results, including the detailed information of the entire HTTP payload transmitted between SvnEdge server and the Browser emulator that the Functional Tests use. As shown below, the location of the test cases reports is highlighted as a hyper-link to the index page of the test reports. Clicking on it results on opening the Eclipe’s built-in browser view with the reports.

svnedge-functiona-tests-execution-smaller.png

This report is generated per execution, and therefore, they are deleted before each new execution. In case you need keep information of a test run, copy the contents of the directory “CSVN_DEV/target/test-reports”, as you will find reports in both HTML and XML. The report for each test suite includes the list of each test case run, the status and time of execution. The report includes 3 main output:

  • Properties: system properties used.
  • System.out: the output of the standout output of the process; same output print in the grails output, but more organized.
  • System.err: the output of the standard error of the process.

The most used output is the System.out. Clicking on this hyper-link takes you to the organized output of the traffic, highlighting the HTTP Headers, HTTP Body, redirects, test assersions and test results.

Identifying Test cases report scope

The link to the System.out output is the most important and used throughout the development of the test case, as the output of the execution of each test case is displayed in this area.

svnedge-functional-tests-raw-report-eclipse.png

Each test case has its own test result scope, and you can easily identify the initialization of the execution of a test case by the key “Output from TEST_CASE_NAME”, where “TEST_CASE_NAME” is the name of the method name that defines the test case. For instance, the log for the execution of the test cases for the LoginFunctionalTests includes the following strings:

--Output from testRootLogin--
--Output from testRegularLogin--
--Output from testDotsLogin--
--Output from testFailLogin--

The output of the execution of the HTTP Request Header of a test case is started with “>>>>>” shown as follows:

>>>>>>>>>>>>>>>>>>>> Making request to / using method GET >>>>>>>>>>>>>>>>>>>>
Initializing web request settings for http://localhost:8080/csvn/
Request parameters:
========================================
========================================
Request headers:
========================================
Accept-Language: en
Accept: */*
========================================

On the other hand, the output of the HTTP Response Header of a test case is started with “<<<<<<” as shown below. The HTTP Reponse Header Parameters are output for verification of anything used by the test cases. Note that following access to the “/csvn/” “root” context results in an HTTP Forward to the “Login Page” identified by the context “/csvn” controller “/login/auth” and, therefore, there is not “Content” available.

<<<<<<<<<<<<<<<<<<<< Received response from GET http://localhost:8080/csvn/ <<<<<<<<<<<<<<<<<<<<
Response was a redirect to
  http://localhost:8080/csvn/login/auth;jsessionid=hueqpw5eaq32 <<<<<<<<<<<<<<<<<<<<
Response was 302 'Found', headers:
========================================
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Set-Cookie: JSESSIONID=hueqpw5eaq32;Path=/csvn
Location: http://localhost:8080/csvn/login/auth;jsessionid=hueqpw5eaq32
Content-Length: 0
Server: Jetty(6.1.21)
========================================
Content
========================================

========================================

#Following redirect to http://localhost:8080/csvn/login/auth;jsessionid=hueqpw5eaq32
>>>>>>>>>>>>>>>>>>>> Making request to http://localhost:8080/csvn/login/auth;jsessionid=hueqpw5eaq32
 using method GET >>>>>>>>>>>>>>>>>>>>

If the HTTP Response contains the body payload, it will be output as is in the Content section:

<<<<<<<<<<<<<<<<<<<< Received response from
  GET http://localhost:8080/csvn/login/auth;jsessionid=hueqpw5eaq32 <<<<<<<<<<<<<<<<<<<<
Response was 200 'OK', headers:
========================================
Expires: -1
Cache-Control: no-cache
max-age: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Language: en
Content-Length: 4663
Server: Jetty(6.1.21)
========================================
Content
========================================
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
  <head>

    <title>CollabNet Subversion Edge Login</title>
    <link rel="stylesheet" href="/csvn/css/styles_new.css"
          type="text/css"/>
    <link rel="stylesheet" href="/csvn/css/svnedge.css"
          type="text/css"/>
    <link rel="shortcut icon"
          href="/csvn/images/icons/fav
......
......
......

Whenever a test case fails, the error message is output as follows:

"functionaltestplugin.FunctionalTestException: Expected content to loosely contain [user.new] but it didn't"

Looking deeper in the raw output for the string “Expected content to loosely contain user.new but it didn’t”, you see what HTML output was used for the evaluation of the test case. Sometimes an error case is related to the current UI or to an external test verification. This specific one is related to teamforge integration as the test server did not have a specific user named “user.new” located in the list of users.

Failed: Expected content to loosely contain [user.new] but it didn't
URL: http://cu082.cubit.sp.collab.net:80/sf/sfmain/do/listUsersAdmin

Writing New Functional Test Cases Suite

This section describes how to create test suites using the Functional Tests plugin. In order to maximize code-reuse, we defined a set of Abstract classes that can be used in specific type of tests as shown in the diagram below. Instead of each test case extend the regular class “functionaltestplugin.FunctionalTestCase“, we created a more general abstract class “AbstractSubversionEdgeFunctionalTests” to define general access to configuration artifact, internationalization (i18n) message keys, among others. In addition to the infrastructural utility methods, the main abstract SvnEdge test class contains a set of “often used” method executions such as “protected void login(username, password)”, which is responsible for trying to perform the login to SvnEdge for a given “username” and “password”. The result of the command can then be verified in the body of the implementing class. More details later in this section. First, any test will be implementing one of the test scenario classes: “AdminLoggedInAbstractSvnEdgeFunctionalTests” or “LoggedOutAbstractSvnEdgeFunctionalTests“. However, the test cases for the conversion process needed a specialized Abstract class “AbstractConversionFunctionalTests“, which is of type “AdminLoggedInAbstractSvnEdgeFunctionalTests” because only the admin user can perform the conversion process.

svnedge-functional-tests-abstract-classes.png

As it is shown in the UML Class Diagram above, the AbstractSvnEdgeFunctionalTests extends from the Grails Functional Test class. In this way, it will inherit all the basic method calls for assertions from JUnit and grails. The class is shown in RED because it is a “PROHIBITED” class. That is, no other classes but the GREEN ones should directly extend from the RED class. The fact is that the test cases implementation in Subversion Edge only has 2 different types of tests and, therefore, new test cases should only inherit from “AdminLoggedInAbstractSvnEdgeFunctionalTests” or “LoggedOutAbstractSvnEdgeFunctionalTests“. Similarly, additional functional tests to verify other scenarios from the conversion process has to inherit the behavior of the abstract class “AbstractConversionFunctionalTests“.

Basic Abstract Classes

As described in the previous sections, the two major types of test cases are related to when the Admin user is logged in and when there is no user logged in. That is, tests that require different users to login can use the latter test class to perform the login and navigate through the UI. Before continuing, It is important to note that the Functional Tests implementation are based on JUnit using the 3x methods name conversions. For instance, the methods “protected void setUp()” and “protected void tearDown()” are called before and after running each test case defined in a test class. Furthermore, it is also important to to call the super implementation of each of the methods because of the dependency on the Grails infrastructure. Take a look at the following JavaDocs to have an idea of the basic utility methods implemented on each of them.

Just as a reminder, upon executing the test cases defined in a class, JUnit executes the method “setUp()”. If any failure occurs in this test, Grails will fail not only the first test case, but ALL the test cases defined in the Test Suite. This is related to the fact that the method “setUp()” is executed before the execution of each test case. Once the execution of a given test case is finished, the execution of the method “tearDown()” is performed. Any failure on this method also results in ALL test cases to fail.

The test cases defined in the abstract classes are defined to give the implementing concrete classes the access to all the important features for the test cases. As mentioned earlier, utility methods to access the configuration properties and internationalization (i18n) messages are provided. In addition, convenient test cases for performing assertions are also implemented in the Abstract classes. The next sections will provide in-depth details in the implementation of the test suites.

Concrete Functional Tests Suites Implementation

The simplest implementation of Functional Tests is the LoginFunctionalTests used as an example before. However, executing the scenario to be implemented using the production version is the first recommended step before writing any piece of code. You need to collect information about the scenario to be executed, choose UI elements to use in your test case, etc. For instance, consider the execution of the Login scenario of a user with wrong username. By default, the development and test environments of Subversion Edge will be bootstrapped with different users such as “admin”, “user” and “user.new”. Considering a scenario where the attempt to login with a wrong username called “marcello” is performed as the result is shown in the screenshot below:

svnedge-functional-test-scenario-login-error.jpg

The test case shows that by entering a wrong username and password, an error message is shown as the server responded with a complete and correct page (HTTP Response Code 200), although an error occurred during the execution of the test case. Based on those information, the automated tests can be written in the test suite to verify the possible test cases for the different users in SvnEdge, including the implementation of the wrong input. Note that the implementation of each test case have the procedures to be verified in the super class through the call to a method “testUserLogin” whereas the implementation of the testFailLogin() is the only implementation that is located in the LoginFunctionalTests. Other abstract and concrete test classes are shown in the UML Class diagram below. Note that the YELLOW classes are the concrete classes that extends the functionality from the abstract classes.

svnedge-functional-tests-abstract-and-concret-classes.png

  • LoginFunctionalTests.html: The concret functional tests class suite that verify the login for each of the different usernames, as well as the failure tests.
package com.collabnet.svnedge.console.ui

import com.collabnet.svnedge.LoggedOutAbstractSvnEdgeFunctionalTests;

class LoginFunctionalTests extends LoggedOutAbstractSvnEdgeFunctionalTests {

    @Override
    protected void setUp() {
        super.setUp();
    }

    @Override
    protected void tearDown() {
        super.tearDown();
    }

    void testRootLogin() {
        this.loginAdmin()
    }

    void testRegularLogin() {
        this.loginUser()
    }

    void testDotsLogin() {
        this.loginUserDot()
    }

    void testFailLogin() {
        this.login("marcello", "xyzt")
        assertContentContains getMessage("user.credential.incorrect",
            ["marcello"] as String[])
    }
}

The fact is that the methods “loginAdmin()”, “loginUser()”, etc, are implemented in the AbstractSvnEdgeFunctionalTests to allow code reuse in other test classes, and therefore, the test case “testFailLogin()” uses the basic method “AbstractSvnEdgeFunctionalTests.login(username, password)” for the verification of a user that does not exist. Also, note that the verification of the login scenario is as simple as verifying if the a given String exists in the resulting HTTP Response output. For instance, when attempting to login with a user that does not exist, the error message “Wrong username/password provided for the user “marcello”. This is due to the fact that the String is located in the messages bundle “user.credential.incorrect” and the method “getMessage()” is the helper method implemented in the class AbstractSvnEdgeFunctionalTests.

Another important thing to keep in mind is about code convention. The name of test cases are defined as cameCase, prefixed by the keyword “test”. The name of the test cases can be as long as “AbstractConversionFunctionalTests.html“. The most important point here is that the name of the method must be coherient to the steps being performed. Also, note that Groovy accepts a more relaxed code notation, which makes it easy to read:

        // JAVA method invocation Notation
        this.login("marcello", "xyzt")

        // GROOVY method invocation Notation
        this.login "marcello", "xyzt"

When it comes to the real implementation of a given scenario, you have to constantly refer to the Grails Functional Tests documentation and that’s where you will find your “best friends”. Yes!!! Your best friends! The assert methods that will help you verify the results of the HTTP Response. But first, let’s take a look at the implementation of the basic method that performs “login” and “logout”. As we know from the definition of the abstract classes, each time a method from a class that extends “LoggedOutAbstractSvnEdgeFunctionalTests” is executed, the method setUp() inherited from this class is executed first.

public abstract class LoggedOutAbstractSvnEdgeFunctionalTests extends AbstractSvnEdgeFunctionalTests {

    @Override
    protected void setUp() {
        //The web framework must be initialized.
        super.setUp()

        get('/')
        assertStatus(200)

        if (this.response.contentAsString.contains(
                getMessage("layout.page.login"))) {
            this.logout()
        }
    }

    @Override
    protected void tearDown() {
        //Stop Svn Server in case it is running
        this.stopSvnServer()

        this.logout()

        //The tear down method terminates all the web-related objects, and
        //therefore, must be performed in the end of the operation.
        super.tearDown()
    }
}

Note that the implementation of the concrete classes MUST make a call to the super.setUp() first, so it executes the depending steps. As you can see in the class implementation below, the method setUp() will first make a request to “/”, that is, “http://localhost:8080/csvn/&#8221; since the RESTful method “get()” uses the base URL + the context name “/csvn”. Then, the first assertion is important to verify that the Server is up and running, as well as verifying that the request did not return any error in the UI. Bookmark the RFC2616 and use the HTTP Response Status Codes as required. The default one to verify is “200”, even though the scenario results in an error message as the test case “LoginFunctionalTests.testFailLogin()”. Finally, after verifying if the status code is as expected, the test uses the object “response” to verify if the HTML content contains the string identified by the key “layout.page.login” in the the i18n artifact “CSVN_DEV/grails-app/i18n/messages.properties”. For this case, the method is verifying for the key:

layout.page.login=Login

Following the way JUnit implements the test execution cycle, the method “tearDown()” is executed right after each method “testXYZ()” is executed. In our case, there are a few steps to be verified before terminating the test case. As it might be necessary, the HTTP server might have been started during a test case, and therefore, the method “stopSvnServer()” is called. This is specially placed in the “highest” abstract class because all types of test cases might want to start the HTTP server from the status page. After an HTTP reques to “/” is performed, the verification to the output is performed to and in case is necessary, the method “logout()” is executed as implemented in the abstrac class “AbstractSvnEdgeFunctionalTests“. That is, if the HTML code from the response object contains the string identified by the key “layout.page.logout” (LOGOUT), then click in the link “LOGOUT”. Then, assert if the HTTP response status was equals to 200 and that the content contains the header string “Login” identified by the key “login.page.auth.header”.

    /**
     * Performs the logout by clicking on the link.
     */
    protected void logout() {
        def logout = getMessage("layout.page.logout")
        if (this.response.contentAsString.contains(logout)) {
            click logout
        }
        assertStatus(200)
        assertContentContains(getMessage("login.page.auth.header"))
    }

Similarly, test cases that perform login will essentially fill out the login form and click on the button “Log in”. The basic implementation of the method “login(username, password) is shown below. The HTTP GET Request to the page “/login/auth” is performed followed by the assert of the status code. Then, if the test environment keeps the user Logged In as a result of a failure of any previous test case, the verification if the user is logged in is performed so that the call to the method “logout()”, as shown above, is performed. Finally, when the user s in the front page, the login form is filled out with the correct values. Please refer to the “Grails Functional Tests Documentation” for details on how to fill out and submit form fields, but it should be straightforward. The only detail needed is to capture the name of the form defined in the HTML code. A good helper way is to use Google Chrome or Firefox “Web Developer” plugin to capture the UI element “ids”. Specifically for the form submission, the ID of the form and the “id”s from the form fields are necessary. Then, the label value of the SUBMIT button is necessary, and as shown in the code below, that string is located in the string with the key “layout.page.login”.

svnedge-functional-tests-browser-show-form-properties.png

    protected void login(username, password) {
        get('/login/auth')
        assertStatus(200)

        if (this.response.contentAsString.contains(
                getMessage("layout.page.login"))) {
            this.logout()
        }
        def login = getMessage("layout.page.login")
        form('loginForm') {
            j_username = username
            j_password = password
            click login
        }
        assertStatus(200)
    }

It is extremely important to note here a very hard problem when it comes to “clickable” items in the UI. Since we are using a mix of the Grails GSP tags and some CSS styles from TeamForge, Grails creates the buttons in a different way for Forms and Places without the HTML form entity. Whenever a form was generated by Grails, the Submit button like the “login” one showed above will only respond to the command “click LABEL” inside of the form() closure. On the other hand, the command “click LABEL” will only perform its action when declared outside of the form() closure. Different examples of these GOTCHAS have been found while the Conversion tests were being written.

To summarize the steps to automate manual tests with corresponding Functional Tests, the suggested steps are as follows:

  1. . Perform the test scenario manually and gather necessary information about the User Interface, choosing unique elements that are present in the resulting action. For the case of login, the verification of the string “Logged in as:” is perfomed. For tests exploring failures and error messages, choose to assert about the existence of these error messages.
  2. . Once you are familiar about how the scenario behaves, create the main Test Case Suite by extending from one of the GREEN abstrac classes in the UML Class Diagram shown above. Choose the names related to the component.
  3. . Propose code reuse by implementing new methods in the AbstractSvnEdgeFunctionalTests if necessary, or if other components will use the same implementation. If not, keep the implementation in the test class developed.
  4. . Add JavaDocs to the methods that are going to be inherited or are difficult to understand. Try documenting the method execution before writing the test case as you will understand the scope of the test better. Next section will provide a good understanding on how to write those supporting documentation.

Advanced Functional Tests Techniques

Once you get used to the way to write automated test cases, you should be able to implement complex test cases that involves not only the local Subversion Edge server, but also external servers such as the TeamForge server used during the tests of conversion process. Don’t forget to document the steps in a structured way inside the JavaDocs, as documentation later makes it easy to understand the purpose of the tests.

Note that the JavaDocs of the classes contain a more detailed specification of the execution of the test cases. For example, the sentences starting with “Verify” are related to the assertions necessary to verify the test case, while “Go to” are related to the HTTP Request method “get()”. Each of the sections are identified so that the implementation of the methods setUp(), tearDown(), and the actual method are explicitly written using Groovy. The source code has more detailed implementation of the test cases.

Test Case 1: Successful conversion to TeamForge Mode

   * SetUp
        * Login to SvnEdge
        * Revert to Standalone Mode in case on TeamForge Mode

   * Steps to reproduce
         * Go to the Credentials Form
         * Enter correct credentials and existing CTF URL and try to convert;

   * Expected Results
         * Successful conversion message is shown
         * Login -> Logout as admin
         * Verify that the server is on TeamForge mode;
         * Login to CTF server and verify that the system ID
            from the SvnEdge server is listed on the list of integration servers

   * Tear Down
         * Revert conversion if necessary
         * Logout from the SvnEdge server

The implementation of complex test cases might require verification of different properties of local and external resources. The example of the conversion process was the first challenge of this nature we had to implement. The following code snippet is the implementation of assertions of the conversion as the Expected results. Note that the method custom assertion methods were written to support this implementation (“assertProhibitedAccessToStandaloneModeLinksWorks()” and “assertConversionSucceededOnCtfServer()”.

    /**
     * Verify that the state of the conversion is persisted:
     * <li>The local server shows the TeamForge URL
     * <li>The CTF server shows the link to the server. This can be verified
     * by the current system ID on the list of integration servers.
     */
    protected void assertConversionSucceeded() {
        // Step 1: verify that the conversion is persistent
        this.logout()
        assertStatus 200

        this.loginAdmin()
        assertStatus 200

        assertContentContains(getMessage("status.page.url.teamforge"))
        // verify that the software version is still shown
        assertContentContains(getMessage("status.page.status.version.software"))
        assertContentContains(
            getMessage("status.page.status.version.subversion"))

        get('/server/edit')
        assertStatus 200
        assertContentContains(getMessage("server.page.leftNav.toStandalone"))

        // verify that prohibited links work
        assertProhibitedAccessToStandaloneModeLinksWorks()

        // Step 2: verify that the CTF server DOES list the system ID
        assertConversionSucceededOnCtfServer()
    }

Using the response object

As seen in some of the examples, the assertions are the way to verify that a given expected value exists in the HTTP Response payload received from the Server. However, whenever the test case needs to make a decision based on the contents of the response object, you can use the direct access to the response object. For instance, instead of failing a test that needs to have the user logged out, this code snippet verifies if the user is logged in and then performs the logout procedure. The same logic can be applied in different scenarios such as verifying if the server is started/stopped by verifying the status page button. Similarly, the test can verify if there are any created repositories in the file-system before creating a new test repository.

        if (this.response.contentAsString.contains(getMessage("layout.page.login"))) {
            this.logout()
        }

Dealing with external resources

The nature of Subversion Edge requires the integration with TeamForge, and how about testing the state of both systems in the same test case? Considering the Grails Plugin allows external HTTP requests during tests, why not performing the same steps an Admin would do to verify the state of the server? This was a bit tricky, but works like a charm. As we had designed before, reusing the configuration was the first step to define which remote TeamForge to use during tests. Then, the Test case could take care of automating the ways to generate the URL for the CTF server based on the configuration parameters during the tests of conversion. Here’s the closure in the file “CSVN_DEV/grails-app/conf/Config.groovy” that one can change which TeamForge server to use (svnedge.ctfMaster).

    ctfMaster {
        ssl = false
        domainName = "cu082.cubit.sp.collab.net"
        username = "admin"
        password = "admin"
        port = 80
        systemId = "exsy1002"
    }

Taking a closer look of what we needed, this is related to the assertions for the last expected result “Login to CTF server and verify that the system ID from the SvnEdge server is listed on the list of integration servers”. So, the translation of this sentence into Groovy code originated the method call “AbstractConversionFunctionalTests.assertConversionSucceededOnCtfServer()”, as the steps to perform this assertion are used by all different scenarios. As implemented, the first step requires that the login to TeamForge take the user to the Administration page “List Integrations” using the method ” this.goToCtfListIntegrationsPage()” before verifying if the system ID saved by the conversion process exists in that page. However, observations on how the HTTP Request flow in TeamForge works was necessary to understand the forwards after the user is logged in. After building the necessary parameters in the method “loginToCtfServerIfNecessary()” was implemented with all the needed values from both the Grails Config.groovy and from the environment. As warned before, the clickable elements of forms can differ from Subversion Edge and TeamForm, and therefore, the grails element “click LABEL” was used here outside the form closure. Finally, don’t be tempted to verify strings in TeamForge using i18n as they are different and Subversion Edge does not have direct access to them. Prefer validating steps using form elements or IDs produced by TeamForge as the UI can change on the remote server.

    /**
     * Verifies that the CTF server lists the current ctf server system ID.
     */
    protected void assertConversionSucceededOnCtfServer() {
        // NOTE: NO I18N HERE SINCE TEAMFORGE IS NOT I18N READY
        this.goToCtfListIntegrationsPage()
        assertContentContains(CtfServer.getServer().mySystemId)

        assertContentContains("Site Administration")
        assertContentContains("SCM Integrations")
        def appServerPort = System.getProperty("jetty.port", "8080")
        def csvnHostAndPort = server.hostname + ":" + appServerPort

        // TeamForge removes any double-quotes (") submitted via the SOAP API.
        assertContentContains("This is a CollabNet Subversion Edge server in " +
            "managed mode from ${csvnHostAndPort}.")
    }

    /**
     * Goes to the list of integrations on the CTF server
     */
    private void goToCtfListIntegrationsPage() {
        // Goes to the list integrations page
        // http://cu073.cloud.sp.collab.net/sf/sfmain/do/listSystems
        get(this.makeCtfUrl() + "/sf/sfmain/do/listSystems")
        this.loginToCtfServerIfNecessary()
    }

   /**
    * Makes login to CTF server from a given point that connects to the server.
    * In case the response content DOES NOT contains the string "Logged in as",
    * then make the login. The resulting page is the redirected page requested
    * earlier.
    */
    private void loginToCtfServerIfNecessary() {
        //NOTE: NO I18N HERE SINCE TEAMFORGE IS NOT I18N READY
        if (!this.response.contentAsString.contains("Logged in as")) {
            assertStatus 200
            def ctfUsername = config.svnedge.ctfMaster.username
            def ctfPassword = config.svnedge.ctfMaster.password
            form("login") {
                username = ctfUsername
                password = ctfPassword
            }
            // the button is a link instead of a form button. Use it outside
            // the form closure.
            click "Log In"
            assertStatus 200
        }
    }

Test Case Suites Needed

A few test cases have been written for specific functionalities of the application. However, here’s some of the test cases that can be developed.

* User Functional Tests
- Create User of each type
  - Login/Logout
  - Verify access to prohibited URLs
  - Access SVN and ViewVC Pages
- List Users
- Delete User
- Change User password
  - Logout and login with the new password.
  - Access SVN and ViewVC pages with new password
- View Self page
- Try changing the server settings, accessing other admin sections

* Repos Functional Tests
- Create Repo
- Discover Repos
- List Repos
- Edit Access Rules
  - Login with users without access to specific repos without access

* Statistics Functional Tests
- Access the pages for statistics

* Administration Functional Tests
- Changing server settings as Admin
- Changing the server Authentication settings
  - Login / Logout and verify changes.
  - Restart server after changing settings

* Server Logs Functional Tests
- Change log level
- View log Files
- View non-existing file
- View existing file
- View log files

* Packages Update Functional Tests
- Update the software packages
- Convert the server and try to update the server to new version

If you have any questions regarding the Functional Tests specification, please don’t hesitate to send an email to dev-svnedge@ctf.open.collab.net.

Marcello de Sales – Software Engineer – CollabNet, Inc.

Advertisements
Categories: java, Subversion
  1. John
    November 5, 2010 at 5:56 am

    Wow i love your blog its awesome nice colors you must have did hard work on your blog. Keep up the good work. Thanks
    toolplace.wordpress.com

  2. November 22, 2010 at 12:48 am

    I’m happy you like it!!! 🙂

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: