An incomplete overview of webtesting tools

Show of Hands

Weapon of Choice

Weapon of Choice

  • Headless
  • Browser
  • Inbetween
  • Clouded

Headless

Simulates a browser in memory without actual rendering

  • envjs (Rhino)
  • HtmlUnit (Java)
  • Zombie.js (Node)

Headless

Pro:

  • Fastest
  • Easy to set up, barely any infrastructure needed
  • Can run local as part of build (before commit)

Headless

Contra:

  • Only a simulation
  • Will fail to show actual browser problems
  • Will show false negatives due to implementation limits
  • No or incomplete JS runtimes

Browser

An actual browser, remote controlled

  • TestSwarm
  • Selenium
  • Watir
  • Windmill

Browser

Pro:

  • No simulation, only real results
  • Will highlight actual cross-browser issues
  • Good (Selenium) to everything (TestSwarm) browser coverage
  • Often includes a recorder

Browser

Contra:

  • Slowest
  • Lots of infrastructure required with multiple OSs/platforms for good coverage
  • Recorder results rarely produce maintainable code

Inbetween

A remote controlled browser engine, without the chrome.

  • Watij (Java/JRuby)
  • PhantomJS (QT)

Inbetween

Pro:

  • Actual browser engines
  • No simulation
  • Fast enough
  • Can run silent, in developer build
  • PhantomJS easy to get running

Inbetween

Contra:

  • Can be problematic to get running (Watij not working on current Debian)
  • Only browser engine, so may lack features (local storage, geolocation, history.pushState)
  • Watij limited to XulRunner for cross-platform, IE on Windows, Safari on OSX
  • PhantomJS relying on heavy dependency: QT

Clouded

Browser testing as a service.

  • http://saucelabs.com/
  • http://browsermob.com/performance-testing
  • http://www.cloudtesting.com/

Clouded

Pro/Contra:

  • In the cloud
  • fixed monthly cost
  • no administration
  • no direct access
  • YMMV

Zombie vs Phantom Showdown

Zombie.js

var zombie = require("zombie");
var assert = require("assert");

zombie.visit("http://google.de/", function (err, browser, status) {
  browser.
    fill("q", "pizza").
    pressButton("btnG", function(err, browser, status) {
      assert.equal(browser.text("title"), "pizza - Google Suche");
    });
});

Zombie.js result

[...]
ck: [Function]
   , jsrt_kill: 1
   , y: { first: [] }
   , x: [Function]
   , dlj: [Function]
   , dstr: []
   , rein: []
   }
, gbar: { qs: [Function], tg: [Function] }
, _listeners: { load: { false: [Object] } }
}

 threw error

{ message: 'Uncaught, unspecified \'error\' event.'
, stack: [Getter/Setter]
}

 handling event

{ _eventType: 'HTMLEvents'
, _type: 'error'
, _bubbles: true
, _cancelable: false
[...]

PhantomJS

if (phantom.state.length === 0) {
    phantom.state = "loading";
    phantom.open("http://google.de/");
} else if (phantom.state == "loading") {
    phantom.state = "submitted";
    document.getElementsByName("q")[0].value = "pizza";
    document.getElementsByName("btnG")[0].click();
} else if (phantom.state == "submitted") {
    console.log("result: " + document.title);
    phantom.exit();
}

PhantomJS result

result: pizza - Google-Suche