Friday, March 20, 2009

Testing Your WebApp

I've been looking into the various ways to test web applications, including performance, acceptance tests, and cross-browser compatibility.  In my travels over the past week-and-a-half, I have decided there are three levels to this:
  1. Testing the design of your site.  Does it look similar enough in all supported browsers?
  2. Testing the function of your site.  Does all the Javascript work?  Is everything that needs to render rendering?
  3. Testing the reliability of your site.  How fast is it?  How many users can it handle?  For how long?
Design Testing

A good first step is to validate with the W3C tools and try to stay 100% standards compliant. But you're doing that already, I'm sure. : )

Adobe has a neat "Meer Meer" application that's coming... sometime. There is also SuperPreview.  Both of these allow cross-browser design using nothing more than whatever web-browser you have installed: they call the real browsers on the server-side, take snapshots, and serve you the images.  The coolest feature here is that you can overlay two browser's snapshots and have the differences jump out at you.  ...Both of these are unreleased, but just around the corner.  Very cool.

If snapshots are enough, you can get these cheap: $50 a month for Litmus.  You can hook these tests into your testing suite. CrossBrowserTesting has a demo video here. Nothing earth-shattering: just note that you have to do these "tests" by hand.

IBM says: "For instance, for a monthly fee, gives you access to a broader range of platforms and browsers—including some mobile platforms—and these pages are dynamic, allowing you to test Ajax apps and other JavaScript-based pages. Some services, like Litmusapp, can integrate with your testing and development workflows, offering versioning support and ways to publish test results to external clients."
Interestingly, if you want to do these things by hand, there are free Microsoft images of common Windows versions/Browser combos for use by developers.  The bummer is that you have to renew them every so often (and only supported OS/browsers are represented), but it's a great way to test your app without spending too much money on setup.

A decent rundown of the "images of site" tools is here; (there were other overviews, not as good) we should see what the JavaScript-capable options are really capable of. If you need older versions, you'll have to install those yourself, but old IE versions are still available on the web (not from Microsoft, however).

Functional (Acceptance) Testing

For acceptance testing your Javascript, there are a few choices in the world of headless (ie: much faster) browsers.  HtmlUnit seems a usable Java "mock" browser. There's a JRuby wrapper for it called Celerity, which could get me started faster. Similar is jsTest, also in Java. Jiffy struck me as overly complex, and hard-wired to Oracle for measures. Joe Dog Siege looks like a usable, mostly command-line driven load tester.

An interesting but probably less-useful alternative is to just check the JS for problems using crosscheck, but as of now it doesn't seem to have usable documentation, so...

If you don't mind (or have a need to be) using real browsers, Selenium is the standard right now for writing acceptance tests, and it's really easy to set up.  Whether it adds overhead is an interesting question; I would assume yes based on some comments from other tool's sites. That said, A) You don't want to mix your acceptance tests with your performance tests, and B) it seems to be a standard right now, and I do recommend using it because of that.  There are some alternatives, though.  For example, JsUnit is a way to do the tests in native JavaScript, which is quite possibly useful.

Performance Testing

"Performance testing" (actually lots of sub-categories; read on) is a rather large arena, and I found this stuff quite interesting.  ; )  I found the best introduction to the topic to be this neat video from Google, which gives some definitions:
  • performance testing - how fast will it respond under load X? You need to test things at least long enough to reach a steady state. Start up slowly: don't just throw 100 users at it, build it up over a warm-up period. The machines under load periodically push performance dat (see blow) to monitoring servers, which analyze and store the data.  These monitoring servers are also monitoring themselves (so you know what the effect of storing the data is). To the extent possible, remove the network: run everything in its own little environment.
  • stress testing - at what point does the system break under load (and does it do so gracefully)?
  • load (or liability) testing - given a high load (80% of max) for an extended period of time (Google does 1 month, 72 hours is a minimum recommendation), how does it perform?  Often there are cron-related or system-level tasks (ex: log rotation) that affect performance only at certain times, so this needs to be examined.
  • benchmark - a simplified measurement that is repeatable, representing a typical customer's activity on the system.
  • profiling - When you change some aspect of the system (memory, load balancer, etc) and measure the differences in benchmark performance.
  • availability testing - when a system fails, how long does it take to come back up? Google operates under the assumption that their equipment sucks, so they will solve that problem in software.
Her reasons for testing are 1) to make honest claims to the client backed by data, 2) test hardware changes, and 3) developers can see the effect of their code over time.  NOT for functional testing.

Some other tips included: run every test at least 5 times before you measure it: inconsistency must be accounted for.  She recommends a single five-minute test run "on demand", when a developer requests it, because you get *some* information from it, but nightlies are still required.

She recommends JMeter to measure loads (but it doesn't handle JavaScript) (There is a demo of how to get started, too), or Grinder (for "APIs").

Note that you should track not only the QPS, response times, etc, but also the CPU load, memory, versions of every piece of software on the machine, and disk i/o.

Interestingly, she says that Linux is typically performant as long as you don't exceed any single resource's 80% mark.

You can buy load testing.  Prices on LoadStorm, for example, range from $30 to $200 a month. LoadRunner is another alternative (software that you use), by HP, which appears to be rather expensive... (Well, when they don't list a price, one must assume...)

BrowserStorm looks reasonable for a paid service, and has an explanation of its features in a blatant plug on Ajaxian.

Hope this helps.

Monday, March 2, 2009

AIM Chatroom Death

...So we EOL developers hang out in an AOL chatroom.  Last week, I got logged out of it, and couldn't get back in.  ...Not sure why.  I hoped the problem would just go away, but it never resolved itself.  The error I was getting was:

Couldn't join chatroom "".

An AIM service error occurred.

Error: Serv:RequestTimeout

Useless message!  But eventually I found a page describing the problem, with the suggestion to change the AIM server port from 5190 to 443.

That worked.