Google Chrome QA and performance tests through units tests of YUI and Scriptaculous

[Update: unit tests on cookies don't work from file:// type of urls. It's intentional: see this page on the Chromium group. It's the way I run them initially. No more issue when I run them from a web server. That removes 50 from the error counter and already raises quality level to almost 97% globally and 99.5% with YUI]

Google Chrome has been released 2 weeks ago: it's still beta. But, according to the comics issues for the launch, a huge emphasis was put on testing for maximum quality at launch: rendering of milions of pages is validated for each build. At page 11,a progressive improvement from 23% (early days) to 99% (launch date) in the Webkit layout testing suite is stated as a proof of quality.

On the other side, I didn't see until today any test against the compatibility of major Javascript framework (Dojo, jQuery, Prototype, script.aculo.us, YUI, etc...).

I am used to working with 2 of them: YUI (currently at v2.5.2) and Prototype/script.aculo.us (currently at v1.8.1). Both of them are delivered with a series of unit tests. I decided to run Google Chrome against those suits to see what would happen.

The results are the followings:

with Script.aculo.us 1.8.1:

with Yahoo YUI 2.5.2:

Various conclusions could be drawn from that point knowing that no adaptation was yet made in the 2 frameworks to cope specifically with "specialties" of Chrome:

• Chrome fails on 74 tests out of 700. Success rate is already above 89%
• More careful analysis proves that 50 of those failures come from a single area: cookie handling in YUI. I will now analyze source code of YUI cookie.js to get all details: but it's probably just a few lines of codes to modify to pass those tests. When that will be done success rate will jump already above 96%
• globally, it's very reassuring to see that issues are confined to a very limited number of areas and not spread all over the place.

With such a starting point, I guess that it won't take long before Google Chrome gets to the level of Firefox which passes 698 of the 700 tests .

We have since this launch heard quite a lot about the V8 javascript engine included in Chrome and its huge improvements over other competing browsers. Those unit tests heavily focused on javascript but involving rendering at the same time (through tests like Editor, Animation, etc..) represent a good pratical benchmark of coumound performances of Chrome.

Using the bundled suite of 114 tests named named yuitest.html and part of YUI, we get the following results:

• 10s for Firefox 2.0.0.16
• 6s for IE 7.0.57.11C0
  
• 2s for Google Chrome 0.2.149.29

On that side, the lead is already clearly by Google Chrome!

As a conclusion, I would state that developpers using such frameworks [who have as a major goal to insulate them from the specific points current of leading browsers: IE, FF, Safari, etc...) won't have to wait long before those frameworks allow their applications to support Google Chrome in 100% effortless manner from a javascript/ajax standpoint.

Source: blog Media & Tech (par didier durand)