The Belgian Usability Awards 2010

For the Belgian Usability Awards 2010 are the following sites nominated:

1. www.uitinvlaamsbrabant.be
2. www.kdg.be
3. www.newpharma.be
4. www.mobistar.be
5. www.toerismevlaamsbrabant.be
6. www.agsob.be
7. www.middelkerke.be
8. www.kriskras.be
9. www.connections.be
10. www.sibelga.be

The scores according to the Dutch Web Guidelines are:

1. www.uitinvlaamsbrabant.be 38 points;
2. www.kdg.be 37 points;
3. www.newpharma.be 35 points;
4. www.mobistar.be 34 points;
5. www.toerismevlaamsbrabant.be 34 points;
6. www.agsob.be 40 points;
7. www.middelkerke.be 31 points;
8. www.kriskras.be 37 points;
9. www.connections.be 29 points;
10. www.sibelga.be/nl 37 points.

None of these sites got the same score as www.helenahoeve.nl (41 points).


The validation results of these sites are:

1. www.uitinvlaamsbrabant.be Not valid, 6 errors;
2. www.kdg.be Not valid, 15 errors, 2 warnings;
3. www.newpharma.be Not valid, 19 errors;
4. www.mobistar.be Not valid, 1 error;
5. www.toerismevlaamsbrabant.be Not valid, 14 errors, 21 warnings;
6. www.agsob.be Not valid, 2 errors;
7. www.middelkerke.be Not valid, 81 errors, 3 warnings;
8. www.kriskras.be Not valid, 27 errors, 2 warnings;
9. www.connections.be Not valid, 32 errors, 19 warnings;
10. www.sibelga.be/nl Not valid, 1 error.

Unlike the Helenahoeve none of the candidates validate.

The loadtime of these sites are:

1. www.uitinvlaamsbrabant.be 7.9 seconds;
2. www.kdg.be 10.7 seconds;
3. www.newpharma.be 0.5 seconds;
4. www.mobistar.be 0.6 seconds;
5. www.toerismevlaamsbrabant.be 7 seconds;
6. www.agsob.be 10.4 seconds;
7. www.middelkerke.be 2.2 seconds;
8. www.kriskras.be 5 seconds;
9. www.connections.be 4.7 seconds;
10. www.sibelga.be/nl 5.3 seconds;

How liguid are the designs (when does the horizontal scrollbar appear on a 1024x768 screen):

1. www.uitinvlaamsbrabant.be 1 x "CTRL +";
2. www.kdg.be 1 x "CTRL +";
3. www.newpharma.be 1 x "CTRL +";
4. www.mobistar.be 0 x "CTRL +"!;
5. www.toerismevlaamsbrabant.be 0 x "CTRL +"!;
6. www.agsob.be 1 x "CTRL +";
7. www.middelkerke.be 0 x "CTRL +"!;
8. www.kriskras.be No horizontal scroll at all!;
9. www.connections.be 1 x "CTRL +";
10. www.sibelga.be 1 x "CTRL +";

Is it easy to contact the organisation. Is there a real e-mail address or a contact form.

1. www.uitinvlaamsbrabant.be  no e-mail address, there is a contact form;
2. www.kdg.be  e-mail address, no contact form;
3. www.newpharma.be  no e-mail address, there is a contact form;
4. www.mobistar.be  no e-mail address, there is a contact form wizard;
5. www.toerismevlaamsbrabant.be  e-mail address and contact form;
6. www.agsob.be  e-mail address and contact form;
7. www.middelkerke.be  e-mail address, no contact form;
8. www.kriskras.be  e-mail address, no contact form;
9. www.connections.be e-mail address, but no link to the promised contact form;
10. www.sibelga.be  no e-mail address, there is a contact form;

So at the end I think that non of these sites should win the usability award seen from a technical perspective. When I browse the sites I think the winner will be www.uitinvlaamsbrabant.be

The usability test

Okay, what to test when you want to test the usability? The ease of use? Okay, lets try.

www.bol.com
Lets buy a book ("priemgetallen").

1. Search for priemgetallen
   The first result is the right one
2. Click on "In winkelwagentje"
3. Click on "Bestellen"
   Login or create an account
4. Create an account
5. Fill in the form
   First error made, filled in name instead of given name
   Got a question about the present service allthough I use the same address
6. Click on continu
   Have to give my e-mail address and a password
   Very bad, I have to un-tick that I want a newsletter
   Very bad, I have to type my e-mail address twice
7. Click on check the given information
8. Click on Pay (and stopped)

It is easy, but some things could be improved. Good is that the postal code and the house number was enough.

www.ditzo.nl
Lets buy an insurrance for my car

1. Given the identification of the car and pressed "Bereken je premie"
2. Given all information, it is clear where you are in the process, however I need to give my email address twice and I have to wait to long to go to the next step
3. Process stopped at the moment I had to give the "meldcode" I was too lazy to search for it

www.eduhub.nl
Lets book a simple course


Selected a course form the homepage, Apple's OSX, selected the first course and then ... did not know what to do next. Process stopped.


www.hotels.nl
Try to book a hotel.


I have to enter the dates first before I can do what I want, I have to search again for the wanted hotel. 
However after found the hotel I want for the price I want I only have to enter my name and e-mail address (twice!). Easy. Process stopped to prevent a reservation.


www.nos.nl
I want to see the weather as mentioned in the journal of 20.00 o'clock. 


Found in two click the weather report of the 18.00 o'clock journal. Great.


www.helenahoeve.nl
Try to camp with my own caravan in august.

1. Look at availability
2. Created a reservation
3. Checking if the reservation is correct. Process stopped

Easy.

Best user experience so far.

1. hotels.nl
2. nos.nl
3. helenahoeve.nl - Much more information is necessary for a reservation than hotels.nl
4. ditzo.nl
5. eduhub.nl

Overall score


In my humble opinion hotels.nl should be the winner of the usability award 2010.

The Design test

Testing the design is subjective, however there are some methods which helps.

The small screen test
The first one, is the small screen test. This test is easy to do and I will tell you the result now.

From the following sites, only the Helenahoeve has a seperate design for small screens.

1. www.bol.com
2. www.ditzo.nl
3. www.eduhub.nl
4. www.hotels.nl
5. www.nos.nl
6. www.helenahoeve.nl

The grayscale test

How good are the designs when you don't see colors at all?

In my opinion, bol.com, ditzo.nl and eduhub.nl are too busy. 

The blur test

Almost the same test, however now we use color, but there was a gaussian blur of 5 pixels. Lets see what it looks like.

Now only from bol.com, eduhub.nl and nos.nl it is clear which sitename is involved. The helenahoeve.nl and nos.nl seems to have the best readable test.

What else could I say about the designs. Personally I prefer the design of nos,nl and helenahoeve.nl

commandshift3.com has too few battles for these sites. I am curious about the opinion of the Dutch web users (so I have to wait for results of the Usability Award 2010 for this one).

  

Content test

This is the most difficult test. For each site I have to know what I expect. Okay, lets give it a try.

www.bol.com
Bol.com is working hard at several disiplines. Second hand books, toys, you name it and they sell it.

What information do they have about the new Nintendo DS XL or alternatively the Nintendo DSi? Founded very quickly, however it was not directly clear to me that the XL also have two camera's. After a search on Google I found that fact.

Do they have the book titled "help mijn dochter is hoogbegaafd". Yes they have it. However the information about the book is poor. More information is available via a Google search, and you even will find the book in Google books.

On a personal scale of 5 stars, I will give bol.com 3 stars (remember this test is subjective).

www.ditzo.nl

Ditzo is about insurrance. I want to know what happens if I buy a new car and how much I have to pay after an accident.

No information found about the insurrance when switching cars (sold the old one and bought a new one). No information found about the monthly costs after an accident.

On a personal scale of 5 stars, ditzo.nl got zero stars.

www.eduhub.nl

Eduhub is about training. It is a very advanced startpage. Lets see what the can tell me about sculpture ("beeldhouwen") and "electro installateur".

Found a 1 day Art event for "beeldhouwen". That's not what I am looking for. Searching for "electro installateur" gave zero results, however searching for "installateur" gave 10 hits. The second hit contained the information I was looking for.

Eduhub got 3 stars (out of 5). It could have been 4 if they have usefull information about "beeldhouwen".

www.hotels.nl

I rarely sleep in a hotel. Too expensive. But hey, lets give it a try. A hotel for a romantic weekend near the sea, and a F1 hotel or something like that near Paris.

Wow, found very quickly a hotel for a romatic weekend I will choose Cape Helius. And now up to the sun. The first day we will make it to Paris (plus minus 50 kilometer). Hmm, something went wrong is a strange message when there are no search results. Ok found that hotels.nl is for hotels in the Benelux.

I will give 4 stars, out of 5.

www.nos.nl

At nos.nl it is all about news.

Recently it was not possible to fly. How many people still need their flight? Not found.
Sony stopped with the 3.5 inch disc. Is that a newsitem? Yes, great article.

I don't use nos.nl often, I prefer nu.nl, however the content seems ok. 3 stars out of 5 for the content.

www.helenahoeve.nl

I build this site, but hey lets fake that I knew nothing. What would I like to find on this site?

Where is the site about?
Where are the shops?

Found directly where the site is about. No useful information found about "winkels" or "boodschappen", however the information about "Sluis" is ok.

I give 3 out of 5 stars.

So the ranking is:

1. hotels.nl with 4 stars;
2. bol.com, eduhub.nl, nos.nl and helenahoeve.nl with 3 stars;
3. ditzo.nl with zero stars.

Navigation test

What should I test when I want to test navigation?

1. I want to go to the homepage within one click;
2. I want to browse to the contact page and the privacy page;
3. I want to go to the sitemap within one click;
4. I want to use the backbutton;
5. I want to find a specific item.

The test sites are the sites nominated for the usability award 2010 and of course www.helenahoeve.nl

To the homepage with one click

1. www.bol.com Yes, one click on the logo
2. www.ditzo.nl Yes, one click on the home tab
3. www.eduhub.nl Yes, one click on the logo
4. www.hotels.nl Yes, one click on the logo
5. www.nos.nl Yes, one click on the logo
6. www.helenahoeve.nl Yes, one click on the logo

To the contact page

1. www.bol.com One click, "klantenservice" at the right upper corner
2. www.ditzo.nl Telephone number and information form at homepage
3. www.eduhub.nl Two clicks "Over Eduhub", "Contact"
4. www.hotels.nl One click, "contact" at the right upper corner
5. www.nos.nl One click, "contact" at the right lower corner
6. www.helenahoeve.nl One click, "contact" at the right upper corner 

To the privacy page

1. www.bol.com One click "privacy policy" in the footer
2. www.ditzo.nl One click "privacy" in the footer
   
3. www.eduhub.nl One click "voorwaarden & privacy" in the footer
   
4. www.hotels.nl One click, "privacy" at the right lower corner
5. www.nos.nl Not found
6. www.helenahoeve.nl One click, "privacy"  at the right lower corner

To the sitemap

1. www.bol.com Not found
2. www.ditzo.nl One click "sitemap" in the footer
3. www.eduhub.nl Not found, however the homepage gives a great overview
4. www.hotels.nl Not found
5. www.nos.nl Not found
6. www.helenahoeve.nl One click, "privacy"  at the right lower corner

Backbutton working at the whole site? (as far as I know)

1. www.bol.com Yes
2. www.ditzo.nl Yes
3. www.eduhub.nl Yes
4. www.hotels.nl Yes
5. www.nos.nl Yes
6. www.helenahoeve.nl Yes

Specific search

1. www.bol.com Search for book "priemgetallen", found directly 
2. www.ditzo.nl Search for "wegenwachthulp", "pechhulp", visible at the homepage
3. www.eduhub.nl Search for "diëtiste", not found, "dietiste", found directly (6 results)
4. www.hotels.nl Search for a hotel in Leiden for tonight, found directly 8 hotels available
5. www.nos.nl Search for "kandidaten groenlinks", not found
6. www.helenahoeve.nl Search for "caravan", found directly

Conclusion so far, all sites are good enough. There is not an outstanding site.

Normal, large, larger and larger

Just another test. How many times is it possible to increase the font-size before there appears a horizontal scrolbar. This time the test is in Firefox on a 1024x768 pixel monitor.

1. bol.com 1x 'CTRL +' gives a horizontal scrollbar
2. funda.nl 4x 'CTRL +' gives a horizontal scrollbar
3. helenahoeve.nl After 8x 'CTRL +' (the maximum fontsize), there is still no horizontal scrollbar
4. nu.nl 5x 'CTRL +' gives a horizontal scrollbar
5. philips.nl 1x 'CTRL +' gives a horizontal scrollbar
6. postbus51.nl 1x 'CTRL +' gives a horizontal scrollbar
7. rabobank.nl 1x 'CTRL +' gives a horizontal scrollbar
8. vrom.nl 1x 'CTRL +' gives a horizontal scrollbar
9. wehkamp.nl 1x 'CTRL +' gives a horizontal scrollbar
10. wnf.nl 2x 'CTRL +' gives a horizontal scrollbar

This gives the following ranking for allowed font-size:

Only helenahoeve.nl did not get an horizontal scrollbar! That's great.

Lipperhey scores

There is a great tool which probably make all other tests unnecessary. It is called Lipperhey. The results are:

1. bol.com Technical 7.2, SEO Score 73%, Populatity 10.0, Overall score 8.2
2. funda.nl Technical 6.4, SEO Score 75%, Populatity 10.0, Overall score 8.0
3. helenahoeve.nl Technical 8.6, SEO Score 76%, Populatity 7.0, Overall score 7.7
4. nu.nl Technical 5.7, SEO Score 48%, Populatity 10.0, Overall score 6.8
5. philips.nl Technical 6.0, SEO Score 59%, Populatity 9.0, Overall score 7.0
6. postbus51.nl Technical 8.0, SEO Score 74%, Populatity 9.0, Overall score 8.1
7. rabobank.nl Technical 7.3, SEO Score 76%, Populatity 10.0, Overall score 8.3
8. vrom.nl Technical 7.2, SEO Score 55%, Populatity 10.0, Overall score 7.6
9. wehkamp.nl Technical 4.0, SEO Score 0%, Populatity 3.0, Overall score 2.3
10. wnf.nl Technical 7.5, SEO Score 67%, Populatity 9.0, Overall score 7.7

The most important thing to do seems to be getting more popular ;-)

Time to work on the backlinks. But first the results as a graph:

SERP Search results

SERP is standing for Search Engine Results Page, searching for the websites gives these results on Google:

1. bol Number 1
2. funda Number 1
3. helenahoeve Number 1
4. nu Number 1
5. philips Number 1
6. postbus51 Number 1
7. rabobank Number 1
8. vrom Number 1
9. wehkamp Number 1
10. wnf Number 1

The same for Bing:

1. bol Number 1
2. funda Number 1
3. helenahoeve Number 3 and 4, number 1 is helenahoeve.hyves.nl
4. nu Number 1
5. philips Number 1
6. postbus51 Number 1
7. rabobank Number 1
8. vrom Number 1
9. wehkamp Number 1
10. wnf Number 1

Last but not least. Yahoo!

1. bol Number 1
2. funda Number 1
3. helenahoeve Not on the first page
4. nu Not on the first page
5. philips Not on the first page, however philips.com (which redirects to philips.nl) is at number 1
6. postbus51 Number 1
7. rabobank Number 5, however rabobank.com is at number 1
8. vrom Number 1
9. wehkamp Number 1
10. wnf Number 1

Hmmm, I think I need to contact Bing and Yahoo about these results.

Pageranks

How visible are the test websites. Time to look at the pageranks.

1. bol.com Pagerank: 6
2. funda.nl Pagerank: 7
3. helenahoeve.nl Pagerank: 3
4. nu.nl Pagerank: 7
5. philips.nl Pagerank: 7
6. postbus51.nl Pagerank: 8
7. rabobank.nl Pagerank: 7
8. vrom.nl Pagerank: 8
9. wehkamp.nl Pagerank: 6
10. wnf.nl Pagerank: 6

See also this graph:

Hot, not or useless

CommandShift3 is a website where people compare two sites a time. Lets see how hot the test websites are...

1. bol.com Preparing for battle
2. funda.nl Preparing for battle
3. helenahoeve.nl 14% won 3 of 21 battles
4. nu.nl 20% won 15 of 75 battles
5. philips.nl Preparing for battle
6. postbus51.nl Preparing for battle
7. rabobank.nl Preparing for battle
8. vrom.nl Preparing for battle
9. wehkamp.nl 20% won 18 of 88 battles
10. wnf.nl Preparing for battle

Hmmm there are not much battles, however the ranking is:

1. nu.nl, wehkamp.nl
2. helenahoeve.nl

It is a pity, I did not learn much from this test. Luckily there are more websites to review websites. Lets take a look at sitecritic.net...

1. bol.com Search results: None found
2. funda.nl Search results: None found
3. helenahoeve.nl Search results: None found
4. nu.nl Search results: None found
5. philips.nl Search results: None found
6. postbus51.nl Search results: None found
7. rabobank.nl Search results: None found
8. vrom.nl Search results: None found
9. wehkamp.nl Search results: None found
10. wnf.nl Search results: None found

Funny, but useless results.

May be we should try Website Judge...

1. bol.com 3 reviews, total score: 9
2. funda.nl 0 reviews
3. helenahoeve.nl 0 reviews
4. nu.nl 4 reviews, total score: 8
5. philips.nl 0 reviews
6. postbus51.nl 0 reviews
7. rabobank.nl 1 review, total score: 6
8. vrom.nl 0 reviews
9. wehkamp.nl 0 reviews
10. wnf.nl 0 reviews

If you want to take this serious, the ranking is:

1. bol.com
2. nu.nl
3. rabobank.nl

As far as I know now, these judge websites are useless.

The need for speed

The speed of a website is very important. According to Akamai nowadays a website should have a respons of  max 2 seconds. According to Jacob Nielsen a good site have a respons of max 1 second.

Yahoo has a great tool for Firebug users. It is called YSlow and gives an indication of the speed of your website. Testing the websites with YSlow give the following results:

1. bol.com Grade E, Overall performance score 58
2. funda.nl Grade B, Overall performance score 80
3. helenahoeve.nl Grade A, Overall performance score 92
4. mobiel.nu.nl Grade C, Overall performance score 78
5. philips.nl Grade C, Overall performance score 70
6. postbus51.nl Grade D, Overall performance score 69
7. rabobank.nl Grade C, Overall performance score 70
8. vrom.nl Grade D, Overall performance score 67
9. m.wehkamp.nl Grade C, Overall performance score 77
10. wnf.nl Grade E, Overall performance score 56

This gives the following ranking:

Testing the websites with tools.pingdom.com gives these results:

1. bol.com 160 objects, 9.1 seconds
2. funda.nl 13 objects, 1.2 seconds
3. helenahoeve.nl 19 objects, 0.7 seconds
4. mobiel.nu.nl 20 objects, 1.1 seconds
5. philips.nl 4 objects, 1.7 seconds
6. postbus51.nl 100 objects, 2.7 seconds
7. rabobank.nl 78 objects, 3.1 seconds
8. vrom.nl 38 objects, 1.6 seconds
9. m.wehkamp.nl 46 objects, 1.7 seconds
10. wnf.nl 285 objects, 6.3 seconds

The ranking order is the following:

helenahoeve.nl is the only site which speed is below one second!

The ranking is:

1. helenahoeve.nl
2. funda.nl, mobiel.nu.nl, philips.nl, vrom.nl, m.wehkamp.nl
3. bol.com, postbus51.nl, rabobank.nl, wnf.nl

There is printing and printing

The last test in the quality test serie is the printer test. Initial I want to test this on a MacBook only. This is because I thought that Apple understood the use of PDF. With a MacBook you can after the print commando directly save the result a a PDF instead of a physical print. The results were a little bit strange. I seems that Flash components of the webpage will not be printed.

So I decided to create also PDF's with PDFCreator on a XP machine. For the third time during these tests I am flabbergasted. The results are not always the same. Huh, what is happening here?

I do not understand this, however I can judge the results:

1. bol.com Inconsistent
2. funda.nl Inconsistent
3. helenahoeve.nl Consistent, specific print layout available
4. nu.nl Consistent
5. philips.nl Inconsistent
6. postbus51.nl Consistent
7. rabobank.nl Inconsistent
8. vrom.nl Inconsistent
9. wehkamp.nl Inconsistent
10. wnf.nl Inconsistent

The helenahoeve is the only website with a specific print lay-out!

See the screenshots (left Apple's Save as PDF, right Windows XP PDFCreator):

Watching television

One of the last tests is viewing the sites on the television by taking a look at the test websites with the Opera Browser for the Wii.

The test results are:

1. bol.com Horizontal scroll needed
2. funda.nl Great
3. helenahoeve.nl Great, seperate design for the television
4. nu.nl Great
5. philips.nl Horizontal scroll needed
6. postbus51.nl Horizontal scroll needed
7. rabobank.nl Horizontal scroll needed
8. vrom.nl Horizontal scroll needed
9. wehkamp.nl Bad design, almost empty page
10. wnf.nl Bad design, almost empty page

Ranking the sites gives the following list:

1. funda.nl, helenahoeve.nl, nu.nl
2. bol.com, philips.nl, postbus51.nl, rabobank.nl, vrom.nl
3. wehkamp.nl, wnf.nl

See also these screenshots:

The HTC TyTN 2 test

Time to test the sites with a real mobile phone. The display of the HTC TyTN 2 is a 2.8 inch, 240 X 320 QVGA TFT-LCD display with adjustable angle and backlight. Testing the sites with this phone gives the following results:

The test results are:

1. bol.com Horizontal scroll needed
2. funda.nl Great, seperate design for mobile devices
3. helenahoeve.nl Great, seperate design for mobile devices
4. nu.nl Great, seperate design for mobile devices (mobiel.nu.nl)
5. philips.nl Very bad, no idea what this site is about
6. postbus51.nl Great, seperate design for mobile devices
7. rabobank.nl Bad design, horizontal scroll needed
8. vrom.nl Very bad, no idea what this site is about, horizontal scroll needed
9. wehkamp.nl Great, seperate design for mobile devices (m.wehkamp.nl)
10. wnf.nl Bad design, however no horizontal scroll needed

Ranking the sites gives the following list:

1. funda.nl, helenahoeve.nl, nu.nl, postbus51.nl, wehkamp.nl
2. bol.com
3. philips.nl, rabobank.nl, vrom.nl, wnf.nl

See also these screenshots: