Testing times for operating systems

In order to get the most from your PC, a comprehensive introduction to the operating system is essential.

Testers agreed that 'Tour Windows XP' offered the best overview of the operating system: slick, colourful graphics, coupled with the choice of narrated animation or simple text and images, received praise from our participants.

Two testers complained that the tour lacked detail, but they found Windows XP Professional's informative help facility compensated for this.

Both Windows 2000 Professional and 98 Second Edition feature a Getting Started guide within the online help, which is also available in the Accessories menu. However, only one tester actually found these tutorials.

Another was impressed with the 'If you've used Windows before' option in Windows 2000 and praised its clear alphabetical indexing.

Although Windows NT 4.0 includes a similar feature, no formal tutorial is available. The testers' only option was to work through "vague" help topics as a way of familiarising themselves with the system.

Using Help and troubleshooting

An efficient operating system should include accessible and informative help files to support users. We asked our testers to search the help facility for specific topics and to troubleshoot flickering display problems.

Our testers considered the help files in Windows 2000 and XP Professional to be the most comprehensive, although one complained that navigating the detailed Help and Support Centre in Windows XP hindered his progress.

All testers were frustrated with the way Windows NT 4.0 displayed help files. "Annoyingly, the main Help Topics window closes whenever I select a particular subject," said one.

The help files in Windows 98 SE also received qualified praise. While one tester appreciated the organised topic links, another considered the glossary confusing.

Our participants liked the use of everyday language, rather than jargon, when accessing Help in Windows 2000 and Windows XP Professional. However, three testers were unable to correct display problems in Windows 98 SE, as neither 'refresh' nor 'flicker' appear as topics in Help.

One tester was impressed with NT 4.0 for including a Test shortcut to determine whether his monitor supported a particular refresh rate.

Navigation and file management

Testers were asked to create a folder and an associated shortcut from the desktop, find a specific file, copy it into their new directory, then change the date on their PC. All the operating systems take a similar approach to these tasks so, on the whole, our testers found few problems.

In addition to changing the date by double-clicking the Time and Date icon in the system Taskbar, testers were impressed that a clear 'Date, Time, Language and Regional Options' link featured in the Control Panel of Windows XP Professional.

Testers felt that, once users became familiar with Windows XP Professional, these new task-oriented menus would enhance navigation.

When attempting to locate files on their PCs, two testers favoured the layout of the Search command in Windows 2000. In this and Windows XP Professional, testers liked being able to access all search options in one display area, instead of having to use a tabbed dialog box.

Many Windows veterans rely on Windows Explorer to carry out file management tasks. Assuming its presence would be fairly prominent, one tester was annoyed that he had difficulty locating this essential function in Windows XP Professional.

Investigation

We asked testers to find specific information about their machines, including the amount of space remaining on their C:/ drive and the name of the device using IRQ 12.

They agreed that establishing free disk space wasn't particularly intuitive in Windows 98 SE. Using My Computer, testers found they couldn't view this information once inside the drive's file listing.

The other operating systems made these details accessible in a number of ways. Testers liked being presented with a tool tip displaying used and free space when hovering their cursor over the drive in Windows 2000 for example.

Being able to determine the IRQ assigned to a particular device is important for resolving resource sharing conflicts. With the exception of Windows 98 SE, testers felt that each system's help files adequately explained the purpose of IRQs.

However, all products failed to spell out a direct route to IRQ listings. With reference to Windows 98 SE, one tester complained: "It was more by luck and perseverance that I found IRQ 12."

Aware that this information could be found somewhere in the Device Manager, one tester opened the Resources tab in each device's Properties box until she came across IRQ 12 using Windows NT 4.0.

Attaching hardware and program management

Finally, we asked testers to install a Canon BJ S500 parallel port printer, remove an application from their PC and open a text file in both Notepad and WordPad.

They experienced few problems when installing the printer and considered the process to be straightforward. In most cases, they found that a wizard detected the printer's presence and then clearly guided them through the installation process.

The wizard in Windows NT 4.0 prompted testers to select a printer port, and testers were surprised at the lack of assistance here. "For a new user, this could create a needless scare," commented one.

Another was frustrated by Windows NT 4.0's slow reboot. One tester followed Windows 98 SE's Add Printer Wizard as the printer's driver application didn't auto-load from the CDRom. This path was abandoned when he couldn't find his printer type in the manufacturer's listings.

Use of the Add/Remove programs function in each operating system made deleting a program simple for two testers. Not only did participants comment on the good level of support in Windows 2000, one also felt that the interface and graphics were "more dynamic" than those she had observed in other parts of the system.

Two testers tried to delete the program file folder from Windows Explorer, but had little success. They praised Windows 2000 for advising that this action wouldn't remove the program and were grateful for a link to Add/Remove programs.

Being able to open a Notepad file in WordPad using the right-click Open With option in Windows 2000 and XP appealed to our testers.

One tester liked the fact that "common icons are displayed to help you choose the correct application". Consequently, they were disappointed that this function wasn't available in Windows NT 4.0 and 98 SE.

Conclusion

By the end of testing, three participants preferred Windows XP Professional. However, our results indicate that Windows 2000 is the better productivity tool.

Some differences in the scores will be attributable to testers never having used Windows XP Professional before and taking some time to get used to the new 'Luna' interface.

Evidence of this learning curve comes from one tester who switched from Category to Classic View in Windows XP Professional to check her whereabouts on the interface.

Overall, testers didn't need to consult help more in Windows XP Professional than in 2000, but did take slightly longer to complete tasks, hence its lower Productivity and Intuitiveness scores.

One tester felt that Windows 2000 achieved a better balance of aesthetics and functionality. "Although Windows 2000 isn't visually as exciting as XP Professional, it does feel easy to use," he said.

Testers felt that Windows 98 SE and NT 4.0 were "no-frills systems that got the job done". Nonetheless, neither help facility consistently provided useful answers to their questions.

For anyone currently using Windows NT 4.0 or 98 SE, XP Professional offers great usability advantages. However, there's less difference between Windows 2000 and XP Professional.

The major changes in the Windows XP Professional interface require some familiarisation, but most users conceded that, if they were to use the operating system for a longer period of time, it would begin to deliver greater benefits.

LAB NOTES

How we tested

Testers used Dell 733MHz Pentium III PCs with 256Mb of Ram fitted for these tests. They carried out the same set of tasks on each operating system in turn. To reduce the risk of software conflicts, a clean version of the operating system was installed between each iteration of the test. To avoid bias, participants tested the products in a different order.

The testers

To obtain meaningful feedback, and avoid bias, our four testers had differing levels of Windows operating system experience. Three were most familiar with Windows 98, while one tester had used Windows 2000 extensively. Two also had experience of Windows NT 4.0.

What we tested

We asked testers to record certain types of data, such as the time they took to complete each activity and how many times they accessed the help system. They also completed a short satisfaction questionnaire.

This information was then used to generate our usability indices:

• Productivity - The amount of accurate work completed over time
• Intuitiveness - Progress made without reference to any form of help
• Satisfaction - Testers' subjective rating of the product's capabilities
• Usability - The average of all of the above scores.

Awarding the Usability Seal of Approval

In order to qualify for a Usability Seal of Approval, a product must achieve a score above the median for the group on each of the constituent Productivity, Intuitiveness and Satisfaction indices, as well as score more than 80 per cent of the highest result on the overall Usability index.

Before we can consider awarding a seal to an individual product, we also have to be confident that the category of software/hardware as a whole is sufficiently mature to merit an award.

To achieve this, the total Satisfaction score for the group as a whole must exceed 60 per cent of the maximum possible Satisfaction score for the evaluation. In this instance, the category achieved 83.14 per cent of the total possible score.

Usability Seal of Approval

Windows 2000 Professional wins our Usability Seal of Approval, beating Windows XP Professional by a very small margin. This success can be put down to a number of different factors.

Windows 2000's navigation is simple and familiar to anyone who has used an earlier version of the operating system. A useful option in its Help system also allowed testers to learn about features that had been renamed or moved.

In contrast, testers took longer to find features for the first time in Windows XP Professional.

In addition to this, they were very impressed by Windows 2000's comprehensive search tools. At the end of testing, one participant summed up: "Windows 2000 has functionality that says 'professional' yet it's intuitive enough to work with comfortably."

Summary:

• Testers felt that users would struggle to get the most out of Windows NT 4.0, since no comprehensive tutorial is included with the software.
• Adjusting to the new look 'Luna' interface of Windows XP Professional slowed our testers on some tasks.
• File management tasks were straightforward in all of the operating systems. An intuitive search command for locating files in Windows 2000 particularly impressed testers.
• Lack of detail in both Windows 98 SE's and Windows NT 4.0's help materials was criticised by testers.
• Similar satisfaction ratings show that testers liked Windows XP Professional almost as much as Windows 2000.
• Navigating menus and drilling down to information wasn't so easy in Windows 98 SE.

Like this story? Spread the news by clicking below:

 del.icio.us     Digg this     reddit!