The following was written for the professional -
software developer.
Pre-Replication Product Testing and Tweaking: Part 2
© 1999, by Terry E. Mercer, Pacific Buyers'
Group
Published by Replication News, 1999
When creating a product, most programmers and developers seem to ignore two very
critical things: 1) that other people use different equipment (hardware &
software) that may alter how their particular program operates, and 2) that the majority
of the computer users of most programs are novices (possibly knowing three or four
programs to an intermediate level). Novice users flood technical support lines,
bad mouth a wide variety of programs for a lack of user-friendliness, and often blame
publishers for crashing a perfectly good computer when THAT PUBLISHER'S software was
installed. The average user is able to effectively do an amazingly wide variety of
inappropriate things to a seemingly flawless program: not reading messages,
clicking mouse buttons (while the program is already trying to go somewhere), hitting the
reset button because a program is taking too long to respond, and deleting a program
through Windows Explorer because they didnt like it, or dont want it. These
are among the most common reasons most software companies are busy with technical support
calls. Many of these new end-users ends up throwing out the offending product in
frustration. The average user reads directions as a LAST RESORT, as
"modern" programs are assumed to be intuitive and interactive, which means instructions
need to be written in such a fashion that the novice can clearly understand the
information presented. Basic testing by a
disinterested third party can save ANY company a lot of money, time, and reputation. The
steps I describe will help maximize your customer's marketing efforts, ability to sell
their product, their need for more CD's and yes, give you a greater
opportunity for more money through both new orders and future testing.
The topic the software covers does NOT matter. All software products have four primary
areas that must be considered and analyzed:
1) installation,
2) intended use,
3) common misuse,
4) uninstallation.
Before I explain what to look for in software, things to try and how you can best help
your customer, it is important to setup at least a basic testing area. This can be a room
or cubicle, to a whole building, depending on how thorough you want to be, and what your
budget is. For the purpose of this article, I will describe an extremely effective,
low-cost testing center.
Staffing: To effectively test a product you should have at minimum of two people.
One must be very knowledgeable and comfortable with both hardware and software, as they
will be responsible for keeping the hard drives going, backing up data and maintaining the
component lists. The second person must be a computer novice. For example, a high school
or college student, retired parent, anyone with very little knowledge. The beginner's job
is to help your high-end person observe how "normal" people will try to operate
the program. This "team" is likely to open up a whole new set of problems that
were never previously considered by either the advanced user or the program developers.
The key is not to think that the beginner is "stupid" or "not following
instructions," it is learning what they do and don't do. These observations will
provide valuable suggestions and recommendations for your client that will make their
program better and help them sell more of the CD-ROM discs you replicate for them.
Equipment: When I set up a test center, I try to have at least three or four
computer systems that start at the minimum requirements the developer or publisher plans
to advertise on the packaging. You don't need the biggest or best, as the average user
won't be using that. You will need good, solid, and upgradeable boxes that offer you the
greatest degree of flexibility and future expansion in your test facility. Today, most
software states that it requires a 486DX system or better. I often take these words to the
extreme, starting with the lowest "common" system, a 486DX33 or DX66. I also try
to have a 486DX133 and a Pentium 66, two very common CPUs with distinctive architecture.
For added testing and programs that state that they require a "Pentium or
better," I usually try to set up a Pentium 90, Pentium 166, Pentium 233, Pentium II
300, and a Pentium II 400. The reason I use BOTH a 300 and a 400 is that there are ECC
(Error Correction Control) options in the 400 that don't exist in the other types of
CPU's. Beginning with the Pentium II-350 (all 400's and better) ECC, RAM, true 100MHz
front-side BUS speeds, and the AGP port all add up to a change in the rules and the
program's action and reaction. I have seen programs that had some very strange and
unpredictable results on high-end computers, which functioned perfectly on the lower-end
systems. If you counted, that sounds like six to nine different computers, and it can be,
in more thorough testing labs or centers there maybe 20 or more, but it is not required
for the basics
or for low over head test centers. The basics can be dealt with
effectively with only two or three systems with removable drives. Keep in mind that
nowhere have I mentioned the AMD, Cyrix, Win Chip, Intel's Celeron, or IDT CPU's - some of
which change the rules about how a program will operate, but seldom effect the four basics
which is the fastest, easiest, and most common cause of technical support. (At one
time, nearly 80% of Microsofts technical support was with the basics, 15% on
intermediate (actual) use, and less than 3% with OEMs and reseller training, and
only 2% on advanced use). Solving the basics is vital its money in the pocket
for all parties involved.
My test center uses 17" monitors because the larger monitors are easier on the
eyes. I set up a switch box with one monitor, keyboard and mouse, capable of effectively
switching between one to eight computer "boxes" at the same time. In addition, I
use removable hard drive chassis for the boot file on my test systems. This setup offers
me the greatest flexibility with the least cost and stress. I usually have a backup of
each computer's operating system, the drivers required for that computer and a few
different "common" programs on either a ZIP, Jazz, tape, or CD-ROM. This means
that when a program "blows out a drive" - it can be quickly restored to its
original settings with minimal down time. One set of hardware, such as a Pentium 166
system, a 3.5" floppy drive, 16X CD-ROM drive, and 32MB of RAM might have between two
to five hard drives set up for it. One drive would have DOS 5.0 and Windows v3.1x, another
would contain the Windows 95 upgrade (from Windows v3.1x), a different hard drive would
contain the full version of Windows 95, the fourth would have Windows 98, and possibly one
more with Windows NT. It is important to note that the upgrade versions are different than
the full versions, which in turn are different than the OEM versions. All need to be
tested if the program uses any strange or unique installation or removal programs. The
hard drives don't have to be large. In most cases 540MB is sufficient. Anything over 2.1
Gigabytes is overkill for testing 99.9% of the software being sold today. It can be
important to be able to test the FAT 16, FAT 32 and NTFS, but this usually isn't necessary
for the primary four testing phases (unless the program is a utility that alters the FAT
(File Attribution Tables) or Partition information). It is important to note that the test
systems should use different brands of video cards, different types of modems, and varying
amounts of RAM. Unless your customer requires testing on specific devices, floppy drives,
CD-ROM drives, keyboards, monitors, speakers, joysticks, and mice have no effect on
software 95% of the time and are an unnecessary expenditure of time and money.
Installation: Does it auto-run? If it does, what happens when the disc is put in
AFTER installation? Any strange or tricky installation screens? What about installation or
registration codes? Are they complicated or difficult to find? Will a fictitious code
work? How will the installation program function on a system? Where will the files be
placed on the hard drive? Is there a danger that the files (mostly DLL's) will be
overwritten with the program on installation, thus causing another program to have
problems? A self-contained program is always the safest and usually the best. How will the
installation program function under different operating systems? Will the instructions on
the packaging be adequately and accurately marked? A self-contained program is always the
safest to use and generally the best.
Intended Use: Who will likely be purchasing and trying to use the program? Is it
likely to conflict with other types of programs that advanced users will notice, but
beginners will overlook? Does it effect file associations? Keep in mind, many fax, modem,
web-related, and printer programs are loaded through the START-UP or the registry. With
more RAM and multi-tasking capability, remember that novices often don't think about
closing other programs before trying to install or run a new program. Will that have any
ill effects on the new program or the user's computer? Once the program is installed, how
does the end-user access it? Through the START/PROGRAMS menu? Is there an icon on the
desktop? Is it auto-loaded? Does the program use "standard" key strokes and menu
selections? Does it effect the shut down or restarting of the computer system? Is there an
effective recovery system for power failures? Does the flow and process make sense? When
the program is ended, does it release the memory it used? Does it clean up any temp files
it created?
Unintentional Misuse: How will an end user likely try to use the program? This is
nearly impossible to guess at. You almost have to watch a novice operate the new program.
There are only three possible answers to this question: 1) everything is fine, 2) better
documentation and on-screen instructions are necessary, or 3) the program needs to be
changed in the areas that caused confusion. Be forewarned, at this point in the game, most
programmers don't want to hear about any changes. However, clearly defined problems
presented with several viable solutions can ease that stress. Generally with tact and
practice, your staff can learn to break the news and accurately evaluate the level or
degree of importance. Use a one to ten scale of importance. Learn to rate the likelihood
of errors or problems.
Uninstalling: Does the program have its own un-install routine? Does
"Add/Remove Program" see its routine? What happens if someone just deletes it
from Explorer? Are any files deleted from the Windows or Windows/System folders deleted
that effect-unrelated programs? Are the references to the program deleted or removed from
the registry correctly? If associations are created on specific file types, are they: 1)
left on the system with program errors when double clicking on a given file, 2) removed,
so that the user needs to recreate associations to a different program, or 3) redirected
to a different program, preferably the one it replaced? Finally, what happens to the
end-user's DATA files during the uninstall? Are they deleted with or without warning? Is
the data left on the drive? Can the data be accessed by another program?
The questions for these four basic problems seem limitless, but there are common
questions and often simple answers. As the saying goes: Prior Planning Prevents Poor
Performance. Testing and gathering the answers to the questions will make a pretty full
day (or two) for a couple of product testers with reasonable equipment and resources at
hand. What is their time worth? What is it worth to your customer to limit or eliminate
problems prior to replication, printing, marketing, distribution and telephones ringing
with unhappy customers?
If you want more details, have specific questions, or want help setting up and
establishing a testing process, contact me at terry@helpus.com.