dear david & allen
last week i did an overhaul on ScreenTest, and i'm quite pleased with
the result. today i succeeded in getting ClutTest to pass the 10-bit
ATI DualHead cards that doug taylor and rhea eskew use.
out of these experiences, i'm drawing a lesson about test programs.
they come in two flavors, both useful.
1. PASS/FAIL. most of our test programs are what you write when you
want to exercise a new function or document what's wrong (eg a bug
report). these programs are pass/fail. they're usually totally boring
to run since, if they work, they don't do anything interesting. it
helps a lot, by the way, if the program prints a conclusion at the
end, e.g. "Success!", so the user is left in no doubt that the
uninteresting display was indeed the proper thing.
2. DIAGNOSTIC. That might not be the right name for this category.
TimeVideo and ScreenTest are the only instances I know of. However, I
think that the new ScreenTest is a much more successful
implementation.
TimeVideo and the old ScreenTest evolved over a long time to test and
report a large number of facts that seemed useful in characterizing a
video driver. They were very useful, both as accompaniments to bug
reports to help us get the context, and in characterizing what is odd
about each new driver, so that we could figure out how to cope with
it gracefully. However, internally, because the facts are so diverse
and were added separately over a long period of time, both programs
became rats nests, very hard to read and maintain.
MODULAR. The new ScreenTest is fairly short. Basically it first calls
DescribeComputer, then DescribeScreen, then it does a little bit of
testing and reporting, and then it calls SetClutWaitTest. Each of the
subroutines is designed to run as a subroutine (when you supply
arguments) or standalone (when you don't). And each subroutine is
quite focussed on one issue, so the code is fairly simple. Of course,
you already know that modular programming is good, but in the past it
hadn't been clear to me that this was possible for ScreenTest.
Furthermore, I now realize that our multiplatform effort makes it
very desirable to be able to use pieces stand-alone. Thus when Allen
gets around to getting ScreenTest to work on Windows, he can work on
one routine at a time, e.g. DescribeComputer, DescribeScreen,or
SetClutWaitTest, getting it to run stand-alone while the rest doesn't
yet work.
GENERAL. However, what i think is most significant about the new
ScreenTest, or, to be precise, SetClutWaitTest, which it calls, is
that it takes a fully general and systematic approach to testing,
testing at chosen sample points that represent the whole space, and
reducing the multidimensional table of results to a simple
conclusion. i couldn't have done that until recently, because i
didn't have a good grasp on what the testing space was that needed to
be sampled, but in the last revision it suddenly was clear. i had
written a large number of closely related SetClut-timing test
programs and it finally become clear that testing results, across
video cards, depend on three parameters: pixelSize, prioritySequence,
and number of elements loaded. Once that was clear, it was
straightforward to write a fully general test and analysis.
I consider that a success. It will now be very easy to run that test,
glance at the results, and add an appropriate conditional to
OpenScreen.m to set things up properly for each new video driver. In
principle we could set things based on testing each time, but
general-case testing takes too long for that.
I think that the next step would be an enhanced version of ClutTest.m
that would characterize how 8+ bit cards handle numbers (i.e. how
video drivers with more-than-8-bit DACs handle color values in
SetClut, GetClut, and Gamma calls). I started writing out what I know
in the help text for ClutTest. David has more experience here than I
do, but it seems that one can enumerate all the possibilities and
test them all, to fully characterize each driver. Again we can
continue with the current practice of using special-case tests for
each driver, but i suspect that these tests, once developed, will
turn out to be quick, so we may want to do runtime testing and
configuration in OpenScreen.m for 8+ cards. Anyway, both schemes can
co-exist.
David: does ClutTest (recently revised) produce sensible results on
your 8+ (10-bit DAC) cards?
best
denis
p.s. i've enclosed the latest ClutTest.m for any forum readers who
wants to try it on his/her 8+ cards. results, thoughts, and
improvements would be welcome.
last week i did an overhaul on ScreenTest, and i'm quite pleased with
the result. today i succeeded in getting ClutTest to pass the 10-bit
ATI DualHead cards that doug taylor and rhea eskew use.
out of these experiences, i'm drawing a lesson about test programs.
they come in two flavors, both useful.
1. PASS/FAIL. most of our test programs are what you write when you
want to exercise a new function or document what's wrong (eg a bug
report). these programs are pass/fail. they're usually totally boring
to run since, if they work, they don't do anything interesting. it
helps a lot, by the way, if the program prints a conclusion at the
end, e.g. "Success!", so the user is left in no doubt that the
uninteresting display was indeed the proper thing.
2. DIAGNOSTIC. That might not be the right name for this category.
TimeVideo and ScreenTest are the only instances I know of. However, I
think that the new ScreenTest is a much more successful
implementation.
TimeVideo and the old ScreenTest evolved over a long time to test and
report a large number of facts that seemed useful in characterizing a
video driver. They were very useful, both as accompaniments to bug
reports to help us get the context, and in characterizing what is odd
about each new driver, so that we could figure out how to cope with
it gracefully. However, internally, because the facts are so diverse
and were added separately over a long period of time, both programs
became rats nests, very hard to read and maintain.
MODULAR. The new ScreenTest is fairly short. Basically it first calls
DescribeComputer, then DescribeScreen, then it does a little bit of
testing and reporting, and then it calls SetClutWaitTest. Each of the
subroutines is designed to run as a subroutine (when you supply
arguments) or standalone (when you don't). And each subroutine is
quite focussed on one issue, so the code is fairly simple. Of course,
you already know that modular programming is good, but in the past it
hadn't been clear to me that this was possible for ScreenTest.
Furthermore, I now realize that our multiplatform effort makes it
very desirable to be able to use pieces stand-alone. Thus when Allen
gets around to getting ScreenTest to work on Windows, he can work on
one routine at a time, e.g. DescribeComputer, DescribeScreen,or
SetClutWaitTest, getting it to run stand-alone while the rest doesn't
yet work.
GENERAL. However, what i think is most significant about the new
ScreenTest, or, to be precise, SetClutWaitTest, which it calls, is
that it takes a fully general and systematic approach to testing,
testing at chosen sample points that represent the whole space, and
reducing the multidimensional table of results to a simple
conclusion. i couldn't have done that until recently, because i
didn't have a good grasp on what the testing space was that needed to
be sampled, but in the last revision it suddenly was clear. i had
written a large number of closely related SetClut-timing test
programs and it finally become clear that testing results, across
video cards, depend on three parameters: pixelSize, prioritySequence,
and number of elements loaded. Once that was clear, it was
straightforward to write a fully general test and analysis.
I consider that a success. It will now be very easy to run that test,
glance at the results, and add an appropriate conditional to
OpenScreen.m to set things up properly for each new video driver. In
principle we could set things based on testing each time, but
general-case testing takes too long for that.
I think that the next step would be an enhanced version of ClutTest.m
that would characterize how 8+ bit cards handle numbers (i.e. how
video drivers with more-than-8-bit DACs handle color values in
SetClut, GetClut, and Gamma calls). I started writing out what I know
in the help text for ClutTest. David has more experience here than I
do, but it seems that one can enumerate all the possibilities and
test them all, to fully characterize each driver. Again we can
continue with the current practice of using special-case tests for
each driver, but i suspect that these tests, once developed, will
turn out to be quick, so we may want to do runtime testing and
configuration in OpenScreen.m for 8+ cards. Anyway, both schemes can
co-exist.
David: does ClutTest (recently revised) produce sensible results on
your 8+ (10-bit DAC) cards?
best
denis
p.s. i've enclosed the latest ClutTest.m for any forum readers who
wants to try it on his/her 8+ cards. results, thoughts, and
improvements would be welcome.