Because Nlpi is a rather complex and specialized software development application, it is very error prone. Basic components are a ZClass product with several nested ZClasses and a mix-in python product class whose classes call complex binaries (basically inform compilers, resource packagers and corresponding code interpreters). The ZClass can be seen as the control/viewer layer and the python class as the underlying model layer. This document tries to describe the testing strategy which was used to stabilize it.
For unit tests Nlpi uses test classes derived from unittest.TestCase, but currently no doctest strings. For functional tests it uses the slightly modified FunctionalTests product with a html_tidy plugin to check the generated html code. FunctionalTests was preferred over other testing frameworks, e.g. Puffin, because it is the only one which integrates into Zope with ZEO - the key to combine the two types of tests.
There is no clear drawing line between unit tests and functional tests. There are simple unit tests checking little class methods, but there are more complex unit tests which rely on many attribute instances generated by other modules. On the other hand, FunctionalTests only checking for a http response 200 would not cover most application failures if they would not verify postconditions, especially the attributes used by the unit tests.
Unit tests should - in theory - test small code pieces in isolation. For sure this means that no import Zope is used: the zodb persistency machinery would not allow to run the tests in a well defined environment in an easy way every time they are run. This had the consequence that a ZopeStub directory had to be appended to sys.path with simple replacement classes that can be imported by the modules under tests. An example is ./ZopeStub/OFS/ObjectManager.py defining
class ObjectManager:
def objectIds(self,f):
return []
Many tests need the path hackery import from ./ZopeStub/Globals.py allowing them to create and read from temporary files at a defined place:
import os.path
SOFTWARE_HOME = os.path.normpath('C:/Programme/Z26/lib/python')
It seems not to make sense to simulate Zope behavior with these stub methods. For that reason the modules under test sometimes need to know whether they are tested or they are running in context. For that reason the common NlpiStub base class provides a Znlpi_UnitTest attribute signaling the class under test not to call Zope methods as in this (shortened) example out of Znlpi_infObj.py. References not present in isolation are bold:
def Znlpi_add_infSrcTree(self,REQUEST,inf_sources):
if not hasattr(self,'Znlpi_UnitTest'): # skip this while unit testing
...
for folder in folder_tree:
obj = current_folder.unrestrictedTraverse(folder,None)
if obj == None:
newobj = current_folder.Control_Panel.Products.NLPIProduct.NLPIClass.Znlpi_FolderClass(folder)
newobj._setId(folder)
current_folder._setObject(folder, newobj)
obj = current_folder.unrestrictedTraverse(folder,None)
The NlpiStub class replaces the ZClass which would in context inherit from the class under test. Nlpi consists of several modules building on each other. The dependencies are abstractely maintained by the Znlpi_manager.py module, but as the unit tests should test each module in isolation, it is not used except for testing itself. The first base class which contains only attributes set up by default or by the user when creating an NLPi instance looks like this:
class NlpiStub_in_Znlpi_iclProcessIcl: Znlpiprop_uploadFileType = 1 Znlpi_icl_file = TestPath.adventureExamplePath + '/Advent.icl'The class used in the unit test inherits from that data stub class and from the class under test (in test_Znlpi_iclProcess.py):
class nlpiIcl(NlpiStub_in_Znlpi_iclProcessIcl,Znlpi_iclProcess.Znlpi_iclProcess): pass
There is a similar stub class representing the expected state (in terms of attributes and values) expected after the method under test was run, here simplified with some example attributes:
class NlpiStub_out_Znlpi_iclProcessIcl: Znlpi_icl_switches = 'DkE2D' Znlpi_icl_relPath = ['', 'advent.sources', 'adv.extern', 'advent.sources/mazes']
The test class instantiates both classes, the input state and the expected output state. The NlpiStub base class provides the verifyAttributes method to test afterwards whether the states match or otherwise fail with a standard message about which attributes were not equal. See this example test method:
def test_icl_read_main_icl(self):
self.nlpi = nlpiIcl()
self.resultNlpi = NlpiStub_out_Znlpi_iclProcessIcl()
self.nlpi.Znlpi_icl_read_main(self.REQUEST, self.PARENTS)
#self.nlpi.printObjectAttributes()
self.failUnless(len(self.nlpi.Znlpi_icl_rawiclfile) > 2,
self.msg('Znlpi_icl_rawiclfile: icl file too short'))
self.failUnless(self.resultNlpi.verifyAttributes(self.nlpi),
self.msg(self.resultNlpi.attrMsg))
Znlpi_icl_read_main is the method under test. The first error condition is obviously for this method only, but the second condition is common for any tests using NlpiStub. In case of mismatch the error message looks as follows (except the Traceback and the unit test failure):
AssertionError: Znlpi_iclProcess: verifyAttributes Znlpi Attribute mismatch: Znlpi_icl_switches Znlpi: 'DkE2D' Stub: 'DkE2D wrong'
The commented printObjectAttributes method prints all the attributes to the tty for easy copy-pasting the expected results into the code when creating new successor stub classes.
A speciall case is the ./NlpiStub/NlpiStub_infObjListDict.py module: It provides one huge data class containing all the inform objects generated by the inform compiler debug information file which is converted into a python pipe stream by infact_python.exe. Implementing that required hacking the compiler (written in c) and the original infact.c code - lots of possible bugs. Thus every time the corresponding test (in test_Znlpi_infObj.py) is run the file ./NlpiStub/NlpiStub_infObjListDict_New.py is overwritten. At the end the test module test_z_compareListDicts.py tests the newly generated class and the reference class for equality. If more information gets added to the debug information file, the file generated afterwards becomes the new reference module.
Unlike testing simple Zope objects it is inadequate to simply test whether a Nlpi instance can be added without generating server errors. Nlpi is an if developing framework. Users typically generate one instance and then use it for a very long time, change preferences, run the application expecting different results, add functionality to their inform source code, navigate through the Nlpi user interface and run it again and again.
For that reasons a lot of http requests must be generated, sometimes containig a lot of crucial POST parameters. Because TkLoggingProxy.py didn't run on win32 and doesn't record POST parameters anyway, a temporariy hack for the Zope http server was needed. For details see the "tty logger" in the references on the bottom. It is just no fun to repeatedly create script entries like
[Prefs_Glulxe] URL: %(portal_url)s/%(site_path)s/%(nlpi_instance)s/NLPI%%20Preferences/Znlpi_changePrefsLocal Authentication: %(userid)s:%(password)s Field_1: Znlpi_icl_file:string=%(advent_source_path)s/Advent.icl Field_2: Znlpiprop_uploadFileType:int=1 Field_3: Znlpi_bin_codeInterpreter:int=3 Field_4: Znlpi_bin_inform_code:int=1 Field_5: btnChangeOptions= Change Expected_Result: 302
by looking at the html source to find out that form values or even filling automatically generated text fields as in
[Sub_addWholeTree] URL: %(portal_url)s/%(site_path)s/%(nlpi_instance)s/%(default_walk_id)s/aSubWalk/Znlpidtml_ZipWalkXmlChangeEdit Authentication: %(userid)s:%(password)s Field_1: Znlpi_ZipWalkTreeXml:lines=<?xml version="1.0" ?>\n<walk id="aSubWalk">\ns\ns\ns\nopen the grate\n<walk id=" Walk">\nenter\ntake all\nexit\n <walk id="aSubWalk">\ns\ns\ns\nopen the grate\n </walk>\n <walk id="aSubWalk2">\ns\ns\ns\nopen the grate\n </walk>\n</walk>\n</walk>\n Expected_Result: 302
Additionally putting it all in one *.zft file was not appropriate, but FunctionalTests didn't count the overall errors. If one early test script failed, but the others succeeded, the failure message was already scrolled out of sight. This was fixed in a little modification, see the references.
There were lots of funny surprises when looking at the application with different browsers. Writing all that nested dtml code to generate html tables and forms lead to a lot of violations of the html format which my current browser nicely accepted, but not the ones not installed on my pc for good reasons. The modification with html_tidy is for sure not the best on the market, especially because it produces warnings about things that normelly are perfectly well. But a) it is maintained and b) it is available for free. See the html_tidy output example in the references for a bigger output listing, here is just an excerpt of the command line output:
C:\Programme\Z26\lib\python\Products\NLPIaux\functional_tests\nlpi_basic>python ..\..\..\FunctionalTests\FTRunner.py -h
-h simple_play.zft pictures.zft simple_variants.zft walk.zft zwiki.zft delete.zft
===============================================================================
Test: Check whether anything is installed to simply compile a game
===============================================================================
Request : URL : Time (s)
-------------------------------------------------------------------------------
Home_Page : 0.340
line 3 column 27 - Warning: <!DOCTYPE> isn't allowed after elements
line 5 column 1 - Warning: discarding unexpected <html>
line 3 column 27 - Warning: <head> isn't allowed in <body> elements
line 3 column 27 - Warning: <title> isn't allowed in <body> elements
line 3 column 27 - Warning: <link> isn't allowed in <body> elements
...
Play_WinFrot: /nlpi_tests/adventure/Znlpi_console 9.584
line 227 column 8 - Warning: <a> converting backslash in URI to slash
line 264 column 20 - Warning: replacing element </p> by <br>
line 264 column 20 - Warning: inserting implicit <br>
line 264 column 336 - Warning: unescaped & or unknown entity "&file"
...
Because Nlpi uses a lot of binary components, correct http response values are only one part of the functional tests. There are several windows applications expected to be started in their own windows by a complete functional test run, namely a text editor opening some inform source files at a defined line and different Z-code and glulx interpreters, some with picture resources, some without.
It is up to the tester to verify whether all the applications are running and the expected resources really are present in the expected instances after the functional tests have succeeded. These error conditions can not be automatically verifiated, at least not without completely different functional tests really simulating keyboard and mouse events and verifying graphical window content.
It doesn't matter whether you call it a problem or you call it a feature: some of the final ZClass attributes are properties inherited from ordinary ZClass PropertySheet instances, other are just instance attributes inherited from the python mix-in class which set them somewhere deep in the code, some come from places nobody knows. Nlpi was developed sometimes iteratively, other times rather incrementally. The point of this is that the programmer can never by sure whether the attributes written down in the stub ZClasses really are present when run in context.
To verify the adequacy of the ZClass stub classes, the powerful mechanism of integrating python code into FunctionalTests scripts is used. For example the simple_play.zft script uses the following test at the end:
[AiclProcess] Filename: ./simple_play.py Function: verifyAttr_iclProcess
The method called is defined in the module simple_play.py as follows, with a little helper method at the beginning:
def __getNlpiInstance(app, test_vars): path = '/' + test_vars['site_path'] + '/' + test_vars['nlpi_instance'] return app.unrestrictedTraverse(path) def verifyAttr_iclProcess(app, test_vars, result): Znlpi = __getNlpiInstance(app, test_vars) refNlpi = NlpiStub.NlpiStub_out_Znlpi_iclProcessIcl() ok = refNlpi.verifyAttributes(Znlpi) sys.stderr.write(refNlpi.attrMsg) return ok
This code matches exactly the code in the Nlpi ZClass Stub section! Znlpi here is the in-context Nlpi instance created by a http request sequence fired by FunctionalTests. Formerly this was just the self.nlpi = nlpiIcl() stub. What was called self.resultNlpi in the unit tests is here called refNlpi. The failUnless error condition in the unit test framework is captured with the return value within the FunctionalTests framework.
In case of attribute mismatches, FunctionalTests generates the same output as unit tests would do. Therefore it is guaranteed that the stub class in the unit tests really represents the state of the ZClass instance, or, the other way round, that the ZClass really generates the attributes expected from the unit tests.