debugging | index e:\vslprf\dlr\languages\ironpython\tests\debugging\__init__.py |
-------------------------------------------------------------------------------
INTRODUCTION
This test plan addresses debugging test coverage for the 2.x series of
IronPython interpreter releases.
EDITORS NOTES
Throughout this document you will find references to documentation in
other packages similar to "See documentation for debugging.pdb_module". This simply
means that relative to this package (i.e., debugging), you should follow
the 'pdb_module (package)' link at the bottom of this page.
-------------------------------------------------------------------------------
FEATURE GOALS
The goals of IP's debugging feature is to provide IronPython users with
complete use of CPython's existing debugger functionality, the "pdb" and
"bdb" modules respectively, while also providing a great interactive debugging
experience using Microsoft tools such as mdbg and the Visual Studio IDE.
-------------------------------------------------------------------------------
PRIMARY TESTING CONCERNS
- new builtin module attributes added to IP to support CPython's "bdb" and "pdb"
modules
- compatibility with CPython's use of the "bdb" and "pdb" modules
- what happens breakpoint-wise when IP steps into .NET code from "pdb"?
- .NET applications hosting IP should be able to debug Python code
- the experience of debugging IP under Microsoft's mdbg and Visual
Studio tools is what existing CSharp and VB.Net would expect; but still
Pythonic
- "-D" command-line flag
-------------------------------------------------------------------------------
PRIMARY TESTING FOCUS
Our principal testing focus will be on testing that IP's support of the
"pdb" module is completely compatible with that of CPython by running existing
CPython tests, and also by creating a new generic side-by-side testing framework.
The secondardy focus will be testing that debugging IP under mdbg is a
pleasant experience. This will initially be acheived by using the CLR Team's
existing debugging test framework, but in the long run we will move to a
new generic, side-by-side testing framework which can be reused for other
side-by-side testing areas.
-------------------------------------------------------------------------------
REFERENCES
* IronPython Test Plan Internal IP SharePoint site
* Feature Specifications None...
* Development Documents None...
* Schedule Documents None...
* CPython pdb module http://docs.python.org/library/pdb.html
* CPython bdb module http://docs.python.org/library/bdb.html
* pdb tutorial http://onlamp.com/pub/a/python/2005/09/01/debugger.html
* MDbg.exe http://msdn.microsoft.com/en-us/library/ms229861.aspx
* Debugging IP in Visual Studio http://devhawk.net/2008/05/08/Debugging+IronPython+Code+In+Visual+Studio.aspx
* Bug queries http://ironpython.codeplex.com/WorkItem/AdvancedList.aspx
and set the component to "Debugging"
* Build drop location Updated at http://ironpython.codeplex.com/SourceControl/ListDownloadableCommits.aspx
on weekdays
* source file tree $/.../dlr/Debugging/ and also $/.../dlr/.../IronPython/*
-------------------------------------------------------------------------------
PERSONNEL
Program Manager: hpierson
Developer: dinov
Tester: dfugate
-------------------------------------------------------------------------------
TESTING SCHEDULE
Phase I - Test Plan and Enabling Existing Tests
- Write test plan for IronPython Debugging feature
- Enable any existing CPython test cases for "pdb" and "bdb" in SNAP
- Enable any existing CPython test cases for Python features supporting
"pdb" and "bdb" in SNAP
Phase II - Examine Coverage of Existing Cases and Write High-priority Cases
- Verify that developer tests for new additions to IP
supporting "pdb" and "bdb" modules are sufficient. Write more test
cases if not
- Write additional test cases for the "pdb" module if CPython's test cases
are insufficient
- Examine our existing mdbg test (cases) looking for holes. Write new test
cases if needed
- Author a few manual sanity test cases for verifying IP can
be debugged under the Visual Studio IDE. These should be folded into
the CodePlex test pass signoff procedure for major releases
Phase III - Design and Implement Side-by-Side Testing Framework
- Based on the needs of test cases added in the previous phase either:
1. Design a side-by-side testing infrastructure to be used with testing
IP's CPython debugging compatibility OR
2. Look for existing side-by-side testing frameworks we could reuse.
The pexpect package might be useful here
- Implement (or add existing) side-by-side testing framework
Phase IV - Implement High-priority Cases
- Implement test cases for additions to IP which support the
"pdb" module IF NEEDED
- Implement test cases for the "pdb" module IF NEEDED
- Implement test cases for mdbg IF NEEDED
Phase IV - Write Lower Priority Cases
- Write test cases around stepping into (hosted) Python code from a
.NET application
- Write test cases around stepping into .NET code from an ipy.exe
process
- Write performance test cases
Phase V - Implement Lower Priority Cases
- Implement test cases around stepping into (hosted) Python code from a
.NET application
- Implement test cases around stepping into .NET code from an ipy.exe
process
- Implement performance test cases
Phase VI - Future
- Develop test cases for the "bdb" module if CPython's test cases are
insufficient
- Automate manual sanity Visual Studio debugging tests
- Migrate existing mdbg tests to the new side-by-side testing framework
- Low-resource test cases and implementation
- Reliability test cases and implementation
- Scalability test cases and implementation
- Stress test cases and implementation
- Internationalization test cases and implementation
Phase P - All Public Releases
- Verify all user documentation with respect to debugging
- Setup testing - verify pdb.py and all supporting modules are installed by the MSI
- Generate code coverage report and verify block coverage as greater
than 80%
-------------------------------------------------------------------------------
FEATURE HISTORY
2.0: Tests implemented for IronPython under mdbg. Scenarios which still
need to be covered are:
* Attach scenario with and without the -D option
* Step into/out when there is no pdb file for IP
binaries
* Multiple statements in one line (e.g., "x = 1;y=2"
* if x and \n y: print 1
* dict/tuple spanning multiple lines
* break/continue, explicit raise
* other multi-line statement scenarios such as:
if True: print 1
if True:\n print 1
* MdbgInterop
able to break into the C# code from Python
able to break back into Python code from C#
2.6B2: First release of IronPython to include support of CPython's pdb module.
Only works if -X:Frames or -X:FullFrames option supplied to ipy.exe.
No pdb-related tests running in SNAP at this point, but we are running
all of CPython's tests WRT supporting functions in the sys module.
Debugging test plan completely rewritten in the form of pydoc strings,
and made public.
-------------------------------------------------------------------------------
FEATURES:
- sys.settrace, sys.gettrace, and sys.call_tracing. NOTE: while each of these
three builtin module members needs to be tested within unit tests on their
own, the real use case here is through the "bdb" and "pdb" modules
- ipy.exe's generation of PDBs
- ipy.exe's PDB visualizers
-------------------------------------------------------------------------------
FILES AND MODULES:
- FILES LIST:
* everything under Microsoft.Scripting.Debugging, although the DLR
officially owns this
* sys.cs
* IronPython.Runtime.PythonTracebackListener
- REGISTRY, INI SETTINGS: None
- SETUP PROCEDURES:
CPython standard library must be installed and present in sys.path to be
usable
- DE-INSTALLATION PROCEDURES
N/A
- DATABASE SETUP AND PROCEDURES
N/A
- NETWORK DOMAIN/TOPOLOGIES CONFIGURATION PROCEDURES
N/A
- PERFORMANCE MONITORING COUNTERS SETUP AND CONFIGURATIONS
N/A
-------------------------------------------------------------------------------
OPERATIONAL ISSUES
N/A. This feature of IP is not being monitored/maintained by operational
staff, and IP is provided on an as-is basis.
-------------------------------------------------------------------------------
SCOPE OF TEST CASES
We'll be getting most of our test coverage on the debugging feature of
IP by unit testing the CPython 'pdb' module. Doing this thoroughly
should guarantee we hit nearly all blocks of code supporting debugging in
IP DLLs. This will be validated and verified through monthly code
coverage runs. As an aside, this should in theory hit most of the DLR's
Microsoft.Scripting.Debugging.dll as well.
As IP currently has no tie-in into the Visual Studio IDE, this is
arguably far less important to test than our support of the 'pdb' module or
even mdbg. As a result, the Microsoft toolset testing emphasis will involve
exahaustively covering all aspects of mdbg. That said, some of our users do in
fact have VS Pro installed implying that we should perform at least minimal,
manual sanity tests before every major public release of IP. Should we
ever provide visualizers for our debug symbols we'll need much more
comprehensive and automated tests for VS of course.
-------------------------------------------------------------------------------
ACCEPTANCE CRITERIA
- no debugging related feature should be checked into the internal source
repository without some form of unit test hooked into SNAP checked in as
well
- we should not publically advertise the existance of any new debugging features
in an IP release unless:
- if the feature is CPython-based; the majority of it's corresponding
CPython unit test cases have been enabled in SNAP and pass
- if the feature is novel to IP; block coverage should be greater
than 70%
- the feature needs to be documented in some form other than a blog
- performance of the feature needs to be "within reason"
- if the feature is not CPython-based; it needs a Pythonic feel to it
- at least half of all bugs opened on the feature since its inception are
fixed
-------------------------------------------------------------------------------
KEY FEATURE ISSUES
- no spec on debugging IP from mdbg, VS, or what debugging symbols
should be generated for various Python constructs
- insufficient test resources to adequately test this feature
- insufficient resources to implement all debugging features called out in this
test plan
- existing mdbg tests are based on CLR infrastructure and require a Perl
installation. We cannot redistribute this publically
- existing mdbg tests in SNAP are flakey
-------------------------------------------------------------------------------
TEST APPROACH
- DESIGN VALIDATION
We have no part in the design of the "pdb" and "bdb" standard modules.
As for debugging under mdbg...there is no current design document on
debugging Python sessions. Generally speaking test will simply ensure that
whatever mdbg functionality exists with regard to IP is "Pythonic",
but at the same time familiar to CSharp and VB developers.
- DATA VALIDATION
pdb/bdb: at a very low-level, all we need to validate is that parameters passed
to a trace function we set via sys.settrace are as expected. In particular,
the first parameter passed in is the current stack frame which we have a
very solid chance of getting wrong.
mdbg/VS: need to validate that generation of PDB symbols "make sense"
with respect to whatever Python source is being compiled. There are at
least two ways to accomplish this:
1. Create a regression test in which we compare the generated PDB of a
known Python script to an expected PDB file. The Python script should
exhaustively cover everything that's possible in the Python grammar.
Testing in this manner is not really maintainable in the long run as we
expect the PDB generated for a given Python script to change over time
reflecting optimizations and new features added to IP
2. Test that the behavior of mdbg, given a known Python script, does not
regress. Again, this Python script should exhaustively cover everything
that's possible in Python's grammar, but this time around we'll also
need to cover everything that's possible under mdbg
- API TESTING
We'll exhaustively cover the entire "pdb" standard module without ever
touching the CSharp APIs directly. Prior coverage runs have shown that
(generally speaking) any block of code that is not directly hittable through
the Python API is likely dead code. Also, testing "bdb" module is lower
priority as "pdb" uses this for its own implementation. The final
justification for this is that there is actually very little in terms of
debugging APIs in IP DLLs. Most of the debugging support is built
directly into the DLR's Microsoft.Scripting.Debugging.dll.
For mdbg/VS, we'll exhaustively cover all mdbg commands.
- CONTENT TESTING
All debugging documentation should be thoroughly reviewed by Test before any
public release of IronPython. Careful attention will be given to duplicating
exactly any command fed into ipy.exe or mdbg.exe, and this should be
automated entirely if feasible.
- LOW-RESOURCE TESTING
Use of the debugging feature has a few side effects which might be interesting
from a limited resources perspective. First and foremost, use of the -X:Frames
and -X:FullFrames IP console flags imply that IP will consume more
memory than under normal circumstances. We should check that IP does not
consume "too much" extra memory. Next, use of the debugging flag, '-D', in
conjunction with the '-X:SaveAssemblies' flag will generate
Snippets.debug.scripting.pdb in the %TMP% directory. What occurs if the drive
containing %TMP% is full?
NOTE: while this is an interesting test area to explore, there are no current
plans to test it. We should perform general low-resource testing of
IP before tackling it for this specific feature.
- SETUP TESTING
The only option of IP's MSI installer capable of affecting
the debugging feature is the ability to selectively install CPython's standard
library. As we generate the list of CPython modules to include in the MSI
dynamically, it would be worthwhile to always check that pdb.py is included.
NOTE: IronPython currently has no automated tests for the MSI. Once we
do, we need to add a check to ensure pdb.py is always included. For the time
being this will have to be done manually.
- MODES AND RUNTIME OPTIONS
We should focus our efforts on testing IP with the following modes:
* -D -X:Frames
* -D -X:FullFrames
* -D -X:Frames -X:SaveAssemblies
* -D
* -X:Frames
*
using both debug and release IP assemblies. Also note that this
needs to be performed for both Python interactive sessions and Python scripts
passed into ipy.exe (e.g., "ipy.exe test_str.py"). Last but
not least, there's the %PYTHONDEBUG% environment variable to consider.
- INTEROPERABILITY
Of primary concern is that IP produces an identical experience as
CPython when given input which utilizes the pdb module. This will be
accomplished via side-by-side testing and verification that the output of
IP is the same as CPython given an identical input.
We will also verify that IP can be debugged from mdbg and the Visual
Studio IDE.
- INTEGRATION TESTING
* Silverlight support of debugging?
* breaking into .NET code from pdb
* breaking into COM from pdb
* breaking into IronRuby from pdb
* breaking into Python code from DLR hosting APIs
* breaking into Python code from IronRuby
- BETA TESTING
The second beta release of IronPython 2.6 will support the debugging feature
in the form of very limited pdb module support with the -X:Frames flag
passed to ipy.exe. As with all beta releases of IP, this will be
released on CodePlex to the general public. We will then use feedback
from the IronPython Community to determine the amount of effort that goes
into fully testing this feature.
- ENVIRONMENT/SYSTEM - GENERAL
The %PYTHONDEBUG% environment variable may have some impact on this feature.
Other than this, the VS IDE will of course need to be installed for testing
the VS debugging experience.
- CONFIGURATION
Need .NET 2.0 Service Pack 1 installed to run IP, and .NET 3.5
Service Pack 1 to build the feature.
- USER INTERFACE
IP provides no user interfaces.
- PERFORMANCE & CAPACITY TESTING
Minimally, we need to run one or more tests in the perf lab under the -X:Frames
and -X:FullFrames test modes. We should also measure the end-to-end run time
of a complete, yet minimal, debugging sample utilizing the pdb module in the
lab. The focus here will simply be on ensuring there are no perf regressions
which are greater than 15% for any given checkin, and also that overall
performance of the debugging feature remains palatable to our userbase.
In terms of capacity testing we should look at:
- if perf gets affected when breaking into deeply nested functions (e.g.,
recursive functions)
- anything else?
- PRIVACY
Does %TMP%\Snippets.debug.scripting.pdb contain sensitive customer data? If
so, does IP disclose the existence of this file?
Is remote debugging supported? If so, what safeguards are in place?
- RELIABILITY
Most end-users are expected to enter into debugging sessions for short
periods of time to diagnose issues with Python code. From this perspective
the so-called "up time" of debugging sessions is less important than the
overall sporadic failure frequency. We can get a good sense of how often
sporadic failures occur by automatically running debugging tests in SNAP
for every IP checkin.
- SCALABILITY
Does debugging still work correctly when stepping into deeply nested (e.g., 1000)
functions?
- STRESS TESTING
- gcstress
- MDA
- assemblies in the GAC
- call pdb functions multiple times and look for memory leaks
- what happens when stepping into unreasonably deeply nested functions?
- VOLUME TESTING
App building exercise or try utilizing pdb from some of our internal Python
tools. It would be interesting to use this from gopackage when exceptions
are encountered.
- INTERNATIONAL ISSUES
As Python is not a localized language and available only in English, we
simply need to confirm that nothing gets localized. This can be done by
running a simple debugbing test against non-English OSes such as Deutsche
Vista.
- ROBUSTNESS
We'll need a dedicated test or two in the stress lab to ensure there are no
memory leaks. Also, we'll need to keep an eye out for sporadic failures of
debugging tests in SNAP.
- ERROR TESTING
What happens when invalid commands are fed to the pdb debugger? Is this
handled identically to CPython?
Is there any conceivable way to fully break debugging without actually
breaking ipy.exe in general? What happens under the mode
"-D -X:SaveAssemblies" when the current user doesn't have write
permissions on %TMP%? What happens if MS.Scripting.Debugging.dll is
removed outright?
- USABILITY
There are three major usability issues with debugging:
* IP offers no visualizer for the VS IDE
* we're largely incompatible with CPython's pdb module
* debugging Python code from .NET apps isn't currently possible
Test will depend on feedback from the IronPython Mailing List and blogs to
determine more usability issues. Resource permitting, we may also do
app building exercises.
The usability goal of this feature is that there are no complaints about
the IP debugging experience from our users.
- ACCESSIBILITY
As this feature provides no new user interfaces to IronPython. and simply
emits output to the stdout stream of command prompts, there should be no
need to invest into accessibility testing.
- USER SCENARIOS
It's anticipated that users of this feature will be limited almost exclusively to
developers. Expert Python users will likely prefer using the pdb
module or debugging their applications directly via the "-i" ipy.exe
option. Existing Microsoft customers with little Python background will
probably be more comfortable with the VS IDE debugging experience. Due to
this we must make sure both experiences are great!
The types of applications being debugged will likely:
* be non-trivial
* consist of many different modules/packages
* make heavy use of the CPython standard library and/or third party Python
packages
* be debugged often during the application development phase
* be debugged infrequently in a production environment when something goes
awry
With these constraints in mind, we should try to debug existing major third
party Python applications and/or add debugging support to our own internal
Python tools.
- BOUNDARIES AND LIMITS
What happens when stepping into a recursive function at the maximum recursion
level?
- OPERATIONAL ISSUES
None
- SPECIAL CODE PROFILING AND OTHER METRICS
Overall block coverage of all assemblies with "ipy" or "IronPython" in their
names should be above 80% and file coverage should stay above 97%. Visual
Studio's mstest will be used to measure code coverage.
-------------------------------------------------------------------------------
TEST ENVIRONMENT
- OPERATING SYSTEMS
32-bit Windows XP
32-bit Windows 2003
64-bit Windows 2003
32-bit Windows Vista
64-bit Windows Vista
32-bit Windows Vista (Deutsche)
- NETWORKS
General Intranet network connection required for SNAP.
May be other special network needs if remote debugging is/becomes a supported scenario.
- HARDWARE
- MACHINES
At least two machines of every OS variety called out above with at least
10 gigs of free hard disk space, 2 gigs of RAM, and a modern CPU
- SOFTWARE
* Visual Studio 2008 Team System Service Pack 1 installed to run the
test suite
* PowerShell 1.0 to run any supporting test scripts
* CPython for side-by-side tests
* mdbg
-------------------------------------------------------------------------------
UNIQUE TESTING CONCERNS FOR SPECIFIC FEATURES
It should be relatively easy to drive pdb or mdbg via the command line in an
automated fashion, but the same cannot be said about the VS IDE debugging
experience. We'll need to reuse the VS team's automation infrastructure and
ensure any machines running these tests in SNAP are never locked at the
username/password screen which would cause the test to fail. Also, should
visualization ever be a supported scenario, we may have no other option than to
test this manually (note - this needs to be investigated).
-------------------------------------------------------------------------------
AREA BREAKDOWN
- CPython's "pdb" module
See debugging.pdb_mod.
- mdbg tool support
See debugging.mdbg_tool.
- VS IDE
See debugging.vs.
- sys module extensions
- sys.settrace
settrace(function)
Set the system's trace function, which allows you to implement a Python
source code debugger in Python. The function is thread-specific; for a
debugger to support multiple threads, it must be registered using
settrace() for each thread being debugged.
Trace functions should have three arguments: frame, event, and arg. frame
is the current stack frame. event is a string: 'call', 'line', 'return',
'exception', 'c_call', 'c_return', or 'c_exception'. arg depends on the
event type.
The trace function is invoked (with event set to 'call') whenever a new
local scope is entered; it should return a reference to a local trace
function to be used that scope, or None if the scope shouldn't be traced.
The local trace function should return a reference to itself (or to another
function for further tracing in that scope), or None to turn off tracing
in that scope.
The events have the following meaning:
'call'
A function is called (or some other code block entered). The global
trace function is called; arg is None; the return value specifies the
local trace function.
'line'
The interpreter is about to execute a new line of code (sometimes
multiple line events on one line exist). The local trace function is
called; arg is None; the return value specifies the new local trace
function.
'return'
A function (or other code block) is about to return. The local trace
function is called; arg is the value that will be returned. The trace
function's return value is ignored.
'exception'
An exception has occurred. The local trace function is called; arg is
a tuple (exception, value, traceback); the return value specifies the
new local trace function.
'c_call'
A C function is about to be called. This may be an extension function
or a builtin. arg is the C function object.
'c_return'
A C function has returned. arg is None.
'c_exception'
A C function has thrown an exception. arg is None.
Note that as an exception is propagated down the chain of callers, an
'exception' event is generated at each level.
- sys.gettrace
Get the trace function as set by settrace().
- sys.call_tracing
call_tracing(func, args) -> object
Call func(*args), while tracing is enabled. The tracing state is
saved, and restored afterwards. This is intended to be called from
a debugger from a checkpoint, to recursively debug some other code.
-------------------------------------------------------------------------------
TEST CASE STRUCTURE
Test cases will be stored directly in the pydoc strings of the functions
implementing the tests. In turn, the modules containing these test case
functions can be found under the "debugging" test package. Generating pydoc for
the "debugging" test package and all sub-modules/sub-packages will generate
this test plan.
-------------------------------------------------------------------------------
SPEC REVIEW ISSUES
There is no IronPython spec for this feature. Any issues observed in the pdb
module, which acts as a spec, and reproducible under CPython should be reported
to http://bugs.python.org.
-------------------------------------------------------------------------------
TEST TOOLS
One of the following:
- reuse CLR's debugging test infrastructure. NOTE: this is only suitable for
mdbg tests
- write our own new side-by-side testing infrastructure entirely in Python
so that we can eventually contribute pdb tests back to CPython
- write our own new side-by-side testing infrastructure in any
language using any framework. http://sourceforge.net/projects/pexpect/ could
be useful as would be PowerShell
- re-use some internal tool that might not be redistributable on CodePlex
which will minimally need to support the following features:
- ability to ignore lines (matching some regular expression) entirely
- ability to substitute strings in lines
- ability to run arbitrary commands (e.g., not limited to "ipy.exe ..." or "mdbg ...")
- ability to match the current output against a reference output stored on disk
- ability to run two or more arbitrary commands and compare their output against each other
As per usual, the plan is to automate all tests using the SNAP checkin system
and VS's mstest tool.
-------------------------------------------------------------------------------
SMOKE TEST (ACCEPTANCE TEST, BUILD VERIFICATION, ETC.)
pdb acceptance test
ipy.exe -c "import pdb"
mdbg
TODO (July 30, 2009)
Visual Studio IDE
TODO (July 30, 2009)
-------------------------------------------------------------------------------
AUTOMATED TESTS
pdb and mdbg tests will be 100% automated in SNAP.
-------------------------------------------------------------------------------
MANUAL TESTS
Setup testing will be manual for now, and we'll eventually move to automated
testing in SNAP using PowerShell support scripts.
Visual Studio IDE testing is also manual for now. Sometime in the
future we'll migrate to using the VS team's automation technologies.
Manual testing will only occur for major IP releases and the
instructions will be documented on the internal IP website under the
"Release Process" wiki.
-------------------------------------------------------------------------------
REGRESSION TESTS
Regression tests will be added to existing test modules under the "debugging"
package before any CodePlex or internal bug dealing with debugging is closed.
As such, the majority of these regressions will be automated and run on every
developer or test checkin. Developers should preferably add a regression test
for every bug fix, and Test will verify the sufficiency of the test, extending
it if necessary.
-------------------------------------------------------------------------------
BUG BASHES
While considered to be extremely useful, we have no current plans for internal
bug bashes on this feature. Quite simply put, we need more IronPython headcount
and/or interest from the rest of Visual Studio Languages to accomplish this.
-------------------------------------------------------------------------------
BUG REPORTING
All bug reports on this feature which do not include information consided to be
Microsoft confidential are to be filed at
http://ironpython.codeplex.com/WorkItem/AdvancedList.aspx under the "Debugging"
component using the bug template found at
http://ironpython.codeplex.com/Wiki/View.aspx?title=IronPython%20Bug%20Template.
Bugs will be triaged on a weekly basis by the entire IronPython Team.
-------------------------------------------------------------------------------
PLAN CONTINGENCIES
Without at least one additional SDET or other test tasks getting dropped, only
phases I and II of this plan are implementable prior to the release of
IP 2.6 RTM.
-------------------------------------------------------------------------------
EXTERNAL DEPENDENCIES
We depend on the DLR for their implementation of Microsoft.Scripting.Debugging.dll
which forms the basis of our support of the CPython pdb module. This plan
assumes that this DLL has been adequately tested.
There are no teams/projects we know about which are taking a dependency on
IP's debugging functionality.
-------------------------------------------------------------------------------
HEADCOUNT REQUIREMENTS
At least two FTE IronPython SDETs will be needed to implement this test plan
fully: one to handle general, day-to-day IP test operations
(i.e., passes/investigations/issues/regressions, etc), and another to work exclusively on the
debugging feature. A loose approximation is that the SDET working full-time
on debugging would need anywhere from one to three months to completely
implement all aspects of this plan. This estimate varies considerably
depending upon how much emphasis is to be given to pdb versus mdbg versus the
VS IDE. Having a second full-time SDET on the debugging feature would have a
not positive effect as the division of labor between testing command-line IP
debugging support and VS IDE support is quite clear.
-------------------------------------------------------------------------------
PRODUCT SUPPORT
IP is not a supported product in the conventional Microsoft sense.
Support for IP is freely given directly by the IP Team via the
IronPython Mailing List.
-------------------------------------------------------------------------------
DROP PROCEDURES
All developer checkins must go through the SNAP checkin system => IronPython
sources are always in a "good" state. Test will build IP from sources
using developer instructions.
-------------------------------------------------------------------------------
RELEASE PROCEDURES
See the IP "Release Process" wiki on the internal IP website.
-------------------------------------------------------------------------------
ALIAS/NEWSGROUPS AND COMMUNICATION CHANNELS
Major changes around IP debugging support should be announced on the
IronPython Mailing List, users@lists.ironpython.com.
-------------------------------------------------------------------------------
REGULAR MEETINGS
- FEATURE TEAM MEETINGS
There is no feature team for debugging.
- PROJECT TEST TEAM MEETINGS
IronPython weekly team meeting. Day and time are subject to change.
We'll triage debugging work items during this meeting and discuss IP
debugging issues as necessary.
- FEATURE TEAM TEST MEETINGS
There is no feature test team for debugging.
-------------------------------------------------------------------------------
DECISION MAKING PROCEDURES
Decisions on this feature will be driven by the following criteria:
- Microsoft business needs
- IronPython Community feature requests
- Previous decisions made by CPython. E.g., the pdb module
- Existing debugging support in Microsoft products such as VS
In the event a conscensus cannot be reached by the team on something, Jim
Hugunin's input should be sought.
Miscellaneous Procedures:
- the gopackage tool is responsible for validating all source pushes to
CodePlex
- at least two IP team members need to sanity check public builds
before they can be released
- full test pass signoff on this feature will be needed for every public, signed
release
- bug triages will be performed with all three software disciplines in attendance
- development design should be reviewed both by the PM and Test disciplines
-------------------------------------------------------------------------------
NOTES
MDBG TEST CASE WRITING TIPS
- use different language structures as the last line in the Python file
- when only one line lives inside a block of code, but the code should not be
stepped through
- correct step-through: no extra step-throughs or lack thereof
-
Package Contents | ||||||
|