From cc8f60d137364e61ed23df32e3dd33fca13f5519 Mon Sep 17 00:00:00 2001 From: Terje Sandstrom Date: Mon, 26 Oct 2026 21:01:50 +0100 Subject: [PATCH 1/7] Add How to Develop and Fix Bugs in NUnit, for the framework and the adapter A common page on reproducing a bug in the issues repositories, running the repros with IssueRunner and testing an unreleased package, with one page for the framework and one for the adapter. The adapter page covers the dump files, checking filters from the command line, and te-dump for Test Explorer. Merge The Teams into Team Practices, so the landing page keeps its size; The-Teams.html redirects there. Co-Authored-By: Claude Opus 5.5 --- cSpell.json | 1 + docs/articles/developer-info/Contributions.md | 3 + .../developer-info/Developing-NUnit.md | 136 ++++++++++++++++++ .../developer-info/Developing-the-Adapter.md | 134 +++++++++++++++++ .../Developing-the-Framework.md | 33 +++++ .../articles/developer-info/Team-Practices.md | 74 +++++++++- docs/articles/developer-info/The-Teams.md | 73 +--------- docs/articles/developer-info/toc.yml | 11 +- .../articles/nunit/release-notes/framework.md | 2 +- docs/index.md | 4 +- 10 files changed, 394 insertions(+), 77 deletions(-) create mode 100644 docs/articles/developer-info/Developing-NUnit.md create mode 100644 docs/articles/developer-info/Developing-the-Adapter.md create mode 100644 docs/articles/developer-info/Developing-the-Framework.md diff --git a/cSpell.json b/cSpell.json index af09856cb..15bafd655 100644 --- a/cSpell.json +++ b/cSpell.json @@ -41,6 +41,7 @@ "Refactorings", "reimplementation", "Remoting", + "repros", "ReSharper", "ruleset", "runtimes", diff --git a/docs/articles/developer-info/Contributions.md b/docs/articles/developer-info/Contributions.md index 7fc7cc10f..6c2bd7185 100644 --- a/docs/articles/developer-info/Contributions.md +++ b/docs/articles/developer-info/Contributions.md @@ -8,6 +8,9 @@ developers. All proposed changes to the code should come in the form of GitHub pull requests. Pull requests are built automatically on the build server and they make it easy to track the origin of any change. +For how to reproduce and fix a bug, and check the fix, see +[How to Develop and Fix Bugs in NUnit](Developing-NUnit.md). + ## Quality When appropriate, pull requests should contain enough unit tests to provide coverage for the changes they introduce. diff --git a/docs/articles/developer-info/Developing-NUnit.md b/docs/articles/developer-info/Developing-NUnit.md new file mode 100644 index 000000000..d6cf01339 --- /dev/null +++ b/docs/articles/developer-info/Developing-NUnit.md @@ -0,0 +1,136 @@ +--- +uid: developingnunit +--- + +# How to Develop and Fix Bugs in NUnit + +This page describes how we work on a bug in the NUnit framework or the NUnit3 VS Adapter: reproduce it, fix it, check +the fix against the repro, and keep the repro as a regression test. The steps for each project are on their own pages: + +* [Developing the Framework](Developing-the-Framework.md) +* [Developing the Adapter](Developing-the-Adapter.md) + +## The workflow + +1. **Reproduce.** Make a repro project for the issue in the project's issues repository, and check that it shows + the bug with the released packages. +2. **Specify.** Write a failing unit test or acceptance test in the project repository. +3. **Fix.** Change the code until the test passes, and run the project's tests. +4. **Check.** Build a package with the fix, point the repro at it, and check that the bug is gone. +5. **Submit.** Raise the pull request, with `Fixed #` at the top of the description, see + [Contributions](Contributions.md). Keep the repro, with the package versions updated. + +## The issues repositories + +Each project has a repository with one repro for each issue that has one: + +| Repository | Repros for | +|---------------------------------------------------------------------------------|----------------------------------| +| [nunit.issues](https://github.com/nunit/nunit.issues) | The NUnit framework | +| [nunit3-vs-adapter.issues](https://github.com/nunit/nunit3-vs-adapter.issues) | The NUnit3 VS Adapter | + +A repro is a folder named `Issue`, for example `Issue1097` for issue 1097, with the smallest project that +shows the bug and builds as it is. Link the folder from the issue. Guidelines for a repro: + +* Use the code from the issue where you can, and add a comment that explains what the bug is. +* Use the current target framework and package versions, unless the bug needs older ones. +* Always reference `Microsoft.NET.Test.Sdk` and `coverlet.collector`, as a real test project does. + +A repro is not required to raise an issue: some bugs are hard to reproduce. When you can make one, it makes the issue +much easier for others to understand and fix. + +### Files in a repro folder + +| File | Purpose | +|-------------------------------------------------|---------------------------------------------------------------------------| +| `*.csproj`, `*.slnx` or `*.sln` | The repro project or solution. | +| `NuGet.config` | Package sources, with a local feed when testing an unreleased package. | +| `.runsettings` | Optional. Settings the bug needs. Visual Studio only picks it up from the solution folder. | +| `run_*.cmd` and `run_*.sh` | Optional. Commands to run instead of plain `dotnet test`, see below. | +| `issue_metadata.json` | The issue's state, milestone and labels, synchronized from GitHub. | +| `issue_initialstate.json` | The target frameworks and package versions the repro was made with. | +| `issue_results.json` | The results of the last run: restore, build and test, with their output. | +| Marker files | Optional, see below. | + +The metadata and result files are written by IssueRunner, see below; you don't need to make them. + +Marker files tell IssueRunner how to treat the folder. A marker can be an empty file or a `.md` file that says why: + +| Marker | Meaning | +|------------------------|-------------------------------------------------------------------| +| `ignore` | Skip the issue in all runs. | +| `explicit` | Run the issue only when it is asked for by number. | +| `wip` | Work in progress, skipped in normal runs. | +| `gui` | The issue shows only in a GUI, such as Visual Studio Test Explorer. | +| `windows` | Windows only, skipped on Linux. | +| `closedasnotplanned` | The issue was closed as not planned. | + +### Custom run scripts + +By default a repro is run with `dotnet test`. When it needs a filter, a runsettings file or only some of its +projects, add a `run_.cmd` and a `run_.sh` with the same commands. Both are needed, since the repros are +run on Windows and Linux. Comments in the first 10 lines can give the expected results: + +```cmd +REM EXPECT_TESTS=1 +REM EXPECT_PASS=1 +dotnet test --filter "FullyQualifiedName~Baz\(1\)" +``` + +The supported keys are `EXPECT_TESTS`, `EXPECT_PASS`, `EXPECT_FAIL` and `EXPECT_SKIP`. The run fails when the +results differ. + +### Running the repros with IssueRunner + +[IssueRunner](https://github.com/nunit/nunit.tools.issuerunner) runs the repros of an issues repository, keeps the +results, and reports regressions. It has a command line tool and a GUI. Build it from its repository, then run it +from the root of the issues repository: + +```cmd +..\nunit.tools.issuerunner\run-tests.cmd --issues 1097,782 --feed Local +``` + +| Option | Meaning | +|----------------------------------------------------------------|----------------------------------------------------------| +| `--issues ` | Run only these issues. | +| `--scope All`, `New`, `NewAndFailed`, `RegressionOnly`, `OpenOnly` | Run a group of issues. `RegressionOnly` runs the closed issues. | +| `--feed Stable`, `Beta`, `Alpha`, `Local` | Where the packages come from. `Local` adds `C:\nuget`. | +| `--rerun-failed` | Run again only what failed last time. | +| `--skip-netfx` | Skip .NET Framework projects, for example on Linux. | + +Other commands keep the repository up to date: `metadata sync-from-github` and `metadata sync-to-folders` update the +`issue_metadata.json` files, `reset` sets the target frameworks and packages back to `issue_initialstate.json`, and +`report generate` writes `TestReport.md`. The IssueRunner README has the details. + +The closed issues are regression tests: run `--scope RegressionOnly` with a new package before a release. + +## Testing an unreleased package + +Both projects get their package version from [MinVer](https://github.com/adamralph/minver): the last tag and the +number of commits since it, for example `6.4.0-alpha.0.25`. Build the package with the project's build script, and copy +it to a local feed, by default `C:\nuget`. A repro then needs a `NuGet.config` with that feed: + +```xml + + + + + + +``` + +Update the repro to the new package with [dotnet-outdated](https://github.com/dotnet-outdated/dotnet-outdated), +instead of editing the version by hand: + +```cmd +dotnet outdated -u -pre Always -inc NUnit3TestAdapter +``` + +Two things to watch out for: + +* **Same version, different code.** Two branches with the same number of commits get the same version. NuGet caches + a package by its version, so a repro can silently use an older package with that version. Give a package a version + of its own with `set MINVERVERSIONOVERRIDE=6.4.0-alpha.0.40` before building and packing, or restore into an + isolated folder with `dotnet restore --packages `. +* **Check what you test.** After a restore, check that the repro uses the package you expect, for example in + `obj\project.assets.json`. diff --git a/docs/articles/developer-info/Developing-the-Adapter.md b/docs/articles/developer-info/Developing-the-Adapter.md new file mode 100644 index 000000000..9452faa02 --- /dev/null +++ b/docs/articles/developer-info/Developing-the-Adapter.md @@ -0,0 +1,134 @@ +# Developing the Adapter + +This page follows the [common workflow](Developing-NUnit.md) for the +[NUnit3 VS Adapter](https://github.com/nunit/nunit3-vs-adapter). The repros are in +[nunit3-vs-adapter.issues](https://github.com/nunit/nunit3-vs-adapter.issues). The adapter connects NUnit to VSTest +and Microsoft.Testing.Platform, so many of its bugs are about finding a test, selecting it with a filter, or showing +it in Visual Studio Test Explorer. This page also shows how to check those. + +## Building and testing + +The repository has a Cake build script, `build.cmd` (or `build.ps1`), in the repository root: + +| Command | What it does | +|-----------------------|-------------------------------------------------------------------------------------------| +| `build` | Builds the solution (default target). | +| `build -t test` | Runs the unit tests on net462 and net8.0, with both VSTest and `dotnet test`. | +| `build -t package` | Creates the NuGet package and zip in `package\`. It does not build first. | +| `build -t acceptance` | Builds, creates the NuGet package and runs the acceptance tests against that package. | + +* Run the acceptance tests with `build -t acceptance`. They test the NuGet package, so running the acceptance test + project directly uses a stale or missing package. +* `build -t package` packs the binaries that are already built. Run `build` first, or the package contains old code. + +Copy the package to the local feed with `copynp.cmd`, giving the version from the file name in `package\`: + +```cmd +.\copynp.cmd 6.4.0-alpha.0.25 +``` + +Then point the repro at it, see [Testing an unreleased package](xref:developingnunit#testing-an-unreleased-package). + +## The adapter's dump files + +Turn on the adapter's dump files in the repro's `.runsettings` while you work on a bug, see +[DumpXmlTestDiscovery and DumpXmlTestResults](xref:tipsandtricks#dumpxmltestdiscovery-and-dumpxmltestresults): + +```xml + + + true + true + + +``` + +The dumps are written to `bin\\\Dump`: + +* `D_.dump` is the discovery: the NUnit XML of every test. +* `E_.dump` is the execution. Its `VS_Input_TestCases` section lists exactly which tests VSTest or Visual + Studio asked the adapter to run, and the `TestFilter` section shows the NUnit filter the adapter made from them. + +These two sections tell you whether a test was lost before it reached the adapter, or inside the adapter. The +execution dump is overwritten by each run; set `DumpNumberedFiles` to keep one for each run. + +## Checking filters from the command line + +Many adapter bugs are about selecting a test: a test is found, but a filter for it runs nothing, or an error. VSTest +can list the fully qualified names and run a filter, without Visual Studio: + +```cmd +dotnet vstest bin\Debug\net10.0\Issue677.dll /ListFullyQualifiedTests /ListTestsTargetPath:fqn.txt +dotnet vstest bin\Debug\net10.0\Issue677.dll "/TestCaseFilter:FullyQualifiedName=TestFixtureSourceError.TestsBroken(Windows Edgium).Test1" +``` + +Run each listed name alone as a filter, both as written and with `\`, `(`, `)`, `&`, `|`, `=`, `!` and `~` escaped +with a `\`, and compare the number of tests run with the number listed. A small script that loops over `fqn.txt` does +this for a whole repro. `dotnet test --filter` uses the same filter syntax. + +## Checking Test Explorer with te-dump + +Some bugs show only in Visual Studio Test Explorer: a test in the wrong place in the tree, an extra node that never +runs, or a result that is not shown. `te-dump` +([NUnit.Tools.TestExplorerDump](https://github.com/nunit/NUnit.Tools.TestExplorerDump)) prints the Test Explorer +tree as text, so you, a script or an AI assistant can see exactly what Test Explorer shows, and compare it before and +after a fix. It can also drive Visual Studio: start it on a solution, rebuild, run all or some tests, and close it. It +reads Test Explorer through Windows UI Automation and drives Visual Studio through EnvDTE; nothing is installed into +Visual Studio. It runs on Windows only. Its README has the install steps and all options. + +### A full round trip + +Clean the repro first, so no results or test lists from earlier runs are left: delete the `.vs`, `bin` and `obj` +folders, for example with [vsclean](https://github.com/OsirisTerje/VSClean) `-d`. Then run a session in the repro +folder: + +```cmd +cd Issue1408 +te-dump session +``` + +`session` starts a new Visual Studio on the solution, rebuilds it, waits for test discovery, dumps the tree, runs all +tests, dumps the tree again and closes Visual Studio, also when a step fails. The trees are printed, and written to a +`te-dump` folder next to the solution, for example `te-dump\Issue1408-20261026-194618-discovered.txt` and +`...-after-run.txt`. Add `te-dump/` to `.gitignore`. + +Each line is `Kind: Name [Outcome] (test count)`; a test has no kind and no count: + +```text +# after run +Project: Issue1408 [Passed] (3) + Namespace: NUnitTestExplorerFailure [Passed] (3) + Class: Tests [Passed] (3) + Test Group: Demonstration [Passed] (2) + Demonstration([(0, 0)]) [Passed] + Demonstration([(1, 2), (3, 4)]) [Passed] + Demonstration [Not Run] +``` + +Here the two tests ran, and `Demonstration [Not Run]` is an extra node. It comes from Visual Studio's own source-based +test discovery, which finds the method in the source code but cannot match it to the tests the adapter reports. + +### Running single tests + +Test Explorer often runs only some tests: a single test, or one node of the tree. Bugs in that path do not show with +Run All. Keep Visual Studio open and run the tests one at a time: + +```cmd +te-dump start +te-dump --pid 73760 --expand-all +te-dump run --pid 73760 --filter SimpleTest +te-dump --pid 73760 --expand-all +te-dump close --pid 73760 +``` + +`start` prints the process id to use with the other commands. `--filter` is Test Explorer's search text, for example +`SimpleTest`, `FullName:"Ns.Class.Method"` or `Class:"Class1"`. `te-dump list` shows the Visual Studio instances that +have Test Explorer open. + +Do not run a session on a solution that is already open in another Visual Studio: both use the same `.vs` folder. + +### Reading the results + +When Test Explorer shows a test as Not Run after it has run, look at the adapter's execution dump from that run. If the +test is in `VS_Input_TestCases` and has a result in the dump, the adapter ran it and reported it, and Visual Studio did +not match the result to the test it shows. diff --git a/docs/articles/developer-info/Developing-the-Framework.md b/docs/articles/developer-info/Developing-the-Framework.md new file mode 100644 index 000000000..17418a33c --- /dev/null +++ b/docs/articles/developer-info/Developing-the-Framework.md @@ -0,0 +1,33 @@ +# Developing the Framework + +This page follows the [common workflow](Developing-NUnit.md) for the [NUnit framework](https://github.com/nunit/nunit). +The repros are in [nunit.issues](https://github.com/nunit/nunit.issues). + +## Building and testing + +The repository has one solution, `nunit.slnx`, and a Cake build script: `build.cmd` (Windows), `build.ps1` +(PowerShell) or `build.sh` (Linux and macOS). They take the same arguments: + +| Command | What it does | +|------------------------------|-----------------------------------------------------------------------| +| `build` | Restores and builds everything (default target). | +| `build --target=Rebuild` | Cleans the output and builds everything. | +| `build --target=Test` | Runs all tests, on all target frameworks. | +| `build --target=Package` | Creates the packages in `package\`. It does not build first. | +| `build --showdescription` | Lists all targets. | + +The configuration is `Release` by default; use `--configuration=Debug` to change it. The tests to run are in the +`nunit.framework.tests-*`, `nunit.framework.legacy.tests-*` and `nunitlite.tests-*` projects. The other test projects +contain tests that fail on purpose, for the integration tests. [BUILDING.md](https://github.com/nunit/nunit/blob/main/BUILDING.md) +in the repository has the details. + +## Fixing a bug + +1. Make the repro in [nunit.issues](https://github.com/nunit/nunit.issues): a folder `Issue` with a + `readme.md` that links to the issue, and a project that builds and runs. Link the folder from the issue. +2. Add a failing test to the framework's test projects that shows the bug, then fix the code. Run + `build --target=Test`: the framework targets several frameworks, and a fix can behave differently on each. +3. Run `build` and `build --target=Package`, and copy the `NUnit` package from `package\` to your local feed. +4. Update the repro to that package, see [Testing an unreleased package](xref:developingnunit#testing-an-unreleased-package), + and check that the bug is gone. The repro runs through the NUnit3 VS Adapter, so a bug that only shows in Visual + Studio can be checked there too, see [Checking Test Explorer with te-dump](Developing-the-Adapter.md#checking-test-explorer-with-te-dump). diff --git a/docs/articles/developer-info/Team-Practices.md b/docs/articles/developer-info/Team-Practices.md index 75264f49d..1f6950bfc 100644 --- a/docs/articles/developer-info/Team-Practices.md +++ b/docs/articles/developer-info/Team-Practices.md @@ -2,7 +2,78 @@ uid: teampractices --- -# Team Practices +# Teams and Practices + +## The Teams in the NUnit Project + +The NUnit project is made up of several teams, each with its own focus and responsibilities, and with a set of team +members. + +The team members are a bunch of dedicated people from all over the globe, who are passionate about NUnit and want to +make it better. They are all volunteers, and they do this in their spare time. + +### The current teams + +#### Core Team + +This team has overall responsibility for the NUnit organization. It defines the vision for NUnit, decides what projects +should be included, appoints leads for other teams and maintainers for projects, sets standards and ensures we follow +them. + +Team Lead: [Terje Sandstrom](https://github.com/osiristerje) + +#### Framework Team + +This team maintains the NUnit framework, which is the assembly referenced by user tests, as well as NUnitLite. + +Team Lead: [Rob Prouse](https://github.com/rprouse) + +#### Engine Team + +The Engine Team maintains the NUnit3-Console runner, the Engine, and a number of extensions. + +Team Lead: [Charlie Poole](https://github.com/charliepoole) + +#### VS Extensions Team + +NUnit's extensions to the Visual Studio IDE and dotnet. + +Team Lead: [Terje Sandstrom](https://github.com/osiristerje) + +#### Analyzer Team + +The team that works on the NUnit Analyzers Project. + +Team Lead: [Mikkel Nylander Bundgaard](https://github.com/mikkelbu) + +#### Documentation Team + +The team that works on the documentation for NUnit. + +Team Lead: [Sean Killeen](https://github.com/seankilleen) + +### Membership matrix + +#### Current team memberships, listed in a kind of chronological order + +| Name | Core Team | Framework | Engine | VS Extensions | Analyzer | Documentation | +| -------------------------------------------------------------------- | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | +| [Charlie Poole](https://github.com/charliepoole) :us: | :heavy_check_mark: | | :heavy_check_mark: | | | | +| [Rob Prouse](https://github.com/rprouse) :canada: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | +| [Terje Sandstrom](https://github.com/osiristerje) :norway: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | +| [Mikkel Nylander Bundgaard](https://github.com/mikkelbu) :denmark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | | +| [Sean Killeen](https://github.com/SeanKilleen) :us: | :heavy_check_mark: | | | | :heavy_check_mark: | :heavy_check_mark: | +| [Steven Weerdenburg](https://github.com/stevenaw) :canada: | :heavy_check_mark: | :heavy_check_mark: | | | | | +| [Ihor Sychevskyi](https://github.com/Arhell) :ukraine: | | | | | | :heavy_check_mark: | +| [Manfred Brands](https://github.com/manfred-brands) :australia: | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | | +| [Neil Colvin](https://github.com/oznetmaster) :australia: :scotland: | | | :heavy_check_mark: | | | | + +#### Former members and honorable mentions + +| Name | NUnit membership period | +| ----------------------------------------------------- | ----------------------- | +| [Chris Maddock](https://github.com/ChrisMaddock) :gb: | 2016 - 2022 | +| [Joseph Musser](https://github.com/jnm2) :us: | 2017 - 2026 | ## How We Work As a Team @@ -10,6 +81,7 @@ uid: teampractices | ----------------------------------- | -------------------------------------------------------------------- | | [Issue Tracking](Issue-Tracking.md) | How we track bugs, enhancements and features on GitHub. | | [Contributions](Contributions.md) | Explains how to review and merge code submitted by other developers. | +| [How to Develop and Fix Bugs](Developing-NUnit.md) | How to reproduce a bug, fix it and check the fix. | ## Technical Practices diff --git a/docs/articles/developer-info/The-Teams.md b/docs/articles/developer-info/The-Teams.md index 4a94825fd..c5c096a5b 100644 --- a/docs/articles/developer-info/The-Teams.md +++ b/docs/articles/developer-info/The-Teams.md @@ -1,70 +1,3 @@ -# The Teams in the NUnit Project - -The NUnit project is made up of several teams, each with its own focus and responsibilities, and with a set of team -members. - -The team members are a bunch of dedicated people from all over the globe, who are passionate about NUnit and want to -make it better. They are all volunteers, and they do this in their spare time. - -## The current teams - -### Core Team - -This team has overall responsibility for the NUnit organization. It defines the vision for NUnit, decides what projects -should be included, appoints leads for other teams and maintainers for projects, sets standards and ensures we follow -them. - -Team Lead: [Terje Sandstrom](https://github.com/osiristerje) - -### Framework Team - -This team maintains the NUnit framework, which is the assembly referenced by user tests, as well as NUnitLite. - -Team Lead: [Rob Prouse](https://github.com/rprouse) - -### Engine Team - -The Engine Team maintains the NUnit3-Console runner, the Engine, and a number of extensions. - -Team Lead: [Charlie Poole](https://github.com/charliepoole) - -### VS Extensions Team - -NUnit's extensions to the Visual Studio IDE and dotnet. - -Team Lead: [Terje Sandstrom](https://github.com/osiristerje) - -### Analyzer Team - -The team that works on the NUnit Analyzers Project. - -Team Lead: [Mikkel Nylander Bundgaard](https://github.com/mikkelbu) - -### Documentation Team - -The team that works on the documentation for NUnit. - -Team Lead: [Sean Killeen](https://github.com/seankilleen) - -## Membership matrix - -### Current team memberships, listed in a kind of chronological order - -| Name | Core Team | Framework | Engine | VS Extensions | Analyzer | Documentation | -| -------------------------------------------------------------------- | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | -| [Charlie Poole](https://github.com/charliepoole) :us: | :heavy_check_mark: | | :heavy_check_mark: | | | | -| [Rob Prouse](https://github.com/rprouse) :canada: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | -| [Terje Sandstrom](https://github.com/osiristerje) :norway: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | -| [Mikkel Nylander Bundgaard](https://github.com/mikkelbu) :denmark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | | -| [Sean Killeen](https://github.com/SeanKilleen) :us: | :heavy_check_mark: | | | | :heavy_check_mark: | :heavy_check_mark: | -| [Steven Weerdenburg](https://github.com/stevenaw) :canada: | :heavy_check_mark: | :heavy_check_mark: | | | | | -| [Ihor Sychevskyi](https://github.com/Arhell) :ukraine: | | | | | | :heavy_check_mark: | -| [Manfred Brands](https://github.com/manfred-brands) :australia: | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | | -| [Neil Colvin](https://github.com/oznetmaster) :australia: :scotland: | | | :heavy_check_mark: | | | | - -### Former members and honorable mentions - -| Name | NUnit membership period | -| ----------------------------------------------------- | ----------------------- | -| [Chris Maddock](https://github.com/ChrisMaddock) :gb: | 2016 - 2022 | -| [Joseph Musser](https://github.com/jnm2) :us: | 2017 - 2026 | +--- +redirect_url: ./Team-Practices.html +--- diff --git a/docs/articles/developer-info/toc.yml b/docs/articles/developer-info/toc.yml index f578cc680..4161a4cab 100644 --- a/docs/articles/developer-info/toc.yml +++ b/docs/articles/developer-info/toc.yml @@ -1,7 +1,12 @@ -- name: The Teams - href: The-Teams.md -- name: Team Practices +- name: Teams and Practices href: Team-Practices.md +- name: How to Develop and Fix Bugs + href: Developing-NUnit.md + items: + - name: Developing the Framework + href: Developing-the-Framework.md + - name: Developing the Adapter + href: Developing-the-Adapter.md - name: Specifications topicUid: specifications - name: Best Practices for XML Documentation diff --git a/docs/articles/nunit/release-notes/framework.md b/docs/articles/nunit/release-notes/framework.md index 0ae87ae4d..04b08d0cb 100644 --- a/docs/articles/nunit/release-notes/framework.md +++ b/docs/articles/nunit/release-notes/framework.md @@ -2328,4 +2328,4 @@ those versions of the framework, please contact the NUnit team. * Release Notes for [NUnit 2.4 through 2.4.8](https://nunit.org/?p=releaseNotes&r=2.4.8) * Release Notes for [NUnit 2.0 through 2.2.10](https://nunit.org/?p=releaseNotes&r=2.2.10) -[TeamLink]: https://docs.nunit.org/articles/developer-info/The-Teams.html#current-team-memberships-listed-in-a-kind-of-chronological-order +[TeamLink]: https://docs.nunit.org/articles/developer-info/Team-Practices.html#current-team-memberships-listed-in-a-kind-of-chronological-order diff --git a/docs/index.md b/docs/index.md index c485717e9..4fae341e0 100644 --- a/docs/index.md +++ b/docs/index.md @@ -152,8 +152,8 @@ public class CalculatorTests

Developer info

For contributors to the NUnit projects.

    -
  • The teams
    • -
  • Team practices
    • +
  • Teams and practices
    • +
  • How to develop and fix bugs in NUnit
  • Coding standards
  • XML documentation
  • Contributions
    • From ec1bd22bfcfac680ee72e5909820cd67daf7e1a5 Mon Sep 17 00:00:00 2001 From: Terje Sandstrom Date: Mon, 26 Oct 2026 21:03:01 +0100 Subject: [PATCH 2/7] Group Developer Info into sections, and drop the Team Practices page Team Practices only linked to pages the Developer Info toc and the Packaging page already list. The toc now has the sections Working Together, Code and Packaging, The Teams is the team page again, and Team-Practices.html redirects to it. Co-Authored-By: Claude Opus 5.5 --- .../articles/developer-info/Team-Practices.md | 97 +------------------ docs/articles/developer-info/The-Teams.md | 73 +++++++++++++- docs/articles/developer-info/toc.yml | 57 ++++++----- .../articles/nunit/release-notes/framework.md | 2 +- docs/classic.md | 3 +- docs/index.md | 4 +- 6 files changed, 109 insertions(+), 127 deletions(-) diff --git a/docs/articles/developer-info/Team-Practices.md b/docs/articles/developer-info/Team-Practices.md index 1f6950bfc..9aac33612 100644 --- a/docs/articles/developer-info/Team-Practices.md +++ b/docs/articles/developer-info/Team-Practices.md @@ -1,98 +1,3 @@ --- -uid: teampractices +redirect_url: ./The-Teams.html --- - -# Teams and Practices - -## The Teams in the NUnit Project - -The NUnit project is made up of several teams, each with its own focus and responsibilities, and with a set of team -members. - -The team members are a bunch of dedicated people from all over the globe, who are passionate about NUnit and want to -make it better. They are all volunteers, and they do this in their spare time. - -### The current teams - -#### Core Team - -This team has overall responsibility for the NUnit organization. It defines the vision for NUnit, decides what projects -should be included, appoints leads for other teams and maintainers for projects, sets standards and ensures we follow -them. - -Team Lead: [Terje Sandstrom](https://github.com/osiristerje) - -#### Framework Team - -This team maintains the NUnit framework, which is the assembly referenced by user tests, as well as NUnitLite. - -Team Lead: [Rob Prouse](https://github.com/rprouse) - -#### Engine Team - -The Engine Team maintains the NUnit3-Console runner, the Engine, and a number of extensions. - -Team Lead: [Charlie Poole](https://github.com/charliepoole) - -#### VS Extensions Team - -NUnit's extensions to the Visual Studio IDE and dotnet. - -Team Lead: [Terje Sandstrom](https://github.com/osiristerje) - -#### Analyzer Team - -The team that works on the NUnit Analyzers Project. - -Team Lead: [Mikkel Nylander Bundgaard](https://github.com/mikkelbu) - -#### Documentation Team - -The team that works on the documentation for NUnit. - -Team Lead: [Sean Killeen](https://github.com/seankilleen) - -### Membership matrix - -#### Current team memberships, listed in a kind of chronological order - -| Name | Core Team | Framework | Engine | VS Extensions | Analyzer | Documentation | -| -------------------------------------------------------------------- | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | -| [Charlie Poole](https://github.com/charliepoole) :us: | :heavy_check_mark: | | :heavy_check_mark: | | | | -| [Rob Prouse](https://github.com/rprouse) :canada: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | -| [Terje Sandstrom](https://github.com/osiristerje) :norway: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | -| [Mikkel Nylander Bundgaard](https://github.com/mikkelbu) :denmark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | | -| [Sean Killeen](https://github.com/SeanKilleen) :us: | :heavy_check_mark: | | | | :heavy_check_mark: | :heavy_check_mark: | -| [Steven Weerdenburg](https://github.com/stevenaw) :canada: | :heavy_check_mark: | :heavy_check_mark: | | | | | -| [Ihor Sychevskyi](https://github.com/Arhell) :ukraine: | | | | | | :heavy_check_mark: | -| [Manfred Brands](https://github.com/manfred-brands) :australia: | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | | -| [Neil Colvin](https://github.com/oznetmaster) :australia: :scotland: | | | :heavy_check_mark: | | | | - -#### Former members and honorable mentions - -| Name | NUnit membership period | -| ----------------------------------------------------- | ----------------------- | -| [Chris Maddock](https://github.com/ChrisMaddock) :gb: | 2016 - 2022 | -| [Joseph Musser](https://github.com/jnm2) :us: | 2017 - 2026 | - -## How We Work As a Team - -| | | -| ----------------------------------- | -------------------------------------------------------------------- | -| [Issue Tracking](Issue-Tracking.md) | How we track bugs, enhancements and features on GitHub. | -| [Contributions](Contributions.md) | Explains how to review and merge code submitted by other developers. | -| [How to Develop and Fix Bugs](Developing-NUnit.md) | How to reproduce a bug, fix it and check the fix. | - -## Technical Practices - -| | | -| ------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | -| Building and testing | See BUILDING.md in each repository. | -| [Coding Standards](Coding-Standards.md) | Describes coding standards used in NUnit. | -| [Best practices for XML documentation](Best-practices-for-XML-documentation.md) | Guidelines to consider when dealing with public APIs. | -| [Packaging the Framework](Packaging-the-Framework.md) | How to create the packages for a new NUnit Framework release. | -| [Packaging the Console and Engine](Packaging-the-Console-and-Engine.md) | How to create the packages for a new NUnit Console and Engine release. | -| [Packaging Extensions](Packaging-Extensions.md) | How to create the packages and release NUnit Extensions. | -| [Packaging the Installer](Packaging-the-Installer.md) | How to create and release the NUnit MSI Installer. | -| [Packaging the V2 Adapter](Packaging-the-V2-Adapter.md) | How to create the packages for a new NUnit 2 VS Adapter release. | -| [Packaging the V3 Adapter](Packaging-the-V3-and-V4-Adapter.md) | How to create the packages for a new NUnit 3 VS Adapter release. | diff --git a/docs/articles/developer-info/The-Teams.md b/docs/articles/developer-info/The-Teams.md index c5c096a5b..0251c746c 100644 --- a/docs/articles/developer-info/The-Teams.md +++ b/docs/articles/developer-info/The-Teams.md @@ -1,3 +1,74 @@ --- -redirect_url: ./Team-Practices.html +uid: theteams --- + +# The Teams in the NUnit Project + +The NUnit project is made up of several teams, each with its own focus and responsibilities, and with a set of team +members. + +The team members are a bunch of dedicated people from all over the globe, who are passionate about NUnit and want to +make it better. They are all volunteers, and they do this in their spare time. + +## The current teams + +### Core Team + +This team has overall responsibility for the NUnit organization. It defines the vision for NUnit, decides what projects +should be included, appoints leads for other teams and maintainers for projects, sets standards and ensures we follow +them. + +Team Lead: [Terje Sandstrom](https://github.com/osiristerje) + +### Framework Team + +This team maintains the NUnit framework, which is the assembly referenced by user tests, as well as NUnitLite. + +Team Lead: [Rob Prouse](https://github.com/rprouse) + +### Engine Team + +The Engine Team maintains the NUnit3-Console runner, the Engine, and a number of extensions. + +Team Lead: [Charlie Poole](https://github.com/charliepoole) + +### VS Extensions Team + +NUnit's extensions to the Visual Studio IDE and dotnet. + +Team Lead: [Terje Sandstrom](https://github.com/osiristerje) + +### Analyzer Team + +The team that works on the NUnit Analyzers Project. + +Team Lead: [Mikkel Nylander Bundgaard](https://github.com/mikkelbu) + +### Documentation Team + +The team that works on the documentation for NUnit. + +Team Lead: [Sean Killeen](https://github.com/seankilleen) + +## Membership matrix + +### Current team memberships, listed in a kind of chronological order + +| Name | Core Team | Framework | Engine | VS Extensions | Analyzer | Documentation | +| -------------------------------------------------------------------- | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | +| [Charlie Poole](https://github.com/charliepoole) :us: | :heavy_check_mark: | | :heavy_check_mark: | | | | +| [Rob Prouse](https://github.com/rprouse) :canada: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | +| [Terje Sandstrom](https://github.com/osiristerje) :norway: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | | +| [Mikkel Nylander Bundgaard](https://github.com/mikkelbu) :denmark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | :heavy_check_mark: | | +| [Sean Killeen](https://github.com/SeanKilleen) :us: | :heavy_check_mark: | | | | :heavy_check_mark: | :heavy_check_mark: | +| [Steven Weerdenburg](https://github.com/stevenaw) :canada: | :heavy_check_mark: | :heavy_check_mark: | | | | | +| [Ihor Sychevskyi](https://github.com/Arhell) :ukraine: | | | | | | :heavy_check_mark: | +| [Manfred Brands](https://github.com/manfred-brands) :australia: | :heavy_check_mark: | :heavy_check_mark: | | | :heavy_check_mark: | | +| [Neil Colvin](https://github.com/oznetmaster) :australia: :scotland: | | | :heavy_check_mark: | | | | + +### Former members and honorable mentions + +| Name | NUnit membership period | +| ----------------------------------------------------- | ----------------------- | +| [Chris Maddock](https://github.com/ChrisMaddock) :gb: | 2016 - 2022 | +| [Joseph Musser](https://github.com/jnm2) :us: | 2017 - 2026 | diff --git a/docs/articles/developer-info/toc.yml b/docs/articles/developer-info/toc.yml index 4161a4cab..56aa84146 100644 --- a/docs/articles/developer-info/toc.yml +++ b/docs/articles/developer-info/toc.yml @@ -1,29 +1,34 @@ -- name: Teams and Practices - href: Team-Practices.md -- name: How to Develop and Fix Bugs - href: Developing-NUnit.md +- name: The Teams + href: The-Teams.md +- name: Working Together items: - - name: Developing the Framework - href: Developing-the-Framework.md - - name: Developing the Adapter - href: Developing-the-Adapter.md -- name: Specifications - topicUid: specifications -- name: Best Practices for XML Documentation - href: Best-practices-for-XML-documentation.md -- name: Coding Standards - href: Coding-Standards.md -- name: Contributions - href: Contributions.md -- name: Issue Tracking - href: Issue-Tracking.md + - name: Issue Tracking + href: Issue-Tracking.md + - name: Contributions + href: Contributions.md + - name: How to Develop and Fix Bugs + href: Developing-NUnit.md + items: + - name: Developing the Framework + href: Developing-the-Framework.md + - name: Developing the Adapter + href: Developing-the-Adapter.md +- name: Code + items: + - name: Coding Standards + href: Coding-Standards.md + - name: Best Practices for XML Documentation + href: Best-practices-for-XML-documentation.md + - name: Specifications + topicUid: specifications - name: Packaging href: Packaging.md -- name: Packaging Extensions - href: Packaging-Extensions.md -- name: Packaging the Console and Engine - href: Packaging-the-Console-and-Engine.md -- name: Packaging the Framework - href: Packaging-the-Framework.md -- name: Packaging the V3/V4 Adapter - href: Packaging-the-V3-and-V4-Adapter.md + items: + - name: Packaging the Framework + href: Packaging-the-Framework.md + - name: Packaging the Console and Engine + href: Packaging-the-Console-and-Engine.md + - name: Packaging the V3/V4 Adapter + href: Packaging-the-V3-and-V4-Adapter.md + - name: Packaging Extensions + href: Packaging-Extensions.md diff --git a/docs/articles/nunit/release-notes/framework.md b/docs/articles/nunit/release-notes/framework.md index 04b08d0cb..0ae87ae4d 100644 --- a/docs/articles/nunit/release-notes/framework.md +++ b/docs/articles/nunit/release-notes/framework.md @@ -2328,4 +2328,4 @@ those versions of the framework, please contact the NUnit team. * Release Notes for [NUnit 2.4 through 2.4.8](https://nunit.org/?p=releaseNotes&r=2.4.8) * Release Notes for [NUnit 2.0 through 2.2.10](https://nunit.org/?p=releaseNotes&r=2.2.10) -[TeamLink]: https://docs.nunit.org/articles/developer-info/Team-Practices.html#current-team-memberships-listed-in-a-kind-of-chronological-order +[TeamLink]: https://docs.nunit.org/articles/developer-info/The-Teams.html#current-team-memberships-listed-in-a-kind-of-chronological-order diff --git a/docs/classic.md b/docs/classic.md index 0c2ff9448..95ed5d0b0 100644 --- a/docs/classic.md +++ b/docs/classic.md @@ -20,5 +20,6 @@ working on NUnit or wishing to do so. ## Developer Documentation -* [Team practices](xref:teampractices) describe how NUnit works and how our teams work. +* [The teams](xref:theteams) describe how the NUnit project is organized, and + [How to develop and fix bugs](xref:developingnunit) describes how we work on a bug. * [Specifications](xref:specifications) are descriptions of features we plan to add. diff --git a/docs/index.md b/docs/index.md index 4fae341e0..7c1d3ab3e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -149,10 +149,10 @@ public class CalculatorTests
    -

    Developer info

    +

    Developer info

    For contributors to the NUnit projects.

      -
    • Teams and practices
      • +
    • The teams
    • How to develop and fix bugs in NUnit
    • Coding standards
    • XML documentation
      • From 37fddf45bc0b5aa072c15d5771e18e8e05b7c6b0 Mon Sep 17 00:00:00 2001 From: Terje Sandstrom Date: Mon, 26 Oct 2026 21:08:54 +0100 Subject: [PATCH 3/7] Describe what IssueRunner does with markers and EXPECT comments, update NUnit for framework fixes, and read the execution dump carefully Co-Authored-By: Claude Opus 5.5 --- .../developer-info/Developing-NUnit.md | 32 ++++++++++++------- .../developer-info/Developing-the-Adapter.md | 11 +++++-- 2 files changed, 29 insertions(+), 14 deletions(-) diff --git a/docs/articles/developer-info/Developing-NUnit.md b/docs/articles/developer-info/Developing-NUnit.md index d6cf01339..a1f681d25 100644 --- a/docs/articles/developer-info/Developing-NUnit.md +++ b/docs/articles/developer-info/Developing-NUnit.md @@ -56,14 +56,17 @@ The metadata and result files are written by IssueRunner, see below; you don't n Marker files tell IssueRunner how to treat the folder. A marker can be an empty file or a `.md` file that says why: -| Marker | Meaning | -|------------------------|-------------------------------------------------------------------| -| `ignore` | Skip the issue in all runs. | -| `explicit` | Run the issue only when it is asked for by number. | -| `wip` | Work in progress, skipped in normal runs. | -| `gui` | The issue shows only in a GUI, such as Visual Studio Test Explorer. | -| `windows` | Windows only, skipped on Linux. | -| `closedasnotplanned` | The issue was closed as not planned. | +| Marker | Why it is there | IssueRunner | +|------------------------|-------------------------------------------------------------------|-----------------------------------| +| `ignore` | The repro should not be run. | Skips it. | +| `explicit` | The repro must be run by hand, for example one project at a time. | Skips it. | +| `wip` | Work in progress. | Skips it. | +| `gui` | The bug shows only in a GUI, such as Visual Studio Test Explorer. | Skips it. | +| `closedasnotplanned` | The issue was closed as not planned. | Skips it. | +| `windows` | The repro runs on Windows only. | Skips it with `--skip-netfx`. | + +IssueRunner skips a marked folder also when you ask for the issue by number. To run it with IssueRunner, remove the +marker; or run the repro yourself, with `dotnet test` or its run scripts. ### Custom run scripts @@ -77,8 +80,10 @@ REM EXPECT_PASS=1 dotnet test --filter "FullyQualifiedName~Baz\(1\)" ``` -The supported keys are `EXPECT_TESTS`, `EXPECT_PASS`, `EXPECT_FAIL` and `EXPECT_SKIP`. The run fails when the -results differ. +The keys are `EXPECT_TESTS`, `EXPECT_PASS`, `EXPECT_FAIL` and `EXPECT_SKIP`. They document what the script should +give, but IssueRunner does not check them yet: it only looks at the script's exit code. A script that must catch a +wrong number of tests, for example a filter that runs nothing, has to check the count itself and exit with a non-zero +code. ### Running the repros with IssueRunner @@ -120,10 +125,15 @@ it to a local feed, by default `C:\nuget`. A repro then needs a `NuGet.config` w ``` Update the repro to the new package with [dotnet-outdated](https://github.com/dotnet-outdated/dotnet-outdated), -instead of editing the version by hand: +instead of editing the version by hand. `-inc` updates the packages whose names contain the text, and `-exc` leaves +out the packages whose names contain the text: ```cmd +REM An adapter fix dotnet outdated -u -pre Always -inc NUnit3TestAdapter + +REM A framework fix: the NUnit package, not the adapter or the analyzers +dotnet outdated -u -pre Always -inc NUnit -exc NUnit3TestAdapter -exc NUnit.Analyzers ``` Two things to watch out for: diff --git a/docs/articles/developer-info/Developing-the-Adapter.md b/docs/articles/developer-info/Developing-the-Adapter.md index 9452faa02..52906da1f 100644 --- a/docs/articles/developer-info/Developing-the-Adapter.md +++ b/docs/articles/developer-info/Developing-the-Adapter.md @@ -129,6 +129,11 @@ Do not run a session on a solution that is already open in another Visual Studio ### Reading the results -When Test Explorer shows a test as Not Run after it has run, look at the adapter's execution dump from that run. If the -test is in `VS_Input_TestCases` and has a result in the dump, the adapter ran it and reported it, and Visual Studio did -not match the result to the test it shows. +When Test Explorer shows a test as Not Run after it has run, look at the adapter's execution dump from that run: + +* The test is not in `VS_Input_TestCases`: Visual Studio did not ask the adapter to run it. +* The test is in `VS_Input_TestCases` but has no result in the dump: the adapter's filter did not select it, see the + `TestFilter` section. +* The test has a result in the dump: NUnit ran it, and the adapter received the result. The dump records it before the + adapter converts the result and reports it to VSTest, so a failure in that step is still possible; look for errors + in the Tests output pane. With no errors there, Visual Studio did not match the reported result to the test it shows. From 351ae2dda26552fdc224faae3abc14caa39e698a Mon Sep 17 00:00:00 2001 From: Terje Sandstrom Date: Mon, 26 Oct 2026 21:20:12 +0100 Subject: [PATCH 4/7] Lead with using a single repro; IssueRunner is optional, markers and EXPECT comments are information Co-Authored-By: Claude Opus 5.5 --- .../developer-info/Developing-NUnit.md | 81 ++++++++++--------- 1 file changed, 43 insertions(+), 38 deletions(-) diff --git a/docs/articles/developer-info/Developing-NUnit.md b/docs/articles/developer-info/Developing-NUnit.md index a1f681d25..90de23835 100644 --- a/docs/articles/developer-info/Developing-NUnit.md +++ b/docs/articles/developer-info/Developing-NUnit.md @@ -39,6 +39,21 @@ shows the bug and builds as it is. Link the folder from the issue. Guidelines fo A repro is not required to raise an issue: some bugs are hard to reproduce. When you can make one, it makes the issue much easier for others to understand and fix. +### Using a repro + +Work with one repro at a time, on its own: + +```cmd +cd Issue1097 +dotnet test +``` + +* Run its `run_*.cmd` or `run_*.sh` scripts instead when it has them, see below. +* Open the solution in Visual Studio when the bug is about Visual Studio. A `.runsettings` in the folder is picked up + automatically. For Test Explorer bugs, see + [Checking Test Explorer with te-dump](Developing-the-Adapter.md#checking-test-explorer-with-te-dump). +* Point it at an unreleased package to check a fix, see [Testing an unreleased package](#testing-an-unreleased-package). + ### Files in a repro folder | File | Purpose | @@ -47,32 +62,28 @@ much easier for others to understand and fix. | `NuGet.config` | Package sources, with a local feed when testing an unreleased package. | | `.runsettings` | Optional. Settings the bug needs. Visual Studio only picks it up from the solution folder. | | `run_*.cmd` and `run_*.sh` | Optional. Commands to run instead of plain `dotnet test`, see below. | -| `issue_metadata.json` | The issue's state, milestone and labels, synchronized from GitHub. | -| `issue_initialstate.json` | The target frameworks and package versions the repro was made with. | -| `issue_results.json` | The results of the last run: restore, build and test, with their output. | -| Marker files | Optional, see below. | +| Marker file | Optional. Says what kind of repro it is, see below. | +| `issue_metadata.json`, `issue_initialstate.json`, `issue_results.json` | Written by IssueRunner, see below. | -The metadata and result files are written by IssueRunner, see below; you don't need to make them. +The `issue_*.json` files are information, and you don't need to make them: the issue's state, milestone and labels +from GitHub, the target frameworks and packages the repro was made with, and the results of the last IssueRunner run. -Marker files tell IssueRunner how to treat the folder. A marker can be an empty file or a `.md` file that says why: +A marker file says what kind of repro it is. It can be an empty file, or a `.md` file that says why: -| Marker | Why it is there | IssueRunner | -|------------------------|-------------------------------------------------------------------|-----------------------------------| -| `ignore` | The repro should not be run. | Skips it. | -| `explicit` | The repro must be run by hand, for example one project at a time. | Skips it. | -| `wip` | Work in progress. | Skips it. | -| `gui` | The bug shows only in a GUI, such as Visual Studio Test Explorer. | Skips it. | -| `closedasnotplanned` | The issue was closed as not planned. | Skips it. | -| `windows` | The repro runs on Windows only. | Skips it with `--skip-netfx`. | - -IssueRunner skips a marked folder also when you ask for the issue by number. To run it with IssueRunner, remove the -marker; or run the repro yourself, with `dotnet test` or its run scripts. +| Marker | Meaning | +|------------------------|-------------------------------------------------------------------| +| `ignore` | The repro should not be run. | +| `explicit` | The repro must be run by hand, for example one project at a time. | +| `wip` | Work in progress. | +| `gui` | The bug shows only in a GUI, such as Visual Studio Test Explorer. | +| `closedasnotplanned` | The issue was closed as not planned. | +| `windows` | The repro runs on Windows only. | ### Custom run scripts By default a repro is run with `dotnet test`. When it needs a filter, a runsettings file or only some of its projects, add a `run_.cmd` and a `run_.sh` with the same commands. Both are needed, since the repros are -run on Windows and Linux. Comments in the first 10 lines can give the expected results: +run on Windows and Linux. Comments in the first 10 lines can say what to expect: ```cmd REM EXPECT_TESTS=1 @@ -80,34 +91,28 @@ REM EXPECT_PASS=1 dotnet test --filter "FullyQualifiedName~Baz\(1\)" ``` -The keys are `EXPECT_TESTS`, `EXPECT_PASS`, `EXPECT_FAIL` and `EXPECT_SKIP`. They document what the script should -give, but IssueRunner does not check them yet: it only looks at the script's exit code. A script that must catch a -wrong number of tests, for example a filter that runs nothing, has to check the count itself and exit with a non-zero -code. +The keys are `EXPECT_TESTS`, `EXPECT_PASS`, `EXPECT_FAIL` and `EXPECT_SKIP`. They are information for the reader: +IssueRunner does not check them, it only looks at the script's exit code. A script that must catch a wrong number of +tests, for example a filter that runs nothing, has to check the count itself and exit with a non-zero code. -### Running the repros with IssueRunner +### Running all repros with IssueRunner -[IssueRunner](https://github.com/nunit/nunit.tools.issuerunner) runs the repros of an issues repository, keeps the -results, and reports regressions. It has a command line tool and a GUI. Build it from its repository, then run it -from the root of the issues repository: +[IssueRunner](https://github.com/nunit/nunit.tools.issuerunner) runs all or some repros of an issues repository and +reports which pass, for example to see if they still work with a new package before a release. You don't need it to +work on a bug. Clone it next to the issues repository, build it with its `build.cmd`, and run it from the root of the +issues repository: ```cmd ..\nunit.tools.issuerunner\run-tests.cmd --issues 1097,782 --feed Local +..\nunit.tools.issuerunner\run-tests.cmd --scope RegressionOnly ``` -| Option | Meaning | -|----------------------------------------------------------------|----------------------------------------------------------| -| `--issues ` | Run only these issues. | -| `--scope All`, `New`, `NewAndFailed`, `RegressionOnly`, `OpenOnly` | Run a group of issues. `RegressionOnly` runs the closed issues. | -| `--feed Stable`, `Beta`, `Alpha`, `Local` | Where the packages come from. `Local` adds `C:\nuget`. | -| `--rerun-failed` | Run again only what failed last time. | -| `--skip-netfx` | Skip .NET Framework projects, for example on Linux. | - -Other commands keep the repository up to date: `metadata sync-from-github` and `metadata sync-to-folders` update the -`issue_metadata.json` files, `reset` sets the target frameworks and packages back to `issue_initialstate.json`, and -`report generate` writes `TestReport.md`. The IssueRunner README has the details. +* `--feed Local` adds `C:\nuget`. `--scope RegressionOnly` runs the closed issues, `OpenOnly` the open ones. +* It skips folders with an `ignore`, `explicit`, `wip`, `gui` or `closedasnotplanned` marker, also when you ask for + them by number. `windows` folders are skipped with `--skip-netfx`. +* The issues repositories also run it in their CI workflows. -The closed issues are regression tests: run `--scope RegressionOnly` with a new package before a release. +The IssueRunner README has all commands and options. ## Testing an unreleased package From bee36c4fa204cbe871291dac3fa96ac9ac4357e6 Mon Sep 17 00:00:00 2001 From: Terje Sandstrom Date: Mon, 26 Oct 2026 21:22:07 +0100 Subject: [PATCH 5/7] Use the current IssueRunner scope values Co-Authored-By: Claude Opus 5.5 --- docs/articles/developer-info/Developing-NUnit.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/articles/developer-info/Developing-NUnit.md b/docs/articles/developer-info/Developing-NUnit.md index 90de23835..3e62f79b8 100644 --- a/docs/articles/developer-info/Developing-NUnit.md +++ b/docs/articles/developer-info/Developing-NUnit.md @@ -104,10 +104,10 @@ issues repository: ```cmd ..\nunit.tools.issuerunner\run-tests.cmd --issues 1097,782 --feed Local -..\nunit.tools.issuerunner\run-tests.cmd --scope RegressionOnly +..\nunit.tools.issuerunner\run-tests.cmd --scope Regression ``` -* `--feed Local` adds `C:\nuget`. `--scope RegressionOnly` runs the closed issues, `OpenOnly` the open ones. +* `--feed Local` adds `C:\nuget`. `--scope Regression` runs the closed issues, `Open` the open ones. * It skips folders with an `ignore`, `explicit`, `wip`, `gui` or `closedasnotplanned` marker, also when you ask for them by number. `windows` folders are skipped with `--skip-netfx`. * The issues repositories also run it in their CI workflows. From 7ccff0fb1690b2cd54801bfc5a166a0ae5637a9a Mon Sep 17 00:00:00 2001 From: Terje Sandstrom Date: Mon, 26 Oct 2026 21:24:22 +0100 Subject: [PATCH 6/7] Move IssueRunner to the end, and show nugetc for the local feed Co-Authored-By: Claude Opus 5.5 --- .../developer-info/Developing-NUnit.md | 52 ++++++++++++------- 1 file changed, 33 insertions(+), 19 deletions(-) diff --git a/docs/articles/developer-info/Developing-NUnit.md b/docs/articles/developer-info/Developing-NUnit.md index 3e62f79b8..a7152e6c3 100644 --- a/docs/articles/developer-info/Developing-NUnit.md +++ b/docs/articles/developer-info/Developing-NUnit.md @@ -95,30 +95,21 @@ The keys are `EXPECT_TESTS`, `EXPECT_PASS`, `EXPECT_FAIL` and `EXPECT_SKIP`. The IssueRunner does not check them, it only looks at the script's exit code. A script that must catch a wrong number of tests, for example a filter that runs nothing, has to check the count itself and exit with a non-zero code. -### Running all repros with IssueRunner +## Testing an unreleased package -[IssueRunner](https://github.com/nunit/nunit.tools.issuerunner) runs all or some repros of an issues repository and -reports which pass, for example to see if they still work with a new package before a release. You don't need it to -work on a bug. Clone it next to the issues repository, build it with its `build.cmd`, and run it from the root of the -issues repository: +Both projects get their package version from [MinVer](https://github.com/adamralph/minver): the last tag and the +number of commits since it, for example `6.4.0-alpha.0.25`. Build the package with the project's build script, and copy +it to a local feed, by default `C:\nuget`. A repro then needs a `NuGet.config` with that feed. The dotnet tool +[nugetc](https://github.com/HermitOS/nugetconfigcreator) makes and changes the file: ```cmd -..\nunit.tools.issuerunner\run-tests.cmd --issues 1097,782 --feed Local -..\nunit.tools.issuerunner\run-tests.cmd --scope Regression +dotnet tool install -g nugetc +cd Issue1097 +nugetc add local ``` -* `--feed Local` adds `C:\nuget`. `--scope Regression` runs the closed issues, `Open` the open ones. -* It skips folders with an `ignore`, `explicit`, `wip`, `gui` or `closedasnotplanned` marker, also when you ask for - them by number. `windows` folders are skipped with `--skip-netfx`. -* The issues repositories also run it in their CI workflows. - -The IssueRunner README has all commands and options. - -## Testing an unreleased package - -Both projects get their package version from [MinVer](https://github.com/adamralph/minver): the last tag and the -number of commits since it, for example `6.4.0-alpha.0.25`. Build the package with the project's build script, and copy -it to a local feed, by default `C:\nuget`. A repro then needs a `NuGet.config` with that feed: +`nugetc add local` creates a `NuGet.config` with nuget.org and `C:\nuget`, or adds `C:\nuget` to the file that is +there: ```xml @@ -129,6 +120,10 @@ it to a local feed, by default `C:\nuget`. A repro then needs a `NuGet.config` w ``` +`nugetc add myget` adds the NUnit MyGet feed, `https://www.myget.org/F/nunit/api/v3/index.json`. +`nugetc remove local`, `disable local` and `enable local` change the file again, and `nugetc reset` sets it back to +nuget.org only. + Update the repro to the new package with [dotnet-outdated](https://github.com/dotnet-outdated/dotnet-outdated), instead of editing the version by hand. `-inc` updates the packages whose names contain the text, and `-exc` leaves out the packages whose names contain the text: @@ -149,3 +144,22 @@ Two things to watch out for: isolated folder with `dotnet restore --packages `. * **Check what you test.** After a restore, check that the repro uses the package you expect, for example in `obj\project.assets.json`. + +## Running all repros with IssueRunner + +[IssueRunner](https://github.com/nunit/nunit.tools.issuerunner) runs all or some repros of an issues repository and +reports which pass, for example to see if they still work with a new package before a release. You don't need it to +work on a bug. Clone it next to the issues repository, build it with its `build.cmd`, and run it from the root of the +issues repository: + +```cmd +..\nunit.tools.issuerunner\run-tests.cmd --issues 1097,782 --feed Local +..\nunit.tools.issuerunner\run-tests.cmd --scope Regression +``` + +* `--feed Local` adds `C:\nuget`. `--scope Regression` runs the closed issues, `Open` the open ones. +* It skips folders with an `ignore`, `explicit`, `wip`, `gui` or `closedasnotplanned` marker, also when you ask for + them by number. `windows` folders are skipped with `--skip-netfx`. +* The issues repositories also run it in their CI workflows. + +The IssueRunner README has all commands and options. From bb46410be0f530e21a40e98bef02e4ff5a896dc3 Mon Sep 17 00:00:00 2001 From: Terje Sandstrom Date: Tue, 27 Oct 2026 11:44:51 +0100 Subject: [PATCH 7/7] Updated developing-nunit --- .../developer-info/Developing-NUnit.md | 76 ++++++++++--------- 1 file changed, 41 insertions(+), 35 deletions(-) diff --git a/docs/articles/developer-info/Developing-NUnit.md b/docs/articles/developer-info/Developing-NUnit.md index a7152e6c3..482cd5404 100644 --- a/docs/articles/developer-info/Developing-NUnit.md +++ b/docs/articles/developer-info/Developing-NUnit.md @@ -32,9 +32,9 @@ Each project has a repository with one repro for each issue that has one: A repro is a folder named `Issue`, for example `Issue1097` for issue 1097, with the smallest project that shows the bug and builds as it is. Link the folder from the issue. Guidelines for a repro: +* If there is no csproj in the issue, create a new one using `dotnet new unit`, or create it in Visual Studio, choose the NUnit template. * Use the code from the issue where you can, and add a comment that explains what the bug is. -* Use the current target framework and package versions, unless the bug needs older ones. -* Always reference `Microsoft.NET.Test.Sdk` and `coverlet.collector`, as a real test project does. +* Update the packages to the latest versions, if you're not addressing a particular specific version. A repro is not required to raise an issue: some bugs are hard to reproduce. When you can make one, it makes the issue much easier for others to understand and fix. @@ -43,16 +43,11 @@ much easier for others to understand and fix. Work with one repro at a time, on its own: -```cmd -cd Issue1097 -dotnet test -``` - -* Run its `run_*.cmd` or `run_*.sh` scripts instead when it has them, see below. -* Open the solution in Visual Studio when the bug is about Visual Studio. A `.runsettings` in the folder is picked up - automatically. For Test Explorer bugs, see - [Checking Test Explorer with te-dump](Developing-the-Adapter.md#checking-test-explorer-with-te-dump). -* Point it at an unreleased package to check a fix, see [Testing an unreleased package](#testing-an-unreleased-package). +* Check the repro using CLI or Visual Studio, preferably both. Use`dotnet test` or `dotnet test -s .runsettings` if a runsettings file is present. +* Open the solution in Visual Studio when the bug is about Visual Studio. A `.runsettings` in the folder is picked up automatically. +* For Test Explorer bugs, see + [Checking Test Explorer with te-dump](Developing-the-Adapter.md#checking-test-explorer-with-te-dump), especially if you use AI, then it can use te-dump. +* Refer to an unreleased package to check a fix, see [Testing an unreleased package](#testing-an-unreleased-package). ### Files in a repro folder @@ -60,7 +55,7 @@ dotnet test |-------------------------------------------------|---------------------------------------------------------------------------| | `*.csproj`, `*.slnx` or `*.sln` | The repro project or solution. | | `NuGet.config` | Package sources, with a local feed when testing an unreleased package. | -| `.runsettings` | Optional. Settings the bug needs. Visual Studio only picks it up from the solution folder. | +| `.runsettings` | Optional. Settings the bug needs. Also can enable adapter dumps. Visual Studio only picks it up from the solution folder. | | `run_*.cmd` and `run_*.sh` | Optional. Commands to run instead of plain `dotnet test`, see below. | | Marker file | Optional. Says what kind of repro it is, see below. | | `issue_metadata.json`, `issue_initialstate.json`, `issue_results.json` | Written by IssueRunner, see below. | @@ -68,7 +63,7 @@ dotnet test The `issue_*.json` files are information, and you don't need to make them: the issue's state, milestone and labels from GitHub, the target frameworks and packages the repro was made with, and the results of the last IssueRunner run. -A marker file says what kind of repro it is. It can be an empty file, or a `.md` file that says why: +A marker file, if present, says what kind of repro it is. It can be an empty file, or a `.md` file that says why: | Marker | Meaning | |------------------------|-------------------------------------------------------------------| @@ -98,18 +93,43 @@ tests, for example a filter that runs nothing, has to check the count itself and ## Testing an unreleased package Both projects get their package version from [MinVer](https://github.com/adamralph/minver): the last tag and the -number of commits since it, for example `6.4.0-alpha.0.25`. Build the package with the project's build script, and copy -it to a local feed, by default `C:\nuget`. A repro then needs a `NuGet.config` with that feed. The dotnet tool -[nugetc](https://github.com/HermitOS/nugetconfigcreator) makes and changes the file: +number of commits since it, for example `6.4.0-alpha.0.25`. +Build the package with the project's build script, and copy +it to a local feed, e.g. `C:\nuget`. + +A repro then needs a `NuGet.config` with that feed. + +You use `dotnet new nuget` and `dotnet nuget add` to set this up, or use `nugetc` for a simpler syntax. + +### Using standard `dotnet nuget` + +```cmd +dotnet new nuget.config +dotnet nuget add source c:\nuget --name local +dotnet nuget remove source local +dotnet nuget disable source local +``` + +### Using `nugetc` + +The dotnet tool [nugetc](https://github.com/HermitOS/nugetconfigcreator) makes and changes the file: ```cmd dotnet tool install -g nugetc cd Issue1097 -nugetc add local +nugetc add local // Creates new or updates exsistins and adds default too +nugetc remove local +nugetc disable local ``` -`nugetc add local` creates a `NuGet.config` with nuget.org and `C:\nuget`, or adds `C:\nuget` to the file that is -there: +### Using published alpha releases from Myget + +`nugetc add myget` adds the NUnit MyGet feed, +or use `dotnet nuget add source https://www.myget.org/F/nunit/api/v3/index.json --name myget`. + +### Manually + +You can also do it manually, it should look like: ```xml @@ -120,21 +140,7 @@ there: ``` -`nugetc add myget` adds the NUnit MyGet feed, `https://www.myget.org/F/nunit/api/v3/index.json`. -`nugetc remove local`, `disable local` and `enable local` change the file again, and `nugetc reset` sets it back to -nuget.org only. - -Update the repro to the new package with [dotnet-outdated](https://github.com/dotnet-outdated/dotnet-outdated), -instead of editing the version by hand. `-inc` updates the packages whose names contain the text, and `-exc` leaves -out the packages whose names contain the text: - -```cmd -REM An adapter fix -dotnet outdated -u -pre Always -inc NUnit3TestAdapter - -REM A framework fix: the NUnit package, not the adapter or the analyzers -dotnet outdated -u -pre Always -inc NUnit -exc NUnit3TestAdapter -exc NUnit.Analyzers -``` +## Watch out for Two things to watch out for: