diff --git a/cSpell.json b/cSpell.json index af09856cb..15bafd655 100644 --- a/cSpell.json +++ b/cSpell.json @@ -41,6 +41,7 @@ "Refactorings", "reimplementation", "Remoting", + "repros", "ReSharper", "ruleset", "runtimes", diff --git a/docs/articles/developer-info/Contributions.md b/docs/articles/developer-info/Contributions.md index 7fc7cc10f..6c2bd7185 100644 --- a/docs/articles/developer-info/Contributions.md +++ b/docs/articles/developer-info/Contributions.md @@ -8,6 +8,9 @@ developers. All proposed changes to the code should come in the form of GitHub pull requests. Pull requests are built automatically on the build server and they make it easy to track the origin of any change. +For how to reproduce and fix a bug, and check the fix, see +[How to Develop and Fix Bugs in NUnit](Developing-NUnit.md). + ## Quality When appropriate, pull requests should contain enough unit tests to provide coverage for the changes they introduce. diff --git a/docs/articles/developer-info/Developing-NUnit.md b/docs/articles/developer-info/Developing-NUnit.md new file mode 100644 index 000000000..482cd5404 --- /dev/null +++ b/docs/articles/developer-info/Developing-NUnit.md @@ -0,0 +1,171 @@ +--- +uid: developingnunit +--- + +# How to Develop and Fix Bugs in NUnit + +This page describes how we work on a bug in the NUnit framework or the NUnit3 VS Adapter: reproduce it, fix it, check +the fix against the repro, and keep the repro as a regression test. The steps for each project are on their own pages: + +* [Developing the Framework](Developing-the-Framework.md) +* [Developing the Adapter](Developing-the-Adapter.md) + +## The workflow + +1. **Reproduce.** Make a repro project for the issue in the project's issues repository, and check that it shows + the bug with the released packages. +2. **Specify.** Write a failing unit test or acceptance test in the project repository. +3. **Fix.** Change the code until the test passes, and run the project's tests. +4. **Check.** Build a package with the fix, point the repro at it, and check that the bug is gone. +5. **Submit.** Raise the pull request, with `Fixed #` at the top of the description, see + [Contributions](Contributions.md). Keep the repro, with the package versions updated. + +## The issues repositories + +Each project has a repository with one repro for each issue that has one: + +| Repository | Repros for | +|---------------------------------------------------------------------------------|----------------------------------| +| [nunit.issues](https://github.com/nunit/nunit.issues) | The NUnit framework | +| [nunit3-vs-adapter.issues](https://github.com/nunit/nunit3-vs-adapter.issues) | The NUnit3 VS Adapter | + +A repro is a folder named `Issue`, for example `Issue1097` for issue 1097, with the smallest project that +shows the bug and builds as it is. Link the folder from the issue. Guidelines for a repro: + +* If there is no csproj in the issue, create a new one using `dotnet new unit`, or create it in Visual Studio, choose the NUnit template. +* Use the code from the issue where you can, and add a comment that explains what the bug is. +* Update the packages to the latest versions, if you're not addressing a particular specific version. + +A repro is not required to raise an issue: some bugs are hard to reproduce. When you can make one, it makes the issue +much easier for others to understand and fix. + +### Using a repro + +Work with one repro at a time, on its own: + +* Check the repro using CLI or Visual Studio, preferably both. Use`dotnet test` or `dotnet test -s .runsettings` if a runsettings file is present. +* Open the solution in Visual Studio when the bug is about Visual Studio. A `.runsettings` in the folder is picked up automatically. +* For Test Explorer bugs, see + [Checking Test Explorer with te-dump](Developing-the-Adapter.md#checking-test-explorer-with-te-dump), especially if you use AI, then it can use te-dump. +* Refer to an unreleased package to check a fix, see [Testing an unreleased package](#testing-an-unreleased-package). + +### Files in a repro folder + +| File | Purpose | +|-------------------------------------------------|---------------------------------------------------------------------------| +| `*.csproj`, `*.slnx` or `*.sln` | The repro project or solution. | +| `NuGet.config` | Package sources, with a local feed when testing an unreleased package. | +| `.runsettings` | Optional. Settings the bug needs. Also can enable adapter dumps. Visual Studio only picks it up from the solution folder. | +| `run_*.cmd` and `run_*.sh` | Optional. Commands to run instead of plain `dotnet test`, see below. | +| Marker file | Optional. Says what kind of repro it is, see below. | +| `issue_metadata.json`, `issue_initialstate.json`, `issue_results.json` | Written by IssueRunner, see below. | + +The `issue_*.json` files are information, and you don't need to make them: the issue's state, milestone and labels +from GitHub, the target frameworks and packages the repro was made with, and the results of the last IssueRunner run. + +A marker file, if present, says what kind of repro it is. It can be an empty file, or a `.md` file that says why: + +| Marker | Meaning | +|------------------------|-------------------------------------------------------------------| +| `ignore` | The repro should not be run. | +| `explicit` | The repro must be run by hand, for example one project at a time. | +| `wip` | Work in progress. | +| `gui` | The bug shows only in a GUI, such as Visual Studio Test Explorer. | +| `closedasnotplanned` | The issue was closed as not planned. | +| `windows` | The repro runs on Windows only. | + +### Custom run scripts + +By default a repro is run with `dotnet test`. When it needs a filter, a runsettings file or only some of its +projects, add a `run_.cmd` and a `run_.sh` with the same commands. Both are needed, since the repros are +run on Windows and Linux. Comments in the first 10 lines can say what to expect: + +```cmd +REM EXPECT_TESTS=1 +REM EXPECT_PASS=1 +dotnet test --filter "FullyQualifiedName~Baz\(1\)" +``` + +The keys are `EXPECT_TESTS`, `EXPECT_PASS`, `EXPECT_FAIL` and `EXPECT_SKIP`. They are information for the reader: +IssueRunner does not check them, it only looks at the script's exit code. A script that must catch a wrong number of +tests, for example a filter that runs nothing, has to check the count itself and exit with a non-zero code. + +## Testing an unreleased package + +Both projects get their package version from [MinVer](https://github.com/adamralph/minver): the last tag and the +number of commits since it, for example `6.4.0-alpha.0.25`. +Build the package with the project's build script, and copy +it to a local feed, e.g. `C:\nuget`. + +A repro then needs a `NuGet.config` with that feed. + +You use `dotnet new nuget` and `dotnet nuget add` to set this up, or use `nugetc` for a simpler syntax. + +### Using standard `dotnet nuget` + +```cmd +dotnet new nuget.config +dotnet nuget add source c:\nuget --name local +dotnet nuget remove source local +dotnet nuget disable source local +``` + +### Using `nugetc` + +The dotnet tool [nugetc](https://github.com/HermitOS/nugetconfigcreator) makes and changes the file: + +```cmd +dotnet tool install -g nugetc +cd Issue1097 +nugetc add local // Creates new or updates exsistins and adds default too +nugetc remove local +nugetc disable local +``` + +### Using published alpha releases from Myget + +`nugetc add myget` adds the NUnit MyGet feed, +or use `dotnet nuget add source https://www.myget.org/F/nunit/api/v3/index.json --name myget`. + +### Manually + +You can also do it manually, it should look like: + +```xml + + + + + + +``` + +## Watch out for + +Two things to watch out for: + +* **Same version, different code.** Two branches with the same number of commits get the same version. NuGet caches + a package by its version, so a repro can silently use an older package with that version. Give a package a version + of its own with `set MINVERVERSIONOVERRIDE=6.4.0-alpha.0.40` before building and packing, or restore into an + isolated folder with `dotnet restore --packages `. +* **Check what you test.** After a restore, check that the repro uses the package you expect, for example in + `obj\project.assets.json`. + +## Running all repros with IssueRunner + +[IssueRunner](https://github.com/nunit/nunit.tools.issuerunner) runs all or some repros of an issues repository and +reports which pass, for example to see if they still work with a new package before a release. You don't need it to +work on a bug. Clone it next to the issues repository, build it with its `build.cmd`, and run it from the root of the +issues repository: + +```cmd +..\nunit.tools.issuerunner\run-tests.cmd --issues 1097,782 --feed Local +..\nunit.tools.issuerunner\run-tests.cmd --scope Regression +``` + +* `--feed Local` adds `C:\nuget`. `--scope Regression` runs the closed issues, `Open` the open ones. +* It skips folders with an `ignore`, `explicit`, `wip`, `gui` or `closedasnotplanned` marker, also when you ask for + them by number. `windows` folders are skipped with `--skip-netfx`. +* The issues repositories also run it in their CI workflows. + +The IssueRunner README has all commands and options. diff --git a/docs/articles/developer-info/Developing-the-Adapter.md b/docs/articles/developer-info/Developing-the-Adapter.md new file mode 100644 index 000000000..52906da1f --- /dev/null +++ b/docs/articles/developer-info/Developing-the-Adapter.md @@ -0,0 +1,139 @@ +# Developing the Adapter + +This page follows the [common workflow](Developing-NUnit.md) for the +[NUnit3 VS Adapter](https://github.com/nunit/nunit3-vs-adapter). The repros are in +[nunit3-vs-adapter.issues](https://github.com/nunit/nunit3-vs-adapter.issues). The adapter connects NUnit to VSTest +and Microsoft.Testing.Platform, so many of its bugs are about finding a test, selecting it with a filter, or showing +it in Visual Studio Test Explorer. This page also shows how to check those. + +## Building and testing + +The repository has a Cake build script, `build.cmd` (or `build.ps1`), in the repository root: + +| Command | What it does | +|-----------------------|-------------------------------------------------------------------------------------------| +| `build` | Builds the solution (default target). | +| `build -t test` | Runs the unit tests on net462 and net8.0, with both VSTest and `dotnet test`. | +| `build -t package` | Creates the NuGet package and zip in `package\`. It does not build first. | +| `build -t acceptance` | Builds, creates the NuGet package and runs the acceptance tests against that package. | + +* Run the acceptance tests with `build -t acceptance`. They test the NuGet package, so running the acceptance test + project directly uses a stale or missing package. +* `build -t package` packs the binaries that are already built. Run `build` first, or the package contains old code. + +Copy the package to the local feed with `copynp.cmd`, giving the version from the file name in `package\`: + +```cmd +.\copynp.cmd 6.4.0-alpha.0.25 +``` + +Then point the repro at it, see [Testing an unreleased package](xref:developingnunit#testing-an-unreleased-package). + +## The adapter's dump files + +Turn on the adapter's dump files in the repro's `.runsettings` while you work on a bug, see +[DumpXmlTestDiscovery and DumpXmlTestResults](xref:tipsandtricks#dumpxmltestdiscovery-and-dumpxmltestresults): + +```xml + + + true + true + + +``` + +The dumps are written to `bin\\\Dump`: + +* `D_.dump` is the discovery: the NUnit XML of every test. +* `E_.dump` is the execution. Its `VS_Input_TestCases` section lists exactly which tests VSTest or Visual + Studio asked the adapter to run, and the `TestFilter` section shows the NUnit filter the adapter made from them. + +These two sections tell you whether a test was lost before it reached the adapter, or inside the adapter. The +execution dump is overwritten by each run; set `DumpNumberedFiles` to keep one for each run. + +## Checking filters from the command line + +Many adapter bugs are about selecting a test: a test is found, but a filter for it runs nothing, or an error. VSTest +can list the fully qualified names and run a filter, without Visual Studio: + +```cmd +dotnet vstest bin\Debug\net10.0\Issue677.dll /ListFullyQualifiedTests /ListTestsTargetPath:fqn.txt +dotnet vstest bin\Debug\net10.0\Issue677.dll "/TestCaseFilter:FullyQualifiedName=TestFixtureSourceError.TestsBroken(Windows Edgium).Test1" +``` + +Run each listed name alone as a filter, both as written and with `\`, `(`, `)`, `&`, `|`, `=`, `!` and `~` escaped +with a `\`, and compare the number of tests run with the number listed. A small script that loops over `fqn.txt` does +this for a whole repro. `dotnet test --filter` uses the same filter syntax. + +## Checking Test Explorer with te-dump + +Some bugs show only in Visual Studio Test Explorer: a test in the wrong place in the tree, an extra node that never +runs, or a result that is not shown. `te-dump` +([NUnit.Tools.TestExplorerDump](https://github.com/nunit/NUnit.Tools.TestExplorerDump)) prints the Test Explorer +tree as text, so you, a script or an AI assistant can see exactly what Test Explorer shows, and compare it before and +after a fix. It can also drive Visual Studio: start it on a solution, rebuild, run all or some tests, and close it. It +reads Test Explorer through Windows UI Automation and drives Visual Studio through EnvDTE; nothing is installed into +Visual Studio. It runs on Windows only. Its README has the install steps and all options. + +### A full round trip + +Clean the repro first, so no results or test lists from earlier runs are left: delete the `.vs`, `bin` and `obj` +folders, for example with [vsclean](https://github.com/OsirisTerje/VSClean) `-d`. Then run a session in the repro +folder: + +```cmd +cd Issue1408 +te-dump session +``` + +`session` starts a new Visual Studio on the solution, rebuilds it, waits for test discovery, dumps the tree, runs all +tests, dumps the tree again and closes Visual Studio, also when a step fails. The trees are printed, and written to a +`te-dump` folder next to the solution, for example `te-dump\Issue1408-20261026-194618-discovered.txt` and +`...-after-run.txt`. Add `te-dump/` to `.gitignore`. + +Each line is `Kind: Name [Outcome] (test count)`; a test has no kind and no count: + +```text +# after run +Project: Issue1408 [Passed] (3) + Namespace: NUnitTestExplorerFailure [Passed] (3) + Class: Tests [Passed] (3) + Test Group: Demonstration [Passed] (2) + Demonstration([(0, 0)]) [Passed] + Demonstration([(1, 2), (3, 4)]) [Passed] + Demonstration [Not Run] +``` + +Here the two tests ran, and `Demonstration [Not Run]` is an extra node. It comes from Visual Studio's own source-based +test discovery, which finds the method in the source code but cannot match it to the tests the adapter reports. + +### Running single tests + +Test Explorer often runs only some tests: a single test, or one node of the tree. Bugs in that path do not show with +Run All. Keep Visual Studio open and run the tests one at a time: + +```cmd +te-dump start +te-dump --pid 73760 --expand-all +te-dump run --pid 73760 --filter SimpleTest +te-dump --pid 73760 --expand-all +te-dump close --pid 73760 +``` + +`start` prints the process id to use with the other commands. `--filter` is Test Explorer's search text, for example +`SimpleTest`, `FullName:"Ns.Class.Method"` or `Class:"Class1"`. `te-dump list` shows the Visual Studio instances that +have Test Explorer open. + +Do not run a session on a solution that is already open in another Visual Studio: both use the same `.vs` folder. + +### Reading the results + +When Test Explorer shows a test as Not Run after it has run, look at the adapter's execution dump from that run: + +* The test is not in `VS_Input_TestCases`: Visual Studio did not ask the adapter to run it. +* The test is in `VS_Input_TestCases` but has no result in the dump: the adapter's filter did not select it, see the + `TestFilter` section. +* The test has a result in the dump: NUnit ran it, and the adapter received the result. The dump records it before the + adapter converts the result and reports it to VSTest, so a failure in that step is still possible; look for errors + in the Tests output pane. With no errors there, Visual Studio did not match the reported result to the test it shows. diff --git a/docs/articles/developer-info/Developing-the-Framework.md b/docs/articles/developer-info/Developing-the-Framework.md new file mode 100644 index 000000000..17418a33c --- /dev/null +++ b/docs/articles/developer-info/Developing-the-Framework.md @@ -0,0 +1,33 @@ +# Developing the Framework + +This page follows the [common workflow](Developing-NUnit.md) for the [NUnit framework](https://github.com/nunit/nunit). +The repros are in [nunit.issues](https://github.com/nunit/nunit.issues). + +## Building and testing + +The repository has one solution, `nunit.slnx`, and a Cake build script: `build.cmd` (Windows), `build.ps1` +(PowerShell) or `build.sh` (Linux and macOS). They take the same arguments: + +| Command | What it does | +|------------------------------|-----------------------------------------------------------------------| +| `build` | Restores and builds everything (default target). | +| `build --target=Rebuild` | Cleans the output and builds everything. | +| `build --target=Test` | Runs all tests, on all target frameworks. | +| `build --target=Package` | Creates the packages in `package\`. It does not build first. | +| `build --showdescription` | Lists all targets. | + +The configuration is `Release` by default; use `--configuration=Debug` to change it. The tests to run are in the +`nunit.framework.tests-*`, `nunit.framework.legacy.tests-*` and `nunitlite.tests-*` projects. The other test projects +contain tests that fail on purpose, for the integration tests. [BUILDING.md](https://github.com/nunit/nunit/blob/main/BUILDING.md) +in the repository has the details. + +## Fixing a bug + +1. Make the repro in [nunit.issues](https://github.com/nunit/nunit.issues): a folder `Issue` with a + `readme.md` that links to the issue, and a project that builds and runs. Link the folder from the issue. +2. Add a failing test to the framework's test projects that shows the bug, then fix the code. Run + `build --target=Test`: the framework targets several frameworks, and a fix can behave differently on each. +3. Run `build` and `build --target=Package`, and copy the `NUnit` package from `package\` to your local feed. +4. Update the repro to that package, see [Testing an unreleased package](xref:developingnunit#testing-an-unreleased-package), + and check that the bug is gone. The repro runs through the NUnit3 VS Adapter, so a bug that only shows in Visual + Studio can be checked there too, see [Checking Test Explorer with te-dump](Developing-the-Adapter.md#checking-test-explorer-with-te-dump). diff --git a/docs/articles/developer-info/Team-Practices.md b/docs/articles/developer-info/Team-Practices.md index 75264f49d..9aac33612 100644 --- a/docs/articles/developer-info/Team-Practices.md +++ b/docs/articles/developer-info/Team-Practices.md @@ -1,26 +1,3 @@ --- -uid: teampractices +redirect_url: ./The-Teams.html --- - -# Team Practices - -## How We Work As a Team - -| | | -| ----------------------------------- | -------------------------------------------------------------------- | -| [Issue Tracking](Issue-Tracking.md) | How we track bugs, enhancements and features on GitHub. | -| [Contributions](Contributions.md) | Explains how to review and merge code submitted by other developers. | - -## Technical Practices - -| | | -| ------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | -| Building and testing | See BUILDING.md in each repository. | -| [Coding Standards](Coding-Standards.md) | Describes coding standards used in NUnit. | -| [Best practices for XML documentation](Best-practices-for-XML-documentation.md) | Guidelines to consider when dealing with public APIs. | -| [Packaging the Framework](Packaging-the-Framework.md) | How to create the packages for a new NUnit Framework release. | -| [Packaging the Console and Engine](Packaging-the-Console-and-Engine.md) | How to create the packages for a new NUnit Console and Engine release. | -| [Packaging Extensions](Packaging-Extensions.md) | How to create the packages and release NUnit Extensions. | -| [Packaging the Installer](Packaging-the-Installer.md) | How to create and release the NUnit MSI Installer. | -| [Packaging the V2 Adapter](Packaging-the-V2-Adapter.md) | How to create the packages for a new NUnit 2 VS Adapter release. | -| [Packaging the V3 Adapter](Packaging-the-V3-and-V4-Adapter.md) | How to create the packages for a new NUnit 3 VS Adapter release. | diff --git a/docs/articles/developer-info/The-Teams.md b/docs/articles/developer-info/The-Teams.md index 4a94825fd..0251c746c 100644 --- a/docs/articles/developer-info/The-Teams.md +++ b/docs/articles/developer-info/The-Teams.md @@ -1,3 +1,7 @@ +--- +uid: theteams +--- + # The Teams in the NUnit Project The NUnit project is made up of several teams, each with its own focus and responsibilities, and with a set of team diff --git a/docs/articles/developer-info/toc.yml b/docs/articles/developer-info/toc.yml index f578cc680..56aa84146 100644 --- a/docs/articles/developer-info/toc.yml +++ b/docs/articles/developer-info/toc.yml @@ -1,24 +1,34 @@ - name: The Teams href: The-Teams.md -- name: Team Practices - href: Team-Practices.md -- name: Specifications - topicUid: specifications -- name: Best Practices for XML Documentation - href: Best-practices-for-XML-documentation.md -- name: Coding Standards - href: Coding-Standards.md -- name: Contributions - href: Contributions.md -- name: Issue Tracking - href: Issue-Tracking.md +- name: Working Together + items: + - name: Issue Tracking + href: Issue-Tracking.md + - name: Contributions + href: Contributions.md + - name: How to Develop and Fix Bugs + href: Developing-NUnit.md + items: + - name: Developing the Framework + href: Developing-the-Framework.md + - name: Developing the Adapter + href: Developing-the-Adapter.md +- name: Code + items: + - name: Coding Standards + href: Coding-Standards.md + - name: Best Practices for XML Documentation + href: Best-practices-for-XML-documentation.md + - name: Specifications + topicUid: specifications - name: Packaging href: Packaging.md -- name: Packaging Extensions - href: Packaging-Extensions.md -- name: Packaging the Console and Engine - href: Packaging-the-Console-and-Engine.md -- name: Packaging the Framework - href: Packaging-the-Framework.md -- name: Packaging the V3/V4 Adapter - href: Packaging-the-V3-and-V4-Adapter.md + items: + - name: Packaging the Framework + href: Packaging-the-Framework.md + - name: Packaging the Console and Engine + href: Packaging-the-Console-and-Engine.md + - name: Packaging the V3/V4 Adapter + href: Packaging-the-V3-and-V4-Adapter.md + - name: Packaging Extensions + href: Packaging-Extensions.md diff --git a/docs/classic.md b/docs/classic.md index 0c2ff9448..95ed5d0b0 100644 --- a/docs/classic.md +++ b/docs/classic.md @@ -20,5 +20,6 @@ working on NUnit or wishing to do so. ## Developer Documentation -* [Team practices](xref:teampractices) describe how NUnit works and how our teams work. +* [The teams](xref:theteams) describe how the NUnit project is organized, and + [How to develop and fix bugs](xref:developingnunit) describes how we work on a bug. * [Specifications](xref:specifications) are descriptions of features we plan to add. diff --git a/docs/index.md b/docs/index.md index c485717e9..7c1d3ab3e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -149,11 +149,11 @@ public class CalculatorTests
-

Developer info

+

Developer info

For contributors to the NUnit projects.