Resolving Problem Reports

Resolving Problem Reports

Resolving Problem Reports

Introduction

The NetBSD Project uses the GNATS “Problem Report” database to accept and track bug/problem reports from all users of NetBSD. When used properly, this facility allows us to make sure that no problem with the NetBSD software goes unfixed.

The GNATS database uses Internet E-mail as its principal submission mechanism, and keeps the problem reports (commonly abbreviated “PR”) in Internet E-mail format, with an extended header format in the body of the message. The database uses one file per PR, and each category is a directory, in a manner similar to the MH mail system, and NetNews.

Accessing the Problem Report Database

Web Interface

There is a web-based interface to the GNATS database that has both a tree of database summary pages, and a search facility. The web interface has some limitations:

  • The web interface does not provide access to any PR marked “confidential”.
  • PRs displayed by a web browser have been through sufficient transformation that “cut & paste” is not likely to do what you expect (i.e. patches will not apply properly because of white space substitution, uuencoded text will not decode properly).
The netbsd-bugs Mailing List

All PRs except confidential ones are sent to the netbsd-bugs mailing list, so that subscribers can see each PR as it is added to the database.

The GNATS commands on gnats.NetBSD.org

The GNATS database lives on the host gnats.NetBSD.org. All developers are given an account on that host so that they can directly manipulate PRs, and access confidential PRs. The commands edit-pr and query-pr live in /usr/pkg/bin; Make sure you have that directory in your $PATH.

The query-pr command is a proper database query interface; it has a large number of options to search the database. Once you know the PR number of the PR you wish to manipulate, you can:

  • query-pr --full <number>

    This will dump out the full PR without any transformation to standard output.

  • edit-pr <number>

    This command will start up a text editor (vi by default; but this can be overridden by the $EDITOR or $VISUAL environment variables), so that changes can be made to the PR.

    Please don't kill your edit-pr without checking for a left-over shell afterwards (also check if you lost your connection with edit-pr running).

For more information please consult edit-pr(1) or query-pr(1) man pages. As with most GNU software, there are also “info” pages available through the info command. Also, invoking edit-pr or query-pr without any arguments will cause it to give its usage message.

The Common Reasons to edit-pr

The most common changes made to a PR are:

  1. Change the PR's >State: field to one of the values listed in state as it progresses through the process of resolving it.

    The person listed in the >Responsible: field of the PR should be making these state changes, as it is necessary.

  2. Change the PR's >Responsible: field to the account name of a developer who will handle the PR. This person becomes the PR Submitter's primary contact for getting the problem resolved.

    This field can have any username from /etc/passwd on gnats.NetBSD.org and anyone listed in the /home/gnats/gnatsdb/gnats-adm/responsible file.

    All PRs get a default Responsible Person when they are initially filed, appropriate to the category in which the PR was filed (e.g. “security-officer” for PRs in the “security” category). The /home/gnats/gnatsdb/gnats-adm/categories file lists the default Responsible Person for each category.

    There is also a table of developers responsible for current PRs.

  3. Change the PR's >Category: field.

    It is not unusual for a PR Submitter to have made a poor category choice. There is a list of PR categories and their definitions and a table of current PRs by category.

    The /home/gnats/gnatsdb/gnats-adm/categories file lists the valid categories and the default Responsible Person. It is usually necessary to change the >Responsible: field at the same time to a more appropriate person. Most often, the correct Responsible Person is the default Responsible Person for the new category.

  4. Add themselves to the PR's >Notify-List: field.

    If you have an interest in the PR without being the submitter nor >Responsible for it, add yourself to this field. You may have to insert it.

  5. Change other PR Fields according to the analysis of the responsible developer.

Edit each field you think needs modifying, then save the file and exit the editor. The edit-pr will then prompt for a short explanation to be typed for each key field change (mostly >State: and >Responsible:). This text is entered one line at time, ending with ^D.

This text is then sent via e-mail to the PR Submitter, the Responsible developer, and . It is also appended to the PR by edit-pr along with the user ID of the developer making the change, a timestamp, and the entered text.

Unfortunately, no external editor can be invoked at this point; if you make a mistake, you'll have to use edit-pr to correct it.

Resolving Problem Reports

The Ideal Process

In an ideal world, the process for a problem report is as follows:

  1. A NetBSD user has a problem with NetBSD. He invokes the send-pr command on his system (assuming it's still stable enough to do that), and files a Problem Report. Hopefully, he follows all the advice found in "What goes into a Problem Report."

    If the user's own system is not stable enough to use send-pr, there is a web interface that can be used to submit problem reports.

  2. When the PR arrives at gnats.NetBSD.org, the GNATS database system examines it, and files it. If the PR is malformed, it will be filed in the pending category, marked confidential, awaiting manual intervention by the GNATS database administrator.

    If the format is OK, the PR is assigned a PR number, filed into the requested category, and E-mailed out again to the default responsible party for the category, and to the netbsd-bugs mailing list. A notice of the PR number and default Responsible Person is also E-mailed back to the PR Submitter.

  3. The Responsible Person should read and analyze the PR. Any other person who has insight into the problem should also add whatever information they can to the problem report (this is why the report is mailed out to a mailing list; a wide audience increases the probability that a key insight needed to solve the problem will be discovered).

    If the default Responsible Person determines that another developer is a more appropriate Responsible Person, the PR should be reassigned with edit-pr. The new Responsible Person should read and analyze the PR.

  4. Once a cause and potential fix has been identified, a description should be added to the PR, and the state should be changed to analyzed. At this point, implementation of the fix for the problem begins.

    This part of the process should begin as quickly as possible, since a user with a current problem is suffering, but also has his attention engaged, and his hardware available for testing potential fixes. If a PR is allowed to languish, the opportunity to reproduce the problem and test potential fixes may be lost.

  5. Once the implementation of the fix is completed and communicated to the PR submitter, the PR state should be changed to feedback, awaiting response from the PR submitter that the fix really works.

    A PR should also be put into feedback state when input is required from the submitter to complete the analysis of the PR (i.e. when you ask them a question), or when you need information from some other source (essentially, feedback is a wait state).

  6. Once it is confirmed that the problem has been fixed, it should be pulled up to the appropriate active branches. After sending the pullup request(s), you should set the PR state to pending-pullups, and list the pullup number(s) in the reason.

    When the pullup request(s) have not yet been sent (perhaps because they are complicated) the state should be set to needs-pullups to draw this fact to the attention of passersby.

  7. Once the PR submitter confirms that the fix works, and all pullup requests have been issued and completed, the PR can be closed.

At each step of the PR handling process, make sure that feedback and other analysis and commentary is appended to the PR by using a proper E-mail subject line and making sure that the messages are copied to . Having a complete record of information about the PR is valuable both while hunting down the bug and for future system maintenance.

If at all possible, it is important to get any fix committed to the CVS trunk pulled up by NetBSD Release Engineering to the "-release" branch, so that people who are tracking that branch can get the fix right away rather than waiting for the next major release of NetBSD. This also makes it possible for the next point release of NetBSD to have the fix.

The NetBSD Community is a whole lot of very smart, and very experienced people. If you're having trouble analyzing a problem report, ask questions in the appropriate mailing list; more than likely, someone will be able to help.

Other Ways Things Get Done

The Ideal Process was described above. That's not the only way that problem reports get handled. All of the principal people involved in NetBSD are pretty busy, and can't devote full attention to this project. As such, if you see a PR that you can solve that hasn't been attended to yet, go claim it by setting yourself as the Responsible Person with edit-pr, and solve it.

Even if you don't feel qualified to hack the code yourself, if you can offer a test case or other information, send it along to GNATS to be appended to the PR. “Many hands make light work.”

Some problem reports are so trivial that the fix is obvious (or perhaps the fix was provided by the submitter), that they go directly from open to closed immediately after the fix is committed.

If for some reason you find that you're unable to finish handling the PR, reset the >Responsible: field to whoever had responsibility before you took the PR over. Don't prevent others from making progress on the PR because they think you're taking care of it.

As long as you're marked as the Responsible Person for a PR, you'll receive a monthly E-mail reminder about it. Use those reminders to drive you to review PRs and put them into their correct states as time passes.

When a PR is in feedback state, the PR submitter gets an E-mail monthly reminder at the same time as the Responsible Persons do, to prompt or prod them into responding. Generally, if there has been no response for more than three months (three reminder cycles), it's pretty safe to assume that the submitter is gone or no longer cares. At that point, whether to close the PR becomes a judgement call for the Responsible Person - how serious is it? Should it be solved without further input from the Submitter?

The other way we use the GNATS PR database is to keep track of problems which are waiting for larger issues to be solved. The oldest PR in the database at this writing, lib/13 (yes, of course it would be 13!) begs the entire internationalization of the NetBSD system. I18N is a hard problem that requires a wholesale overhaul of the system, which is why that PR is still open after seven years. This doesn't mean we'll never solve it; just that it isn't as critical as some other problems reported in the database.

In effect, this usage of the GNATS PR database is as a long-term project tracking system.

Priorities, Severities, and Releases

In an ideal world, the GNATS PR database would be empty, we'd release perfect software, and everyone would get along.

The >Priority: field in the PR reflects this ideal, in that high priority is supposed to be fixed immediately; medium is supposed to be resolved before the next release of NetBSD (major or minor?), and low priority gets solved “eventually”.

In practice, PR resolution is dependent on the right mix of submitter interest, developer interest, problem reproduceability, hardware availability, and good timing. If any of the required elements is missing from the mix, the PR will sit.

If we were really diligent about PRs, we would adjust the priority of each PR to reflect its actual importance, and probability of getting fixed according to the definitions. Unfortunately, that requires an overall evaluation of Release Engineering goals and targets and all PRs relative to each other, which is difficult for a dispersed group to do in an organized fashion.

In contrast, the >Severity: field is really an expression of the amount of pain the user is going through with the problem being reported, and it's something we really shouldn't adjust without careful consideration.

The proper procedure would be to review all PRs in the database at each release point, to decide on a per-PR basis whether to “fix now”, “fix later”, “suspend” and adjust priorities. Perhaps one day we'll have the resources and manpower for that.

Remote GNATS Operations

For people who find logging into a remote host tedious, the following csh aliases might be useful:

alias query-pr  'ssh gnats.NetBSD.org query-pr --full \!* | tee pr-\!*'
alias edit-pr   ssh -t gnats.NetBSD.org edit-pr
Back to  NetBSD Developer Documentation