4. Contributing to PyPop

Contributions to PyPop are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.

4.1. Reporting and requesting

Did you find a bug?

When reporting a bug please use one of the provided issue templates if applicable, otherwise just start a blank issue and describe your situation. Here is a checklist:

  • Check previous issues. Ensure the bug was not already reported by searching on GitHub under Issues.

  • Provide complete self-contained examples. If you’re unable to find an open issue addressing the problem, open a new one. Be sure to include a title and clear description, as much relevant information as possible, and a code sample or an executable test case (including any input files) demonstrating the expected behavior that is not occurring.

  • Use templates. If possible, use the relevant bug report templates to create the issue. For a standard bug report (including installation issues), please use this: bug report template, for feature requests or documentation issues, see below.

  • Provide full commands and errors as plaintext, not screenshots. When you are including the output of an error in your bug report (whether an installation error, a build error, an error running pypop or an error building docs), please cut-and-paste from your console application or terminal, the entire set of commands leading up to the error, along with the complete error output as a single plaintext output. E.g. here is an example error from running pypop on a badly formed .ini file:

    $ pypop -c minimal.ini USAFEL-UchiTelle-small.pop
Traceback (most recent call last):
  File "/home/user/.conda/envs/pypop/bin/pypop", line 8, in <module>
    sys.exit(main())
  File "/home/user/.conda/envs/pypop/lib/python3.10/site-packages/PyPop/pypop.py", line 250, in main
    config = getConfigInstance(configFilename, altpath)
  File "/home/user/.conda/envs/pypop/lib/python3.10/site-packages/PyPop/Main.py", line 62, in getConfigInstance
    config.read(configFilename)
  File "/home/user/.conda/envs/pypop/lib/python3.10/configparser.py", line 698, in read
    self._read(fp, filename)
  File "/home/user/.conda/envs/pypop/lib/python3.10/configparser.py", line 1086, in _read
    raise MissingSectionHeaderError(fpname, lineno, line)
configparser.MissingSectionHeaderError: File contains no section headers.
file: 'minimal.ini', line: 4
'   j[General]\n'

    Please do not just post screenshots of commands and error output. It’s OK if you want to also include a screenshot as supplement, but be sure you also include the commands and output as plaintext as well. (If the output is too long for including inline as a comment on the issue, you can save it in a file, and drag-and-drop it into an issue comment).

  • Include environment. When reporting bugs, especially during installation, please run the following and include the output of:

    echo $CPATH
echo $LIBRARY_PATH
echo $PATH
which python

    If you are running on MacOS, and you used the MacPorts installation method, please also run and include the output of:

    port installed

  • Keep each issue focused on one specific problem. Each issue should be focused on one problem. Don’t use an issue for open-ended discussion, or as a place to collect all issues with pypop you run into. If, during the comments, you discover another bug, unrelated to the current issue, please open up a new issue and reference it in the current issue.

  • Run the test suite. In many cases, especially if you are investigating a new platform (e.g. new architecture) developers may ask you run the full test suite via pytest, see run unit tests. in “verbose” mode (i.e. pytest -v). If you do this, please supply the output of the resulting temporary directories on your issue (see the unit test section for more details). Note that you will likely need to clone the main repository as the unit tests are not distributed with the binary wheels.

Documentation improvements

pypop could always use more documentation, whether as part of the official docs, in docstrings, or even on the web in blog posts, articles, and such. Write us a documentation issue describing what you would like to see improved in here.

If you are able to contribute directly (e.g., via a pull request), please read our website contribution guide.

Feature requests and feedback

The best way to send feedback is to file an issue using the feature template.

If you are proposing a feature:

  • Explain in detail how it would work.

  • Keep the scope as narrow as possible, to make it easier to implement.

  • Remember that this is a volunteer-driven project, and that code contributions are welcome

4.2. Making a code contribution

To contribute new code that implement a feature, or fix a bug, this section provides a step-by-step guide to getting you set-up. The main steps are:

  1. forking the repository (or “repo”)

  2. cloning the main repo on to your local machine

  3. making a new branch

  4. installing a development version on your machine

  5. updating your branch when “upstream” (the main repository) has changes to include those changes in your local branch

  6. running code quality checks, like unit tests and pre-commit checks, using nox (or the tools directly)

  7. updating AUTHORS.rst

  8. making a pull request (including a description of your changes suitable for inclusion in NEWS.md)

Fork this repository

Fork this repository before contributing. Forks creates a cleaner representation of the contributions to the project.

Clone the main repository

Next, clone the main repository to your local machine:

git clone https://github.com/alexlancaster/pypop.git
cd pypop

Add your fork as an upstream repository:

git remote add myfork git://github.com/YOUR-USERNAME/pypop.git
git fetch myfork

Make a new branch

From the main branch create a new branch where to develop the new code.

git checkout main
git checkout -b new_branch

Note the main branch is from the main repository.

Build locally and make your changes

Now you are ready to make your changes. First, you need to build pypop locally on your machine, and ensure it works, see the separate section on building and installing a development version.

Once you have done the installation and have verified that it works, you can start to develop the feature, or make the bug fix, and keep regular pushes to your fork with comprehensible commit messages.

git status
git add # (the files you want)
git commit # (add a nice commit message)
git push myfork new_branch

While you are developing, you should use nox to frequently run unit tests and check for code quality (or temporary wheels), as described in use nox for testing and code quality.

Keep your branch in sync with upstream

You should keep your branch in sync with the upstream main branch. For that:

git checkout main  # return to the main branch
git pull  # retrieve the latest source from the main repository
git checkout new_branch  # return to your devel branch
git merge --no-ff main  # merge the new code to your branch

At this point you may need to solve merge conflicts if they exist. If you don’t know how to do this, I suggest you start by reading the official docs

You can push to your fork now if you wish:

git push myfork new_branch

And, continue doing your developments are previously discussed.

Use nox for testing and code quality

In addition to installing PyPop for development (see earlier sections), we have developed configurations for nox to simplify common developer tasks such as running tests, code formatting checks, and documentation builds. nox runs tasks in isolated Python environments, ensuring consistency across contributors and CI pipelines. Install nox (within the same environment as used for your build and install):

pip install nox

You can list the currently configured developer tasks available to run via nox, by calling nox itself:

nox --list

We describe some of the tasks that we recommend contributors run regularly during development of code contributions, below.

Run unit tests

To run the full (pytest-based) unit test suite:

nox --sessions tests

You should also frequently use nox to run the unit tests as you are developing your code, to ensure that you don’t inadvertently break anything, especially before commits, or creating a Pull Request from your branch. (-s is the short option --sessions).

These are exactly the same tests that will be performed online via Github Actions continuous integration (CI). This project follows CI good practices (let us know if something can be improved).

Preserving output from unit tests

Running nox -s tests uses pytest within an isolated environment. To debug tests or preserve output from temporary directories, you may prefer to run pytest directly:

pytest -s -v tests

Supplying the -v verbose option will preserve the run-time output of unit tests that write files to disk in temporary directories unique for each run. The format of the output directories is `run_test_<test-name>_<unique_id>, e.g. the directories created will look similar to the following:

run_test_AlleleColon_HardyWeinberg_u3dnf99y
run_test_USAFEL_49h_exhg

These directories store per-test outputs but are normally deleted unless output is preserved.

If you run into errors during your initial installation, please first carefully repeat and/or check your installation. If you still get errors, file a bug, and include the output of pytest run in verbose mode and capturing the output, as described above.

Run pre-commit checks

All pull requests (PRs) submitted to PyPop will be automatically run through a series pre-configured pre-commit checks (called “hooks”), configured in the .pre-commit-config.yaml YAML file.

To run these checks locally:

nox -s precommit

These checks reformat code and catch errors in:

  • Python code (uses ruff and, ruff-format hooks)

  • C extension code (uses clang-format to format code according to the LLVM style)

  • Common formatting errors in documentation, including Markdown and RST

  • Check code and documentation for spelling errors (via codespell)

Running the precommit checks will result in either:

  1. All checks passing (no action needed)

  2. Some checks fail, this can be due either to:

    • Code being reformatted to coding standards (use git diff to see the additional changes), but are otherwise OK. Generally, all you need to do then is to re-run the nox -s precommit command, and it will proceed according to (1)

    • An error is detected in the code that requires manual intervention (e.g. non-standard Python construct, formatting issue, spelling error). Please fix this and re-run nox -s precommit until it passes.

pre-commit checks are the same checks that will be enforced automatically in your PR via GitHub Actions CI. Running them locally before submitting a PR will speed up acceptance. The automated pre-commit checks will be run just once upon initial PR submission, and results posted on the PR. This may also result in changes to the code (mainly reformatting that can be applied automatically). You will need to ensure that you pull these changes back to your local checkout before applying new changes.

pre-commit Git Hook

We highly recommend you enable pre-commit Git hook in your local checkout, before you commit to your PR branch, so you can catch errors early. Ensuring your code passes pre-commit checks will speed the merging of your PR into the main branch, as the code will already be in a good state for merging.

To enable checks, first ensure that pre-commit is installed, and then install the hooks:

pip install pre-commit
pre-commit install --install-hooks

You can then manually trigger checks using pre-commit (i.e. outside nox):

pre-commit run

If you attempt to commit to the repo, e.g. using a command like git commit -a, pre-commit checks will run on your changed files, and behave as if pre-commit run had been called directly. Once all checks pass the git commit command will commit to the repository and you can git push your changes.

Build distribution packages

To locally build sdist and wheel packages for testing, you can run the build command:

nox -s build

Update AUTHORS.rst

Also add your name to the author table at AUTHORS.rst, so you will also be included in the periodic Zenodo software releases (see also the section on Crediting contributors).

Make a Pull Request

Once you are finished, create a pull request to the main repository and engage with the developers.

When you create the pull request in the initial submission box, you should create a description of your changes with an explanatory bullet list of the contributions. Please note if any of your changes will break existing behaviour or anything else that would be important for an end-user to know. This description should be in Markdown format. Here is an example:

### New features

- here goes my new additions, explain them shortly and well
- this feature will require an an update to your `.ini` file

This will be used to populate the Release Notes and eventually be included in the NEWS.md file.

If you need some code review or feedback while you’re developing the code, you can also make a pull request, even if you’re not fully finished.

However, before submitting a Pull Request, verify your development branch passes all tests as described above . If you are developing new code you should also implement new test cases.

Pull Request checklist

Before requesting a final merge, you should:

  1. Make sure your PR passes all existing pytest tests.

  2. Add unit tests if you are developing new features and make sure these also pass.

  3. Run and address the pre-commit checks as described above.

  4. Update documentation when there’s new API, functionality etc.

  5. In the submission for the PR, include a description of the changes, in markdown format, suitable for eventual inclusion in NEWS.md.

  6. Add yourself to AUTHORS.rst.

Note

Note that the pre-commit checks are automatically run on all new PRs, and this may result in changes to your code, please approve or otherwise ensure these changes make it back into your development branch.

4.3. Installation for developers

Once you have setup your branch as described in making a code contribution, above, you are ready for the four main steps of the developer installation:

  1. install a build environment

  2. build

  3. run tests

Note

Note that you if you need to install PyPop from source, but do not intend to contribute code, you can skip creating your own forking and making an additional branch, and clone the main upstream repository directly:

git clone https://github.com/alexlancaster/pypop.git
cd pypop

For most developers, we recommend using the miniconda approach described below.

Install the build environment

To install the build environment, you should choose either conda or system packages. Once you have chosen and installed the build environment, you should follow the instructions related to the option you chose here in all subsequent steps.

Install build environment via system packages (advanced)

Unix/Linux:

  1. Ensure Python 3 version of pip is installed:

    python3 -m ensurepip --user --no-default-pip

    Note the use of the python3 - you may find this to be necessary on systems which parallel-install both Python 2 and 3, which is typically the case. On newer systems you may find that python and pip are, by default, the Python 3 version of those tools.

  2. Install packages system-wide:

    1. Fedora/Centos/RHEL

      sudo dnf install git swig gsl-devel python3-devel

    2. Ubuntu

      sudo apt install git swig libgsl-dev python-setuptools
MacOS X

  1. Install the developer command-line tools: https://developer.apple.com/downloads/ (includes git, gcc). (Note that you may have to sign-in/create a developer account with Apple using your Apple ID to access this link.). You may also be able to install via the terminal and skip the above step by running xcode-select –-install (but first check to see if you already have a version installed, see https://mac.install.guide/commandlinetools/4.html for more details).

  2. Visit https://www.macports.org and follow the instructions there to install the latest version of MacPorts for your version of MacOS X.

  3. Set environment variables to use macports version of Python and other packages, packages add the following to ~/.bash_profile

    export PATH=/opt/local/bin:$PATH
export LIBRARY_PATH=/opt/local/lib/:$LIBRARY_PATH
export CPATH=/opt/local/include:$CPATH

  4. Rerun your bash shell login in order to make these new exports active in your environment. At the command line type:

    exec bash -login

  5. Install dependencies via MacPorts and set Python version to use (FIXME: currently untested!)

    sudo port install swig-python gsl py39-numpy py39-lxml py39-setuptools py39-pip py39-pytest
sudo port select --set python python39
sudo port select --set pip pip39

  6. Check that the MacPorts version of Python is active by typing: which python, if it is working correctly you should see /opt/local/bin/python.

Windows

You will need a compiler installed, the GitHub Action is tested using Microsoft Visual Studio 17 2022 (you can also download 2019 and earlier versions). We also recommend that you setup the NuGet package repository to install following build-time dependencies.

Note

Please note that we have not directly tested building on standalone Windows machines, only via the GitHub runner workflows. In addition, the ARM64 wheels are cross-compiled on the GitHub runner, which cannot run the resulting wheels, therefore all unit tests are skipped.

  1. Install swig: when compiled on a GitHub runner, the swig package is part of the default image. If compiled on a standalone-mode Windows machine, swig may be available as NuGet package, and installable (untested):

    nuget install swig

  2. Install gsl.

    • X64: install the gsl package:

      nuget install gsl-msvc14-x64 -Version 2.3.0.2779

    • ARM64: The NuGet repository doesn’t have an ARM64 version of gsl, it is necessary to build a .nupkg from source (see details in DEV_NOTES.md). We have made this available in vendor-binaries directory of the repo. To install the package from top-level repository run:

      nuget install gsl-msvc14-arm64 -Source "%CD%\\vendor-binaries

  3. Before starting the build process, you need to modify the environment variables CPATH and LIBRARY_PATH to point to the installed gsl package, e.g. for X64:

    CPATH="gsl-msvc14-x64.2.3.0.2779\\build\\native"
LIBRARY_PATH="gsl-msvc14-x64.2.3.0.2779\\build\\native\\static"

Build PyPop

You should choose either of the following two approaches. Don’t try to mix-and-match the two. The build-and-install approach is only recommended if don’t plan to make any modifications to the code locally.

Cleaning up build

To clean up, first uninstall PyPop (whether you installed in editable mode or not):

pip uninstall pypop-genomics

In addition, to clean-up any compiled files and force a recompilation from scratch, run the clean command:

./setup clean --all

4.4. Install package from GitHub Releases

Packages that are released to PyPI, are also available via the releases on the GitHub release page:

Warning

We recommend installing binary packages using the main PyPI repository, not via the GitHub release packages. However from time to time, we also sometimes make binary packages that are not necessarily also released via PyPI. In addition, if PyPI is unavailable, you may want to install directly from the GitHub release. These instructions will help you do that.

Installing these packages is similar to installing via PyPI, except that you need to explicitly provide a URL to the release page.

  1. First, visit the release page, and choose the release version you wish to install (usually the most recent), and note the release tag (e.g. v1.0.0).

    Release version numbers

    Note that version of the release is slightly different to the git tag. This is because the git tag follows Semantic Versioning, which Python internally normalizes and abbreviates. So the release with the git tag v1.0.0 is actually version 1.0.0 of the pypop-genomics package, and the version that pip “sees” (the difference is more notable with prereleases which might have a git tag of v1.0.0-rc2 but the PyPI version will be 1.0.0rc2).

  2. Next, use pip to install the package by running a command of the form (this will select and install the correct wheel for your Python version and operating system automatically):

    pip install pypop-genomics -f https://github.com/alexlancaster/pypop/releases/expanded_assets/<TAG_NAME>

    where <TAG_NAME> is replaced with a specific tag, e.g. for the example given above, you would run:

    pip install pypop-genomics -f https://github.com/alexlancaster/pypop/releases/expanded_assets/v1.0.0

    You can also manually download the specific wheel from the github release webpage and install directly, e.g.:

    pip install pypop_genomics-1.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

4.5. Making a documentation or website contribution

Interested in maintaining the PyPop website and/or documentation, such as the PyPop User Guide? Here are ways to help.

Overview

All the documentation (including the website homepage) are maintained in this directory (and subdirectories) as reStructuredText (.rst) documents. reStructuredText is very similar to GitHub markdown (.md) and should be fairly self-explanatory to edit (especially for pure text changes). From the .rst “source” files which are maintained here on github, we use sphinx to generate (aka “compile”) the HTML for both the pypop.org user guide and and PDF (via LaTeX) output. We have setup a GitHub action, so that as soon as a documentation source file is changed, it will automatically recompile all the documentation, update the gh-pages branch (which is synced to the GitHub pages) and update the files on the website.

Here’s an overview of the process:

.rst files -> sphinx -> HTML / PDF -> push to gh-pages branch -> publish on pypop.org

This means that any changes to the source will automatically update both website home page the documentation.

Once any changes are pushed to a branch (as described below), the GitHub action will automatically rebuild the website, and the results will be synced to a “staging” version of the website at:

Structure

Here’s an overview of the source files for the website/documentation located in the website subdirectory at the time of writing. Note that some of the documentation and website files, use the include:: directive to include some “top-level” files, located outside website like README.rst and CONTRIBUTING.rst:

  • index.rst (this is the source for the homepage at http://pypop.org/)

  • conf.py (Sphinx configuration file - project name and other global settings are stored here)

  • docs (directory containing the source for the PyPop User Guide, which will eventually live at http://pypop.org/docs).

    • index.rst (source for the top-level of the PyPop User Guide)

    • guide-chapter-install.rst (pulls in parts of the top-level README.rst)

    • guide-chapter-usage.rst

    • guide-chapter-instructions.rst

    • guide-chapter-contributing.rst (pulls in top-level CONTRIBUTING.rst that contains the source of the text that you are reading right now)

    • guide-chapter-changes.rst (pulls in top-level NEWS.md and AUTHORS.rst)

    • licenses.rst (pulls in top-level LICENSE)

    • biblio.rst

    • pypop.bib (BibTeX source file for bibliography)

  • html_root (any files or directories committed in this directory will appear at the top-level of the website)

  • reference (directory containing the old DocBook-based documentation, preserved to allow for unconverted files to be converted later, this directory is ignored by the build process)

Modifying documentation

Minor modifications

For small typo fixes, moderate copyedits at the paragraph level (e.g. adding or modifying paragraphs with little or no embedded markup), you can make changes directly on the github website.

  1. navigate to the .rst file you want to modify in the GitHub code directory, you’ll see a preview of how most of the .rst will be rendered

  2. hover over the edit button - you’ll see an “Edit the file in a fork in your project” (if you are already a project collaborator, you may also have the optional of creating a branch directly in the main repository).

  3. click it and it will open up a window where you can make your changes

  4. make your edits (it’s a good idea to look at the preview tab periodically as you make modifications)

  5. once you’ve finished with the modifications, click “Commit changes”

  6. put in an a commit message, and click “Propose changes”

  7. this will automatically create a new branch in your local fork, and you can immediately open up a pull-request by clicking “Create pull request”

  8. open up a pull-request and submit - new documentation will be automatically built and reviewed. if all is good, it will be merged by the maintainer and made live on the site.

Major modifications

For larger structural changes involving restructuring documentation or other major changes across multiple .rst files, it is highly recommended that you should make all changes in your own local fork, by cloning the repository on your computer and then building the documentation locally. Here’s an overview of how to do that:

  1. make a fork of pypop if you haven’t already (see previous section)

  2. clone the fork and add your fork as an upstream repository on your local computer, and make a new branch. Note that you do not have to build the PyPop software first in order to build the documentation, you can build them separately.

  3. make your changes to your .rst files and/or conf.py

  4. build the HTML documentation, using nox, which creates a temporary virtual environment with sphinx and sphinx extensions (see nox installation):

    nox -s docs

  5. view the local documentation: you can open up browser and navigate to the index.html in the top-level of the newly-created _htmlbuild directory

  6. use git commit to commit your changes to your local fork.

  7. open up a pull-request against the upstream repository

Building the PDF for the PyPop User Guide is a bit more involved, as you will need to have various TeX packages installed.

  1. install the LaTeX packages (these are packages needed for Ubuntu, they may be different on your distribution):

    sudo apt-get install -y latexmk texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended texlive-fonts-extra texlive-luatex texlive-xetex

    Ubuntu LaTeX packages

    The most up to date list of LaTeX packages for Ubuntu are in the “Sphinx build” section of the workflow .github/workflows/documentation.yaml that builds the documentation upon deployment. Consult this if the list of packages, becomes out of date.

  2. build the LaTeX and then compile the PDF, using the nox command:

    nox -s docs_pdf

  3. the user guide will be generated in _latexbuild/pypop-guide.pdf

Note

To change the target output directory of either the HTML or PDF documentation, you can supply the directory name on the nox command-line, after a -- e.g.

nox -s docs -- output_dir

4.6. Crediting contributors

Note

These guidelines were heavily adapted from similar guidelines in the PyGMT project.

We define contributions in a broad way: including both writing code as well as documentation, and reviewing issues and PRs etc. Here are some ways we credit contributors:

AUTHORS.rst, NEWS.md and GitHub Release Notes

Anyone who has contributed a pull request to the project is welcome to add themselves (or request to be added) to AUTHORS.rst, which is part of the repository and included with with distributions.

Every time we make a release, everyone who has made a commit to the repository since the previous release will be mentioned in either the NEWS.md or in the GitHub Release Notes.

Authorship on Zenodo archives of releases

Anyone who has contributed to the repository (i.e., appears on git log) will be invited to be an author on the Zenodo archive of new releases.

To be included as an author, you must add the following to the AUTHORS.rst file of the repository:

  1. Full name (and optional link to your website or GitHub page)

  2. ORCID (optional)

  3. Affiliation (optional)

The order of authors is generally defined by the number of commits to the repository (git shortlog -sne). The order can also be changed on a case-by-case basis, such as contributions to PyPop project that due not relate to commit numbers, such as writing grants/proposals, and other programming efforts (including reviewing PRs).

If you have contributed and do not wish to be included in Zenodo archives, either don’t add yourself to AUTHORS.rst, or open an issue or file a PR that:

  1. Removes yourself from AUTHORS.rst, or;

  2. Indicates next to your name on AUTHORS.rst that you do not wish to be included with something like (not included in Zenodo).

Note that authors included in the Zenodo archive will also have their name listed in the CITATION.cff file. This is a machine (and human) readable file that enables citation of PyPop easily.

Scientific publications (papers)

From time to time we may write academic papers for PyPop, e.g., for major changes or significant new components of the package.

To be included as an author on the paper, you must have

  1. either made multiple and regular contributions to the PyPop repository; or, have made other non-coding contributions (or both);

  2. have participated in the writing and reviewing of the paper.

  3. added your full name, affiliation, and (optionally) ORCID to the paper.

  4. written and/or read and review the manuscript in a timely manner and provide comments on the paper