See the Scikit-HEP Developer introduction for a detailed description of best practices for developing Scikit-HEP packages.

Contributing#

Building from source#

This repository has dependencies in submodules. Check out the repository like this:

git clone --recursive https://github.com/scikit-hep/boost-histogram.git
cd boost-histogram
Faster version (click to expand) 
git clone https://github.com/scikit-hep/boost-histogram.git
cd boost-histogram
git submodule update --init --depth 10

Setting up a development environment#

Nox#

The fastest way to start with development is to use nox. If you don’t have nox, you can use pipx run nox to run it without installing, or pipx install nox. If you don’t have pipx (pip for applications), then you can install with with pip install pipx (the only case were installing an application with regular pip is reasonable). If you use macOS, then pipx and nox are both in brew, use brew install pipx nox.

To use, run nox. This will lint and test using every installed version of Python on your system, skipping ones that are not installed. You can also run specific jobs:

$ nox -l # List all the defined sessions
$ nox -s lint  # Lint only
$ nox -s tests-3.9  # Python 3.9 tests only
$ nox -s docs -- serve  # Build and serve the docs
$ nox -s make_pickle  # Make a pickle file for this version

Nox handles everything for you, including setting up an temporary virtual environment for each run.

Pip#

While developers often work in CMake, the “correct” way to develop a python package is in a virtual environment. This is how you would set one up with Python’s built-in venv:

python3 -m venv .env
source ./.env/bin/activate
pip install -U pip
pip install -ve .[all]
Optional: External Jupyter kernel (click to expand)

You can set up a kernel for external Jupyter then deactivate your environment:

python -m ipykernel install --user --name boost-hist
deactivate

Now, you can run notebooks using your system JupyterLab, and it will list the environment as available!

To rebuild, rerun pip install -ve . from the environment, if the commit has changed, you will get a new build. Due to the -e, Python changes do not require a rebuild.

CMake#

CMake is common for C++ development, and ties nicely to many C++ tools, like IDEs. If you want to use it for building, you can. Make a build directory and run CMake. If you have a specific Python you want to use, add -DPYTHON_EXECUTABLE=$(which python) or similar to the CMake line. If you need help installing the latest CMake version, visit this page; one option is to use pip to install CMake.

Note: Since setuptools uses a subdirectory called build, it is slightly better to avoid making your CMake directory build as well. Also, you will often have multiple CMake directories (build-release, build-debug, etc.), so avoiding the descriptive name build is not a bad idea.

You have three options for running code in python:

  1. Run from the build directory (only works with some commands, like python -m pytest, and not others, like pytest

  2. Add the build directory to your PYTHONPATH environment variable

  3. Set CMAKE_INSTALL_PREFIX to your site-packages and install (recommended for virtual environments).

Here is the recommendation for a CMake install:

python3 -m venv env_cmake
source ./env_cmake/bin/activate
pip install -r dev-requirements.txt
cmake -S . -B build-debug \
    -GNinja \
    -DCMAKE_INSTALL_PREFIX=$(python -c "import distutils.sysconfig; print(distutils.sysconfig.get_python_lib(plat_specific=False,standard_lib=False))")
cmake --build build-debug -j4
cmake --install build-debug # Option 3 only

Note that option 3 will require reinstalling if the python files change, while options 1-2 will not if you have a recent version of CMake (symlinks are made).

This could be simplified if pybind11 supported the new CMake FindPython tools.

Testing#

Run the unit tests (requires pytest and NumPy).

python3 -m pytest

For CMake, you can also use the test target from anywhere, or use python3 -m pytest or ctest from the build directory.

The build requires setuptools_scm. The tests require numpy, pytest, and pytest-benchmark. pytest-sugar adds some nice formatting.

Benchmarking#

You can enable benchmarking with --benchmark-enable when running tests. You can also run explicit performance tests with scripts/performance_report.py.

python3 -m pytest --benchmark-enable --benchmark-sort fullname

For example, if you want to benchmark before and after a change:

python3 -m pytest --benchmark-enable --benchmark-autosave
# Make change
python3 -m pytest --benchmark-enable --benchmark-autosave

pytest-benchmark compare 0001 0002 --sort fullname --histogram

Note, while the histogram option (--histogram) is nice, it does require pygal and pygaljs to be installed. Feel free to leave it off if not needed.

Formatting#

Code should be well formatted; CI will check it and one of the authors can help reformat your code. If you want to check it yourself, you should use pre-commit.

Just install pre-commit, probably using brew on macOS or pip on other platforms, then run:

pre-commit install

Now all changed files will be checked every time you git commit. You can check it yourself (even without installing the hooks) using:

pre-commit run --all-files

We do not check check-manifest every time locally, since it is slow. You can trigger this manual check with:

pre-commit run --all-files --hook-stage manual check-manifest

Developers should update the pre-commit dependencies once in a while, you can do this automatically with:

pre-commit autoupdate

Note about skipping Docker

Pre-commit uses docker to ensure a consistent run of clang-format. If you do not want to install/run Docker, you should use SKIP=docker-clang-format when running pre-commit, and instead run clang-format -style=file -i <files> yourself.

Clang-Tidy#

To run Clang tidy, the following recipe should work. Files will be modified in place, so you can use git to monitor the changes.

docker run --rm -v $PWD:/pybind11 -it silkeh/clang:10
apt-get update && apt-get install python3-dev
cmake -S pybind11/ -B build -DCMAKE_CXX_CLANG_TIDY="$(which clang-tidy);-fix"
cmake --build build

Remember to build single-threaded if applying fixes!

Include what you use#

To run include what you use, install (brew install include-what-you-use on macOS), then run:

cmake -S . -B build-iwyu -DCMAKE_CXX_INCLUDE_WHAT_YOU_USE=$(which include-what-you-use)
cmake --build build

Common tasks#

Updating dependencies (click to expand)

This will checkout new versions of the dependencies. Example given using the fish shell.

for f in *
    cd $f
    git fetch
    git checkout boost-1.75.0 || echo "Not found"
    cd ..
end
Making a new release (click to expand)

  • Finish merging open PRs that you want in the new version

  • Add most recent changes to the docs/CHANGELOG.md

  • Sync master with develop using git checkout master; git merge develop --ff-only and push

  • Make sure the full wheel build runs on master without issues (will happen automatically on push to master)

  • Make the GitHub release in the GitHub UI. Copy the changelog entries and links for that version; this has to be done as part of the release and tag procedure for archival tools (Zenodo) to pick them up correctly.

    • Title should be "Version <version number>"

    • Version tag should be "v" + major + "." + minor + "." + patch.

  • GHA will build and send to PyPI for you when you release.

  • Conda-forge will automatically make a PR to update within an hour or so, and it will merge automatically if it passes.

Making a compiler flamegraph (click to expand)

This requires LLVM 9+, and is based on this post.

brew install llvm         # macOS way to get clang-9
python3 -m venv .env_core # general environment (no install will be made)
. .env_core/bin/activate
pip install -r dev-requirements.txt
CXX="/usr/local/opt/llvm/bin/clang++" cmake -S . -B build-llvm \
    -DCMAKE_CXX_FLAGS="-ftime-trace" \
    -DPYTHON_EXECUTABLE=$(which python)
cmake --build build-llvm/

Now open a browser with SpeedScope, and load one of the files.

Adding a contributor (click to expand)

First, you need to install the all contributor CLI:

yarn add --dev all-contributors-cli

Then, you can add contributors:

yarn all-contributors add henryiii maintenance,code,doc