See the Scikit-HEP Developer introduction for a detailed description of best practices for developing Scikit-HEP packages.
Building from source#
This repository has dependencies in submodules. Check out the repository like this:
git clone --recursive https://github.com/scikit-hep/boost-histogram.git cd boost-histogram
Faster version (click to expand)
git clone https://github.com/scikit-hep/boost-histogram.git cd boost-histogram git submodule update --init --depth 10
Setting up a development environment#
The fastest way to start with development is to use nox. If you don’t have nox,
you can use
pipx run nox to run it without installing, or
pipx install nox.
If you don’t have pipx (pip for applications), then you can install with with
pip install pipx (the only case were installing an application with regular
pip is reasonable). If you use macOS, then pipx and nox are both in brew, use
brew install pipx nox.
To use, run
nox. This will lint and test using every installed version of
Python on your system, skipping ones that are not installed. You can also run
$ nox -l # List all the defined sessions $ nox -s lint # Lint only $ nox -s tests-3.9 # Python 3.9 tests only $ nox -s docs -- serve # Build and serve the docs $ nox -s make_pickle # Make a pickle file for this version
Nox handles everything for you, including setting up an temporary virtual environment for each run.
While developers often work in CMake, the “correct” way to develop a python package is in a virtual environment. This is how you would set one up with Python’s built-in venv:
python3 -m venv .env source ./.env/bin/activate pip install -U pip pip install -ve .[all]
Optional: External Jupyter kernel (click to expand)
You can set up a kernel for external Jupyter then deactivate your environment:
python -m ipykernel install --user --name boost-hist deactivate
Now, you can run notebooks using your system JupyterLab, and it will list the environment as available!
To rebuild, rerun
pip install -ve . from the environment, if the commit has
changed, you will get a new build. Due to the
-e, Python changes do not require
CMake is common for C++ development, and ties nicely to many C++ tools, like
IDEs. If you want to use it for building, you can. Make a build directory and
run CMake. If you have a specific Python you want to use, add
-DPYTHON_EXECUTABLE=$(which python) or similar to the CMake line. If you need
help installing the latest CMake version, visit this
one option is to use pip to install CMake.
Note: Since setuptools uses a subdirectory called
build, it is slightly better to avoid making your CMake directory
buildas well. Also, you will often have multiple CMake directories (
build-debug, etc.), so avoiding the descriptive name
buildis not a bad idea.
You have three options for running code in python:
Run from the build directory (only works with some commands, like
python -m pytest, and not others, like
Add the build directory to your PYTHONPATH environment variable
CMAKE_INSTALL_PREFIXto your site-packages and install (recommended for virtual environments).
Here is the recommendation for a CMake install:
python3 -m venv env_cmake source ./env_cmake/bin/activate pip install -r dev-requirements.txt cmake -S . -B build-debug \ -GNinja \ -DCMAKE_INSTALL_PREFIX=$(python -c "import distutils.sysconfig; print(distutils.sysconfig.get_python_lib(plat_specific=False,standard_lib=False))") cmake --build build-debug -j4 cmake --install build-debug # Option 3 only
Note that option 3 will require reinstalling if the python files change, while options 1-2 will not if you have a recent version of CMake (symlinks are made).
This could be simplified if pybind11 supported the new CMake FindPython tools.
Run the unit tests (requires pytest and NumPy).
python3 -m pytest
For CMake, you can also use the
test target from anywhere, or use
python3 -m pytest or
ctest from the build directory.
The build requires
setuptools_scm. The tests require
pytest-sugar adds some nice formatting.
You can enable benchmarking with
--benchmark-enable when running tests. You
can also run explicit performance tests with
python3 -m pytest --benchmark-enable --benchmark-sort fullname
For example, if you want to benchmark before and after a change:
python3 -m pytest --benchmark-enable --benchmark-autosave # Make change python3 -m pytest --benchmark-enable --benchmark-autosave pytest-benchmark compare 0001 0002 --sort fullname --histogram
Note, while the histogram option (
--histogram) is nice, it does require
pygaljs to be installed. Feel free to leave it off if not needed.
Code should be well formatted; CI will check it and one of the authors can help
reformat your code. If you want to check it yourself, you should use
Just install pre-commit, probably using brew on macOS or pip on other platforms, then run:
Now all changed files will be checked every time you git commit. You can check it yourself (even without installing the hooks) using:
pre-commit run --all-files
We do not check
check-manifest every time locally, since it is slow. You can trigger
this manual check with:
pre-commit run --all-files --hook-stage manual check-manifest
Developers should update the pre-commit dependencies once in a while, you can do this automatically with:
Note about skipping Docker
Pre-commit uses docker to ensure a consistent run of clang-format. If you do not want to install/run Docker, you should use
SKIP=docker-clang-formatwhen running pre-commit, and instead run
clang-format -style=file -i <files>yourself.
To run Clang tidy, the following recipe should work. Files will be modified in place, so you can use git to monitor the changes.
docker run --rm -v $PWD:/pybind11 -it silkeh/clang:10 apt-get update && apt-get install python3-dev cmake -S pybind11/ -B build -DCMAKE_CXX_CLANG_TIDY="$(which clang-tidy);-fix" cmake --build build
Remember to build single-threaded if applying fixes!
Include what you use#
To run include what you use, install (
brew install include-what-you-use on
macOS), then run:
cmake -S . -B build-iwyu -DCMAKE_CXX_INCLUDE_WHAT_YOU_USE=$(which include-what-you-use) cmake --build build
Updating dependencies (click to expand)
This will checkout new versions of the dependencies. Example given using the fish shell.
for f in * cd $f git fetch git checkout boost-1.75.0 || echo "Not found" cd .. end
Making a new release (click to expand)
Finish merging open PRs that you want in the new version
Add most recent changes to the
Sync master with develop using
git checkout master; git merge develop --ff-onlyand push
Make sure the full wheel build runs on master without issues (will happen automatically on push to master)
Make the GitHub release in the GitHub UI. Copy the changelog entries and links for that version; this has to be done as part of the release and tag procedure for archival tools (Zenodo) to pick them up correctly.
Title should be
"Version <version number>"
Version tag should be
"v" + major + "." + minor + "." + patch.
GHA will build and send to PyPI for you when you release.
Conda-forge will automatically make a PR to update within an hour or so, and it will merge automatically if it passes.
Making a compiler flamegraph (click to expand)
This requires LLVM 9+, and is based on this post.
brew install llvm # macOS way to get clang-9 python3 -m venv .env_core # general environment (no install will be made) . .env_core/bin/activate pip install -r dev-requirements.txt CXX="/usr/local/opt/llvm/bin/clang++" cmake -S . -B build-llvm \ -DCMAKE_CXX_FLAGS="-ftime-trace" \ -DPYTHON_EXECUTABLE=$(which python) cmake --build build-llvm/
Now open a browser with SpeedScope, and load one of the files.
Adding a contributor (click to expand)
First, you need to install the all contributor CLI:
yarn add --dev all-contributors-cli
Then, you can add contributors:
yarn all-contributors add henryiii maintenance,code,doc