How to Do a Release

Periodically, the yt development community issues new releases. Since yt follows semantic versioning, the type of release can be read off from the version number used. Version numbers should follow the scheme MAJOR.MINOR.PATCH. There are three kinds of possible releases:

  • Bugfix releases

    These releases are regularly scheduled and will optimally happen approximately once a month. These releases should contain only fixes for bugs discovered in earlier releases and should not contain new features or API changes. Bugfix releases should increment the PATCH version number. Bugfix releases should not be generated by merging from the master branch, instead bugfix pull requests should be manually backported. Version 3.2.2 is a bugfix release.

  • Minor releases

    These releases happen when new features are deemed ready to be merged into the stable branch and should not happen on a regular schedule. Minor releases can also include fixes for bugs if the fix is determined to be too invasive for a bugfix release. Minor releases should not include backwards-incompatible changes and should not change APIs. If an API change is deemed to be necessary, the old API should continue to function but might trigger deprecation warnings. Minor releases should happen by merging the master branch into the stable branch. Minor releases should increment the MINOR version number and reset the PATCH version number to zero. Version 3.3.0 is a minor release.

  • Major releases

    These releases happen when the development community decides to make major backwards-incompatible changes. In principle a major version release could include arbitrary changes to the library. Major version releases should only happen after extensive discussion and vetting among the developer and user community. Like minor releases, a major release should happen by merging the master branch into the stable branch. Major releases should increment the MAJOR version number and reset the MINOR and PATCH version numbers to zero. If it ever happens, version 4.0.0 will be a major release.

The job of doing a release differs depending on the kind of release. Below, we describe the necessary steps for each kind of release in detail.

Doing a Bugfix Release

As described above, bugfix releases are regularly scheduled updates for minor releases to ensure fixes for bugs make their way out to users in a timely manner. Since bugfix releases should not include new features, we do not issue bugfix releases by simply merging from the development master branch into the stable branch. Instead, we manually cherry-pick bugfixes from the from master branch onto the stable branch.

You may find the pr_backport.py script located in the scripts folder at the root of the repository to be helpful. This script uses the github API to find the list of pull requests made since the last release and prompts the user to backport each pull request individually. Note that the backport process is fully manual. The easiest way to do it is to download the diff for the pull request (the URL for the diff is printed out by the backport script) and then use git apply to apply the patch for the pull request to a local copy of yt with the stable branch checked out.

Once you’ve finished backporting push your work to Github. Once you’ve pushed to your fork, you will be able to issue a pull request containing the backported fixes just like any other yt pull request.

Doing a Minor or Major Release

This is much simpler than a bugfix release. All that needs to happen is the master branch must get merged into the stable branch, and any conflicts that happen must be resolved, almost always in favor of the state of the code on the master branch.

Incrementing Version Numbers and Tagging a Release

Before creating the tag for the release, you must increment the version numbers that are hard-coded in a few files in the yt source so that version metadata for the code is generated correctly. This includes things like yt.__version__ and the version that gets read by the Python Package Index (PyPI) infrastructure.

The paths relative to the root of the repository for the three files that need to be edited are:

  • doc/source/conf.py

    The version and release variables need to be updated.

  • setup.py

    The VERSION variable needs to be updated

  • yt/__init__.py

    The __version__ variable must be updated.

Once these files have been updated, commit these updates. This is the commit we will tag for the release.

To actually create the tag, issue the following command from the stable branch:

git tag <tag-name>

Where <tag-name> follows the project’s naming scheme for tags (e.g. yt-3.2.1). Once you are done, you will need to push the tag to github:

git push origin --tags

This assumes that you have configured the remote origin to point at the main yt git repository. If you are doing a minor or major version number release, you will also need to update back to the development branch and update the development version numbers in the same files.

Uploading to yt-project.org

Before uploading the release to the Python Package Index (pypi.org) we will first upload the package to yt-project.org. This facilitates building binary wheels for pypi and binary conda packages on conda-forge before doing the “official” release. This also ensures that there isn’t a period of time when users do pip install yt and end up downloading the source distribution instead of one of the binary wheels.

To create the source distribution, issue the following command in the root of the yt repository:

$ python setup.py sdist

This will generate a tarball in a dist/ directory located in the root of the repository.

Access to yt-project.org mediated via SSH login. Please contact one of the current yt developers for access to the webserver running yt-project.org if you do not already have it. You will need a copy of your SSH public key so that your key can be added to the list of authorized keys. Once you login, use e.g. scp to upload a copy of the souce distribution tarball to https://yt-project.org/sdist, like so:

$ scp dist/yt-3.5.1.tar.gz yt_analysis@dickenson.dreamhost.com:yt-project.org/sdist

You may find it helpful to set up an ssh config for dickenson to make this command a bit easier to execute.

Updating conda-forge and building wheels

Before we finish the release, we need to generate new binary builds by updating yt’s conda-forge feedstock and the yt-wheels repository.

conda-forge

Conda-forge packages for yt are managed via the yt feedstock, located at https://github.com/conda-forge/yt-feedstock. To update the feedstock, you will need to update the meta.yaml file located in the recipe folder in the root of the feedstock repository. Most likely you will only need to update the version number and the SHA256 hash of the tarball. If yt’s dependencies change you may also need to update the recipe. Once you have updated the recipe, propose a pull request on github and merge it once all builds pass.

Wheels and multibuild

Binary wheels for yt are managed via the multibuild project. For yt the main point of access is at https://github.com/yt-project/yt-wheels. Take a look at the pull requests from the previous few releases to get an idea of what to do, but briefly you will need to update the multibuild and yt submodules to their latest state and then commit the changes to the submodules:

$ cd multibuild
$ git pull origin devel
$ cd ../yt
$ git pull origin stable
$ cd ..
$ git commit -am "updating multibuild and yt submodules"

Next you will need to update the .travis.yml and appveyor.yaml files to build the latest tag of yt. You may also need to update elsewhere in the file if yt’s dependencies changed or if yt dropped or added support for a Python version. To generate new wheels you need to push the changes to GitHub. A good process to follow is to first submit a pull request to test the changes and make sure the wheels can be built. Once they pass, you can merge the changes into master and wait for the wheel files to be uploaded to http://wheels.scipy.org (note that the wheels will not be uploaded until the changes have been merged into master). Once the wheels are uploaded, download the wheel files for the release and copy them to the dist folder in the yt repository so that they are sitting next to the source distribution we created earlier. Here’s a one-liner to download all of the wheels for the yt 3.5.1 release:

$ wget -r --no-parent -A 'yt-3.5.1-*.whl' http://wheels.scipy.org/

Uploading to PyPI

To actually upload the release to the Python Package Index, you just need to issue the following command:

twine upload dist/*

Please ensure that both the source distribution and binary wheels are present in the dist folder before doing this. Directions on generating binary wheels are described in the section immediately preceding this one.

You will be prompted for your PyPI credentials and then the package should upload. Note that for this to complete successfully, you will need an account on PyPI and that account will need to be registered as an “owner” or “maintainer” of the yt package. Right now the following people have access to upload packages: Matt Turk, Britton Smith, Nathan Goldbaum, John ZuHone, Kacper Kowalik, and Madicken Munk. The yt package source distribution should be uploaded along with compiled binary wheel packages for various platforms that we support.

After the release is uploaded to PyPI, you should send out an announcement e-mail to the yt mailing lists as well as other possibly interested mailing lists for all but bugfix releases.