Documentation

How to Write Documentation

Writing documentation is one of the most important but often overlooked tasks for increasing yt’s impact in the community. It is the way in which the world will understand how to use our code, so it needs to be done concisely and understandably. Typically, when a developer submits some piece of code with new functionality, the developer should also include documentation on how to use that functionality (as per Requirements for Code Submission). Depending on the nature of the code addition, this could be a new narrative docs section describing how the new code works and how to use it, it could include a recipe in the cookbook section, or it could simply be adding a note in the relevant docs text somewhere.

The documentation exists in the main code repository for yt in the doc directory (i.e. $YT_GIT/doc/source where $YT_GIT is the path of the yt git repository). It is organized hierarchically into the main categories of:

  • Visualizing

  • Analyzing

  • Analysis Modules

  • Examining

  • Cookbook

  • Quickstart

  • Developing

  • Reference

  • FAQ

  • Help

You will have to figure out where your new/modified doc fits into this, but browsing through the existing documentation is a good way to sort that out.

All the source for the documentation is written in Sphinx, which uses ReST for markup. ReST is very straightforward to markup in a text editor, and if you are new to it, we recommend just using other .rst files in the existing yt documentation as templates or checking out the ReST reference documentation.

New cookbook recipes (see The Cookbook) are very helpful for the community as they provide simple annotated recipes on how to use specific functionality. To add one, create a concise Python script which demonstrates some functionality and pare it down to its minimum. Add some comment lines to describe what it is that you’re doing along the way. Place this .py file in the source/cookbook/ directory, and then link to it explicitly in one of the relevant .rst files in that directory (e.g. complex_plots.rst, etc.), and add some description of what the script actually does. We recommend that you use one of the sample data sets in your recipe. When the full docs are built, each of the cookbook recipes is executed dynamically on a system which has access to all of the sample datasets. Any output images generated by your script will then be attached inline in the built documentation directly following your script.

After you have made your modifications to the docs, you will want to make sure that they render the way you expect them to render. For more information on this, see the section on Building the Documentation. Unless you’re contributing cookbook recipes or notebooks which require a dynamic build, you can probably get away with just doing a ‘quick’ docs build.

When you have completed your documentation additions, commit your changes to your repository and make a pull request in the same way you would contribute a change to the codebase, as described in the section on Making and Sharing Changes.

Building the Documentation

The yt documentation makes heavy use of the Sphinx documentation automation suite. Sphinx, written in Python, was originally created for the documentation of the Python project and has many nice capabilities for managing the documentation of Python code.

While much of the yt documentation is static text, we make heavy use of cross-referencing with API documentation that is automatically generated at build time by Sphinx. We also use Sphinx to run code snippets (e.g. the cookbook and the notebooks) and embed resulting images and example data.

Essential tools for building the docs can be installed alongside yt itself. From the top level of a local copy, run

$ python -m pip install -e . -r requirements/docs.txt

Quick versus Full Documentation Builds

Building the entire set of yt documentation is a laborious task, since you need to have a large number of packages in order to successfully execute and render all of the notebooks and yt recipes drawing from every corner of the yt source. As a quick alternative, one can do a quick build of the documentation, which eschews the need for downloading all of these dependencies, but it only produces the static docs. The static docs do not include the cookbook outputs and the notebooks, but this is good enough for most cases of people testing out whether or not their documentation contributions look OK before submitting them to the yt repository.

If you want to create the full documentation locally, then you’ll need to follow the instructions for building the full docs, so that you can dynamically execute and render the cookbook recipes, the notebooks, etc.

Building the Docs (Quick)

In order to tell Sphinx not to do all of the dynamic building, you must set the $READTHEDOCS environment variable to be True by run from the command line (using bash syntax for example), as

export READTHEDOCS=True

This variable is set for automated builds on the free ReadTheDocs service but can be used by anyone to force a quick, minimal build.

Now all you need to do is execute Sphinx on the yt doc source. Go to the documentation directory and build the docs:

cd $YT_GIT/doc
make html

This will produce an html version of the documentation locally in the $YT_GIT/doc/build/html directory. You can now go there and open up index.html or whatever file you wish in your web browser.

Building the Docs (Full)

As alluded to earlier, building the full documentation is a bit more involved than simply building the static documentation.

The full documentation makes heavy use of custom Sphinx extensions to transform recipes, notebooks, and inline code snippets into Python scripts, IPython notebooks, or notebook cells that are executed when the docs are built.

To do this, we use Jupyter’s nbconvert module to transform notebooks into HTML. to simplify versioning of the notebook JSON format, we store notebooks in an unevaluated state.

To build the full documentation, you will need yt, jupyter, and all dependencies needed for yt’s analysis modules installed. The following dependencies were used to generate the yt documentation during the release of yt 3.2 in 2015.

  • Sphinx 1.3.1

  • Jupyter 1.0.0

  • RunNotebook 0.1

  • pandoc 1.13.2

  • Rockstar halo finder 0.99.6

  • SZpack 1.1.1

  • ffmpeg 2.7.1 (compiled with libvpx support)

  • Astropy 0.4.4

You will also need the full yt suite of yt test data, including the larger datasets that are not used in the answer tests.

You will need to ensure that your testing configuration is properly configured and that all of the yt test data is in the testing directory. See How to Run the Answer Tests for more details on how to set up the testing configuration.

Now that you have everything set up properly, go to the documentation directory and build it using Sphinx:

cd $YT_GIT/doc
make html

If all of the dependencies are installed and all of the test data is in the testing directory, this should churn away for a while (several hours) and eventually generate a docs build. We suggest setting suppress_stream_logging = True in your yt configuration (See The Configuration) to suppress large amounts of debug output from yt.

To clean the docs build, use make clean.

Building the Docs (Hybrid)

It’s also possible to create a custom Sphinx build that builds a restricted set of notebooks or scripts. This can be accomplished by editing the Sphinx conf.py file included in the source directory at the top level of the docs. The extensions included in the build are contained in the extensions list. To disable an extension, simply remove it from the list. Doing so will raise a warning when Sphinx encounters the directive in the docs and will prevent Sphinx from evaluating the directive.

As a concrete example, if one wanted to include the notebook, and notebook-cell directives, but not the python-script or autosummary directives, one would just need to comment out the lines that append these extensions to the extensions list. The resulting docs build will be significantly quicker since it would avoid executing the lengthy API autodocumentation as well as a large number of Python script snippets in the narrative docs.