# Contributing [](TOC) - [Contributing](#contributing) - [Introduction](#introduction) - [Setup](#setup) - [Setup SSH keys](#setup-ssh-keys) - [Setup Virtual Environments](#setup-virtual-environments) - [Project Types](#project-types) - [Project Layout](#project-layout) - [Dependency Management](#dependency-management) - [These are not used on production, or staging, only](#these-are-not-used-on-production-or-staging-only) - [on development machines and the CI environment.](#on-development-machines-and-the-ci-environment) - [These are the requirements produced for specific builds. They can be](#these-are-the-requirements-produced-for-specific-builds-they-can-be) - [used to debug version compatatbility issues . They are generated](#used-to-debug-version-compatatbility-issues-they-are-generated) - [using pip freeze](#using-pip-freeze) - [Vendoring](#vendoring) - [Development](#development) - [Linting](#linting) - [Type Checking](#type-checking) - [Documentation](#documentation) - [Setup to run docker](#setup-to-run-docker) - [PyCharm](#pycharm) - [Sublime Text](#sublime-text) - [Best Practices](#best-practices) [](TOC) ## Introduction Friction for new contributors should be as low as possible. Ideally a new contributor, starting any unix[^1] system can go through these steps and not encounter any errors: 1. `git clone ` 2. `cd ` 3. `make install` 4. `# get some coffee` 5. `make lint` 6. `make test` 7. `make serve` If you as a new contributor encounter any errors, then please create an issue report and you will already have made a great contribution! ## Setup The development workflow described here is documented based on a Unix environment. Hopefully this will reduce discrepancies between development and production systems. ### Setup SSH keys Projects which depend on private repositories require ssh to connect to remote servers. If this is the case, you should make sure that your ssh keys are available in `${HOME}/.ssh`, or you will have to do `ssh-keygen` and install the generated public key to host system. If this is not done, `pip install` will fail to install these dependencies from your private repositiories with an error like this ```shell Downloading/unpacking git+git://...git Cloning Git repository git:// Permission denied (publickey). fatal: The remote end hung up unexpectedly ---------------------------------------- Command /usr/local/bin/git clone ... failed with error code 128 ``` ### Setup Virtual Environments The first setup can take a while, since it will install miniconda and download lots of dependencies for the first time. If you would like to know more about conda, there is a good article written by Gergely Szerovay: https://medium.freecodecamp.org/85f155f4353c ```shell dev@host:~ $ git clone git@../group/project.git Cloning Git repository git@../group/project.git to project ... $ cd project dev@host:~/project $ make install Solving environment: ... ``` This will do quite a few things. 1. Install miniconda3, if it isn't already installed. It checks the path `$HOME/miniconda3` for an existing installation 2. Creates python virtual environments for all supported python versions of this project. 3. Installs application and development dependencies to the environments. 4. Installs vendored dependencies into `vendor/` If installation was successful, you should be able to at least run the linter (assuming previous developers have a bare minimum of diligence). ```console $ make lint flake8 .. ok mypy .... ok doc ..... ok ``` If this is the first time conda has been installed on your system, you'll probably want to enable the `conda` command: ``` $ echo ". ${HOME}/miniconda3/etc/profile.d/conda.sh" >> ~/.bashrc $ conda --version conda 4.5.11 ``` You can also activate the default virtual environment as follows. ```shell (myproject_py36) dev@host:~/myproject $ source ./activate $ which python /home/dev/miniconda3/envs/myproject_py36/bin/python $ ipython Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51) t Type 'copyright', 'credits' or 'license' for more information IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help. In [1]: import sys In [2]: sys.path Out[2]: ['/home/dev/miniconda3/envs/pycalver_py36/bin', '/home/dev/myproject/src', '/home/dev/myproject/vendor', ... In [3]: import myproject In [4]: myproject.__file__ Out[4]: '/home/dev/myproject/src/myproject/__init__.py' ``` Note that the `PYTHONPATH` has been set up to import modules of the project. You can review the definition for `make ipy` to see how to set up `PYTHONPATH` correctly. ```shell $ make ipy --dry-run ENV=${ENV-dev} PYTHONPATH=src/:vendor/:$PYTHONPATH \ /home/dev/miniconda3/envs/myproject_py36/bin/ipython $ make ipy Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51) Type 'copyright', 'credits' or 'license' for more information IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help. In [1]: import myproject In [2]: myproject.__file__ Out[2]: '/home/dev/myproject/src/myproject/__init__.py' ``` ## Project Types These guidelines written for different kinds of projects, each of which is ideally: small, focosued and reusable. These projects can be: 1. Services: Projects which are deployed and run continuously. 2. Libraries: Projects which are not deployed by themselves but installed and used by others. 3. CLI Tools: Projects which are installed and mainly used by developers and admins. The choices made here are intended to make it easy to start new projects by reducing the burdon of project setup to a minimum. ## Project Layout src/ # source code of project vendor/ # vendored dependencies stubs/ # mypy .pyi stub files test/ # pytest test files (files begin with test_) scripts/ # miscalenious scripts used deployment and ops requirements/ # dependency metadata files docs/ # documentation source files data/ # fixtures for unit tests and db initialization setup.py # main python package metadata setup.cfg # misc python tooling configuration README.md # project overview and status CONTRIBUTING.md # guide for developers CHANGELOG.md # short documentation of release history LICENSE # for public libraries (MIT preferred) makefile # project specific configuration # variables and make targets makefile.bootstrapit.make # bootstrapit make include library docker_base.Dockerfile # base image for CI (only conda envs) Dockerfile # image with source of the project ### Dependency Management Dependencies are managed using a set of requirements/\*.txt files. You only need to know about this if you want to add or change a dependency. ```shell requirements/conda.txt # installed via conda from main or conda-forge requirements/pypi.txt # installed via pip from pypi to virutal environments requirements/vendor.txt # installed via pip from pypi to vendor/ # These are not used on production, or staging, only # on development machines and the CI environment. requirements/development.txt # useful packgages for development/debugging requirements/integration.txt # used for linting/testing/packaging # These are the requirements produced for specific builds. They can be # used to debug version compatatbility issues. They are generated # using make freeze requirements/20190214t212403_freeze.txt ``` When adding a new dependency please consider: - Only specify direct dependencies of the project, not transitive dependencies of other projects. These are installed via their own respective dependency declarations. - Whenever possible, the specifier for a package should be only its name without a version specifier. With this as the default, the project remains up to date in terms of security fixes and other library improvements. - Some packages consider some of their dependancies to be optional, in which case you will have to specify their transitive dependencies. - Only specify/pin/freeze a specific (older) version if there are known issues, or your project requires features from an unstable (alpha/beta) version of the package. Each pinned version should document why it was pinned, so that it can later be determined if the issue has been resolved in the meantime. One argument against this approach is the issue of rogue package maintainers. A package maintainer might release a new version which you automatically install using `make update`, and this new code opens a back door or proceeds to send data from your production system to a random server on the internet. The only prodection pypi or conda-forge have against this is to remove packages that are reported to them. If you are paranoid, you could start pinning dependencies to older versions, for which you feel comfortable that any issues would have been noticed. This is only a half measure however, since the issues may not be noticed even after months. Ultimately, if data breaches are a concern you should talk to your network admin about firewall rules and if data loss is a concern you should review your backup policy. Further Reading: https://hackernoon.com/building-a-botnet-on-pypi-be1ad280b8d6 https://python-security.readthedocs.io/packages.html Dependencies are installed in this order: - `conda.txt` - `pypi.txt` - `vendor.txt` - `development.txt` - `integration.txt` Please review the documentation header at the beginning of each `requirements/*.txt` file to determine which file is appropriate for the dependency you want to add. Choose a file: - `conda.txt` is appropriate for non python packages and packages which would require compilation if they were downloaded from pypi or cannot be downloaded from pypi (such as openjdk or node). - `pypi.txt` is for dependencies on python packages, be they from pypi or git repositories. - `vendor.txt` is appropriate for pure python libaries which are written using mypy. This allows the mypy type checker to work with types defined in other packages After adding a new dependency, you can run `make update` ```shell (myproject_py36) dev@host:~/myproject $ make update Solving environment: done Downloading and Extracting Packages requests-2.19.1 | 94 KB conda-forge ... ``` Normally make update only does something if you update one of the `requirements/*.txt` files has changed. If you know a dependency was updated, and `make update` is not having an effect, you can force the update using `make force update`. ### Vendoring Vendored dependencies are usually committed to git, but if you trust the package maintainer and the installation via `vendor.txt`, then it's not required. There are a few reasons to vendor a dependency: 1. You want the source to be easilly accessible in your development tools. For example mypy can access the types of vendored projects. 2. You don't trust the maintainer of a dependency, and want to review any updates using git diff. 3. There is no maintainer or downloadable package, so your only option is to download it into a local directory. For example you may want to use some of the modules from https://github.com/TheAlgorithms/Python If you do vendor a dependency, avoid local modifications, instead contribute to the upstream project when possible. ## Development The typical commands used during development are: - `make install`: Setup virtual environment - `source activate`: Activate virtual environment - `make help`: Overview of tasks - `make fmt`: Format code - `make lint`: Linting - `make mypy`: Typecheck - `make devtest`: Run unittests with dev interpreter against code from `src/`. Slightly less common but good to run before doing `git push`. - `make test`: Run unitests on all supported interpreters after installing using `python setup.py install`. This tests the code as the users of your library will have installed. - `make citest`: Run `make test` but inside a docker container, which is as close to the ci environment as possible. This is quite useful if you don't want to trigger dozens of CI builds to debug a tricky issue. ### Docker The base image of the project is `docker_base.Dockerfile` which is used to create images that have only the conda virtual environment needed to run the project. The CI environment uses the image generated by `make docker_build`. While this means that the CI setup is simpler and faster, as you don't have to build the image for the test run in the CI environment, it does mean that you have to run `make docker_build` every time one of your dependencies is updated. The `docker_base.Dockerfile` uses the multi stage builder pattern, so that 1. your private key doesn't end up in the published image 2. the published image is as small as possible. ``` $ make docker_build Sending build context to Docker daemon 7.761MB Step 1/20 : FROM registry.gitlab.com/mbarkhau/bootstrapit/env_builder AS builder ... conda create --name myproject_py36 python=3.6 ... Solving environment: ...working... done ... conda create --name myproject_py35 python=3.5 ... Solving environment: ...working... done docker push ``` As is the case for your local development setup, every version of python that you have configured to be supported, is installed in the image. If you want to create a minimal image for a production system, you may wish to trim this down. ### Documentation Documentation is written in Github Flavoured Markdown. Typora is decent cross platform editor. TODO: `make doc` ### Editor Setup https://gitlab.com/mbarkhau/straitjacket#editortooling-integration TODO: Expand how to set editor, possibly by sharing editor config files? ## Best Practices While not all practices linked here are followed (they are contradictory to each other in places), reading them will give you a good overview of how different people think about structuring their code in order to minimize common pitfalls. Please read, view at your leasure: - Talks: - [Stop Writing Classes by Jack Diederich](https://www.youtube.com/watch?v=o9pEzgHorH0) - [The Naming of Ducks: Where Dynamic Types Meet Smart Conventions by Brandon Rhodes](https://www.youtube.com/watch?v=YklKUuDpX5c) - [Transforming Code into Beautiful, Idiomatic Python by Raymond Hettinger](https://www.youtube.com/watch?v=OSGv2VnC0go) - [Beyond PEP 8 -- Best practices for beautiful intelligible code by Raymond Hettinger](https://www.youtube.com/watch?v=wf-BqAjZb8M) - Articles, Essays, Books: - Short ebook for Novice to Intermediate Pythonistas: [How to Make Mistakes in Python](https://www.oreilly.com/programming/free/how-to-make-mistakes-in-python.csp) - [The Little Book of Python Anti-Patterns](https://docs.quantifiedcode.com/python-anti-patterns/) - Style Guides: - https://www.python.org/dev/peps/pep-0008/ - https://github.com/amontalenti/elements-of-python-style - https://github.com/google/styleguide/blob/gh-pages/pyguide.md Keep in mind, that all of this is about the form of your code, and catching common pitfalls or gotchas. None of this releives you of the burdon of thinking about your code. The reason to use linters and type checking is not to have a tool to make your code correct, but to support you to make your code correct. For now I won't go into the effort of writing yet another style guide. Instead, if your code passes `make fmt lint`, then it's acceptable. Every time you encounter a linting error, consider it as an opportinity to learn a best practice and look up the error code. [^1]: Linux, MacOS and [WSL](https://docs.microsoft.com/en-us/windows/wsl/install-win10)