Python Environment 2020
ref -- see
https://packaging.python.org/overview/
Make development easy and consistent
It's 2020 and I need to refresh my Python development environment to work on some new and old projects. Anything new will be in Python 3 of course, but there's still some Python 2 to transition. I develop mostly on macOS primarily, though sometimes on Linux, Windows, and FreeBSD too, so I need a consistent setup on all of those systems. (Windows is a special case of Linux, as I use WSL there).
My projects are typically built using a CI system, and then artifacts of that build are deployed on Linux systems -- either directly or via a Docker container (which itself may be in a system like Kubernetes). I don't control the configuration of the production environment directly, nor do I want to have to worry too much about that.
What goes where -- tools -- even if they are written in Python are installed separately from the library or application under development. A C-based project may depend on gcc or llvm for instance, but that is typically not installed in the project directly. Consequently, if a project depends on (e.g. mypy) for it's build, that will need to be a dependency.
From development to production
Dev system designed for development ergonomics
For Python, that means being close to production environment. Use the same version of Python and the same versions of libraries as much of possible -- certainly for fully native Python modules. Beneath those modules are shared native libraries (like database and SSL libraries). To the largest extent possible those should be as close to production as possible -- but the goal is to maintain the same Python libraries. Our goal is to create a productive developer experience, not to mimic the production environment.
CI is for transitioning to production
If the development system is designed for developer ergonomics, the CI system should be used to transition the project to production system testing. Getting exactly like production is usually not possible (if for no other reason that the CI system should not be running in production!). We might consider two CI passes -- one designed for quick build and test verification, and another perhaps slower pass designed to build, test, and package the system for production. Dev is for development, prod is for production, and CI is the glue in between.
Now that we know our goal is to setup a Python development environment that focuses on the developer, where do we start?
System installed Python? Brew?
Do not start with the system-installed Python runtime, even if it's the same version you want to target. The system Python is there to support the system -- it may or may not be configured as you want it. It certainly won't have the 3rd party libraries you need. And it is unlikely to be updated on a schedule that matches your own need. Do not use the system-installed Python runtime. Pick a python version, and deploy it with PyEnv.
Even Brew may install Python as a dependency for some other brew installed tool
-- for example gdb drags in Python 3.8. Ignore that too. Again - you may need
control over the specific Python version used in production and hence in your
development environment. Let it exist, but always use pyenv to manage the Python
runtimes in use for development.
Other tools that happen to be written in Python
lots of tools that use python -- pycowsay , calibre, and dev tools too. Some come
with their own runtime (calibre) some depend on others. Either use brew or system
package manager for those tools and let the developer and packagers worry about
compatibity issue, or use pipx.
Don't
install them in the system env,
flake8, black, mypy -- need python. Do not install into OS python -- again that's
not for you. Use pipx to install each into an encapsulated runtime (just like
your production). Treat them the specialized tools they are -- like git -- they
don't belong in your code and you may need control over what version(s) you want
to use. Pipx solves this problem nicely.
Alternatively, some of these tools (like flake8) may be packaged by the system or Homebrew, and in some cases using that tool could work. However I like to always use pipx for Python development tools as it enforces consistency.
From Zero to Python Project
steps to go from a brand new machine to a well structured Python 3 project.
FreeBSD
start with basic freebsd system
$ sudo pkg install curl bash git
$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv
$ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
$ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
$ echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
$ pyenv
$ pyenv install -l
$ pyenv install 3.8.2
$ pyenv install 2.7.17
$ pyenv global 3.8.2
$ pyenv versions
$ pyenv global 3.8.2 2.7.17
$ pyenv global
$ which python
/home/dcreemer/.pyenv/shims/python
$ python -V
Python 3.8.2
$ pip -V
pip 19.2.3 from /home/dcreemer/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pip (python 3.8)
$ which python2
/home/dcreemer/.pyenv/shims/python2
$ python2 -V
Python 2.7.17
$ pip2 -V
pip 19.2.3 from
/home/dcreemer/.pyenv/versions/2.7.17/lib/python2.7/site-packages/pip (python 2.7)
Update pip in the "global" envs -- this is the only step I ever take in my installed Python envs.
$ pip2 install -U pip
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting pip
Downloading https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl (1.4MB)
|################################| 1.4MB 2.8MB/s
Installing collected packages: pip
Found existing installation: pip 19.2.3
Uninstalling pip-19.2.3:
Successfully uninstalled pip-19.2.3
Successfully installed pip-20.0.2
$ pip install -U pip
Collecting pip
Using cached https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 19.2.3
Uninstalling pip-19.2.3:
Successfully uninstalled pip-19.2.3
Successfully installed pip-20.0.2
$ sudo pkg install -y py37-pipx
$ echo 'export PATH="$PATH:$HOME/.local/bin"' >> .~/.bash_profile
$ export PATH="$PATH:$HOME/.local/bin"
$ pipx list
nothing has been installed with pipx
$ pipx install pycowsay
installed package pycowsay 0.0.0.1, Python 3.7.7
These apps are now globally available
- pycowsay
⚠️ Note: '/usr/home/dcreemer/.local/bin' is not on your PATH environment variable. These apps will not be globally accessible until your PATH is updated. Run `pipx ensurepath` to automatically add it, or manually modify your PATH in your shell's config file (i.e. ~/.bashrc).
done! ✨ 🌟 ✨
$ pipx list
venvs are in /usr/home/dcreemer/.local/pipx/venvs
apps are exposed on your $PATH at /usr/home/dcreemer/.local/bin
package pycowsay 0.0.0.1, Python 3.7.7
- pycowsay
$ pycowsay
< >
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
$
this works, but is not using the envs we just installed, but the system env. This is OK, but I want a bit more control. Let's back this out and use the pyenv-installed python:
$ sudo pkg remove -y py37-pipx && sudo pkg autoremove -y
$ python3 -m pip install --user -U pipx
$ pipx install pycowsay
installed package pycowsay 0.0.0.1, Python 3.8.2
These apps are now globally available
- pycowsay
done!
Now pycosway is installed, using it's own virtualenv based on pyenv installed
Python 3.8.2, and is on our path. Pip, and pipx are the only tools I ever
install in the "base" Python runtimes installed via pyenv.
Ubuntu
$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv
$ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
$ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
$ echo 'export PATH="$PATH:$HOME/.local/bin"' >> .~/.bashrc
$ echo 'eval "$(pyenv init -)"' >> ~/.bashrc
logout & back in
$ sudo apt install -y build-essential libssl-dev zlib1g-dev libncurses-dev libreadline-dev libgdbm-dev libdb-dev libbz2-dev liblzma-dev libsqlite3-dev libffi-dev
...
$ pyenv install 3.8.2
Downloading Python-3.8.2.tar.xz...
-> https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tar.xz
Installing Python-3.8.2...
Installed Python-3.8.2 to /home/dcreemer/.pyenv/versions/3.8.2
$ pyenv versions
3.8.2
$ pyenv global 3.8.2
$ pyenv versions
* 3.8.2 (set by /home/dcreemer/.pyenv/version)
$ python -V
Python 3.8.2
$ pip install -q -U pip
$ python3 -m pip install --user -U pipx
$ pipx install pycowsay
installed package pycowsay 0.0.0.1, Python 3.8.2
These apps are now globally available
- pycowsay
done!
Mac OS
(assumes hombew, bash, and that the xcode sdk is installed).
$ brew install pyenv pkg-config gdbm openssl@1.1 readline sqlite xz
$ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
$ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
$ echo 'export PATH="$PATH:$HOME/.local/bin"' >> .~/.bashrc
Create a new terminal window to pickup the environment settings, then
$ pyenv install 3.8.2
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.1.tar.xz...
-> https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tar.xz
Installing Python-3.8.2...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.2 to /Users/dcreemer/.pyenv/versions/3.8.2
$ pyenv global 3.8.2
$ pyenv versions
system
* 3.8.2 (set by /Users/dcreemer/.pyenv/version)
$ python -V
Python 3.8.2
$ pip install -q -U pip
$ python3 -m pip install --user -U pipx
$ pipx install pycowsay
installed package pycowsay 0.0.0.1, Python 3.8.2
These apps are now globally available
- pycowsay
done!
Python Summary
We now have a consistent Python installation on three different operating
environments. python and python3 mean version 3.8.2, with a reasonably
consistent build (not exact, but it never can be on different operating systems).
More importantly, we have a plan for keeping our toolset consistent, clean, and
updated. Below is the process I used to migrate my system from Python 3.8.1 to
Python 3.8.2:
$ pyenv install 3.8.2
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.2.tar.xz...
-> https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tar.xz
Installing Python-3.8.2...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.2 to /Users/dcreemer/.pyenv/versions/3.8.2
$ pyenv versions
system
2.7.17
* 3.8.1 (set by /Users/dcreemer/.pyenv/version)
3.8.2
$ pipx list
venvs are in /Users/dcreemer/.local/pipx/venvs
apps are exposed on your $PATH at /Users/dcreemer/.local/bin
package pycowsay 0.0.0.1, Python 3.8.1
- pycowsay
$ pyenv global 3.8.2
$ pip uninstall pipx
Found existing installation: pipx 0.15.1.3
Uninstalling pipx-0.15.1.3:
Would remove:
/Users/dcreemer/.local/bin/pipx
/Users/dcreemer/.local/lib/python3.8/site-packages/pipx-0.15.1.3.dist-info/*
/Users/dcreemer/.local/lib/python3.8/site-packages/pipx/*
Proceed (y/n)? y
Successfully uninstalled pipx-0.15.1.3
$ pip install --user -U pipx
Collecting pipx
Using cached pipx-0.15.1.3-py3-none-any.whl (35 kB)
Requirement already satisfied, skipping upgrade: argcomplete<2.0,>=1.9.4 in ./.local/lib/python3.8/site-packages (from pipx) (1.11.1)
Requirement already satisfied, skipping upgrade: userpath in ./.local/lib/python3.8/site-packages (from pipx) (1.3.0)
Requirement already satisfied, skipping upgrade: click in ./.local/lib/python3.8/site-packages (from userpath->pipx) (7.1.1)
Installing collected packages: pipx
Successfully installed pipx-0.15.1.3
$ pipx reinstall-all --python "$(which python)"
uninstalled pycowsay! ✨ 🌟 ✨
installed package pycowsay 0.0.0.1, Python 3.8.2
These apps are now globally available
- pycowsay
done!
The only annoying bit is removing and instlling pipx in the new Python 3.8.2 environment. After that, a reinstall-all of pipx-installed packages will migrate everything to the new runtime.
For more on wsup, see wsup's GitHub repository.