Managing Python installations in 2023

The following is a short, opinionated guide on how to manage Python installations for scientific computing in 2023.

Motivation

Why write this post? I work in and around the scientific Python ecosystem, one of the projects I'm involved in is called napari. Some people without Python experience want to use napari, they often end up frustrated.

I wish I could use the wonderful @napari_imaging. but it's painful as a biologist to update all the associated items such as 'pip' . As a non-python user it's hard. I just don't have the time to look into all of the many dependencies. Needs a @fiji-like Update button

— Darren Thomson (@fungalhyphae) May 5, 2023

This makes me sad 🙁

Good news, there is a happy path! Once you're on it, I promise working with Python can be a great experience. I'm writing this to share the way I work and help you get on the right track.

Introduction

We are going to use virtual environments to manage our Python installation(s). Specifically, conda environments managed with micromamba.

Why do I need environments?

Python is ubiquitous, it's probably used in lots of different places on your computer already. Code written for specific versions of either the Python language or Python packages won't necessarily work with different versions. A virtual environment is a little box you can put Python and various packages into which is isolated from the rest of your system. What you do in the box won't affect what's going on outside the box.

Why micromamba?

One of the easiest ways to mess up a conda installation is to install a bunch of stuff into the base environment. micromamba doesn't give you a base environment so you can't mess it up! It's also wicked fast.

Installation

MacOS/LinuxWindows

1. Follow the instructions from the mamba documentation

2. Set conda as an alias of micromamba

   Note

   Add the following to your ~/.bashrc (Linux) or ~/.zshrc (macOS)

   alias conda="micromamba"
   

3. Reload your shell so that changes take effect

4. Set conda-forge as the default channel

   Note

   conda config --add channels conda-forge
   

1. Follow the instructions from the mamba documentation

2. Set conda as an alias of micromamba

   Note

   Create the alias and add it to your PowerShell profile

   New-Alias -Name conda -Value micromamba
   Export-Alias -Path "$HOME\Documents\PowerShell\Profile.ps1" -Append -Name conda
   

3. Reload your shell so that changes take effect

4. Set conda-forge as the default channel

   Note

   conda config --add channels conda-forge
   

Usage

Creating and activating environments

To create an environment with a specific version of Python

conda create --name my-new-env python=3.10

To work in the environment we first need to activate it. We do this with

conda activate my-new-env

Once inside the environment, you have access to the packages you have installed there.

Note

(my-new-env) ➜ python --version
Python 3.10.11

Installing packages

You can install most packages into your environment with conda or pip. Install what you can with conda. If a package is available on PyPI but not conda-forge then use pip.

condapip

(my-new-env) ➜ conda install numpy

(my-new-env) ➜ pip install numpy

Deactivating environments

We can exit an environment with

conda deactivate

Removing environments

We can remove an environment with

conda env remove --name my-new-env

Running commands from outside an environment

If you want to run a command in a specific environment from outside the environment you can

conda run --name my-new-env command

This is useful for workflows which rely on software with incompatible dependencies.

tips

• get comfortable with creating/destroying and activating/deactivating environments
• one environment per project is a useful ideal, in practice a general purpose default environment is useful for quick scripting/analysis

Closing

That's it! Working in conda environments empowers you to install things without worrying about messing up your whole system.