Transitioning Idem to Python 3.8 and Beyond

As we near the end of life (EOL) for Python 3.7, it's time to start planning for the transition. This means dropping support for Python 3.7 in Idem, its plugins, and its dependencies. This blog post will outline the implications of making Python 3.8 our base version, the new advantages this brings, what we might miss from Python 3.7, and what adding or removing support for a Python version entails.

Advantages of the New Base

Transitioning to Python 3.8 as our base brings several advantages:

1. Assignment Expressions (The Walrus Operator): Python 3.8 introduces the walrus operator :=, which allows you to assign values to variables as part of an expression.

2. Positional-Only Parameters: In Python 3.8, you can specify positional-only parameters using the / syntax. This gives you more control over how functions can be called.

3. The Protocol Class and Static Typing Improvements: Python 3.8 provides the Protocol class, which is a base class for structural type checking. This, along with other improvements, enhances Python's static typing abilities.

4. Improved Debugging: Python 3.8 includes better error messages and stack traces, making debugging easier.

Of course, we'll also benefit from all the enhancements and fixes made in the subsequent versions up to Python 3.11.

What We'll Miss from Python 3.7

Python 3.7 was a stable and reliable version that served us well. Its introduction of data classes, breakpoint() built-in function, and the upgraded asyncio module were significant features. However, we believe that the benefits of moving to newer versions outweigh the comfort of sticking with Python 3.7.

The Transition Process

Transitioning Idem to support Python 3.8 to Python 3.11 involves several steps. Here's a handy bash script that recursively checks through a project directory for every instance of "3.7", so you can identify all the places that need to be updated:

#!/bin/bash

PATTERN="\b3[., ]*7\b"

matches=()

# Recursively search through the directory for the pattern
while read -r match; do
    # Extract the filename and line number from the grep result
    FILENAME=$(echo "$match" | cut -d: -f1)
    LINE_NUM=$(echo "$match" | cut -d: -f2)

    echo "Found pattern in $FILENAME at line $LINE_NUM"
    matches+=("$FILENAME:$LINE_NUM")
done < <(grep -rPn "${PATTERN}" .)

for match in "${matches[@]}"; do
    FILENAME=$(echo "$match" | cut -d: -f1)
    LINE_NUM=$(echo "$match" | cut -d: -f2)

    vim +"$LINE_NUM" "$FILENAME"
    # Wait for the user to exit vim before continuing
done

I also want to point out a few specific things that need to be updated as a part of adding or removing Python version support:

• In the setup.py, update the "classifiers" to include Python 3.11 and drop Python 3.7. The updated classifiers should look something like this:

classifiers=[
     ...
    "Programming Language :: Python",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    ...
],

• In the pre-commit-config.yaml, remove pip-tools-compile for Python 3.7 and add it for Python 3.11. Similarly, check for any other dependencies in the pre-commit like pyupgrade and black that might need to be updated to Python 3.8.

• Upgrade the .gitlab-ci.yaml, remove the block for testing Python 3.7, and add a block for testing Python 3.11. Here's an example:

tests-3.11:
  extends: .test-suite
  image: python:3.11

• A known issue is the incompatibility of the "asynctest" package with Python 3.11. If it's explicitly listed in your requirements/tests.in, remove it as a dependency and make sure that pytest-pop and pytest-idem, if they are in your test dependencies, are at the latest version.

• Run pre-commit run -a to refresh pre-commit. You will need to commit any changes it makes.

• Use nox -p 3.11 to run the tests locally with the same testing environment as the GitLab pipeline. You may encounter issues that need to be addressed based on these tests. It will require you to have python3.11 and nox installed on your dev computer.

• After making those changes, make a PR to the upstream repo with the changes and request approval from the maintainers.

With these steps, you are all set to support the latest Python and drop support for EOL versions in your Idem plugin or dependency! This is a significant milestone that aligns Idem with the ongoing evolution of Python. It ensures that we stay on top of the latest developments while delivering the best possible experience for our users.