When working with Idem, a crucial aspect of ensuring efficiency and stability is managing dependencies effectively. In this blog post, we will delve into the strategies for managing dependencies, employing extras, and ensuring version flexibility in Idem and its plugins like idem-aws, idem-azure, and idem-gcp.
Section 1: The Philosophy Behind Dependencies in Idem
Idem is fundamentally a library, and as such, it is vital for it to be as flexible as possible with the versions of packages in the requirements/base.txt file. Pinning to higher versions of packages should only be done when the library absolutely depends on functionality from it. This philosophy extends to Idem's plugins as well.
Each plugin, like idem-aws, idem-azure, and idem-gcp, has a different set of maintainers, but they should all test with the latest version of Idem. While users are encouraged to use up-to-date versions of Idem, Idem plugins, and their dependencies for the latest bug fixes, security fixes, and features, the packages themselves should list as flexible versions as possible to avoid issues when integrating different components of a large distributed Python app like Idem.
Section 2: Managing Base Requirements
The requirements/base.txt should contain the bare minimum requirements for the plugin or library to function. This ensures that only the necessary dependencies are installed by default, reducing bloat and potential conflicts.
Section 3: Leveraging Extras for Additional Functionality
For functionalities that are not core to the library or plugin but might be needed in certain scenarios, we employ "extras". Extras are optional dependencies that can be installed when needed. They reside in the requirements/extra folder. POP projects commonly can automatically detect files in "requirements/extra/" and use them to install additional dependencies.
For instance, a file named idem/requirements/extra/my_extras.txt can be installed using:
pip install idem[my_extras]
Moreover, you can use the "full" extras in almost any Idem plugin to install all the available packages:
pip install idem-aws[full]
Idem itself also provides extras named "frozen" for static requirements used for testing based on your environment's python version:
pip install idem[frozen]
As well as static extras for specific Python versions:
pip install idem[frozen-3.11]
Section 4: Gating Plugins Based on Library Availability
In POP/Idem projects, you can conditionally load plugins based on whether a certain library is installed. This is a common practice in POP projects, and you will frequently see plugins gated by a libraries ability to be imported, like this:
HAS_LIBS = True
except ImportError as e:
HAS_LIBS = False, str(e)
Section 5: Adding New Dependencies
When adding a new dependency to Idem or an Idem plugin, ensure that it has an MIT license or an Apache 2 license and that the project is actively maintained. You can verify if a project is actively maintained by checking its upstream git repository for recent commits and if the maintainers are actively addressing issues and accepting merge requests or pull requests. If you want to add a dependency that does not meet those criteria, consult the maintainers first.
Section 6: Preventing Major Version Bumps
In Idem and its plugins, a major version number signifies a behavioral change. A common practice is to add a restriction against installing the next, non-existent release of a dependency. This way, the integration of behavioral changes is deliberate and controlled.
Section 7: Handling Test Dependencies
When working with requirements in Idem, it's important to note that the files located under requirements/py<3.x>/tests.txt should not be edited manually. These files are compiled by the pre-commit pip-tools-compile hooks from requirements/tests.in. This automated process ensures that the dependencies are managed consistently and reduces the likelihood of errors. If you need to update the static requirements used for testing, then make the change to requirements/tests.in then re-run pre-commit.
Effectively managing dependencies in Idem and its plugins is key to building stable and efficient systems. By understanding and employing strategies like managing base requirements, leveraging extras, gating plugins, and being deliberate with versioning, you can ensure that your Idem-based projects are flexible, reliable, and maintainable. Happy coding!