Writing a pop-create-idem plugin

Introduction

In this post, we'll explore how to construct a pop-create-idem plugin, empowering you to rapidly craft your idem cloud project from the ground up.

Project structure

If you're starting a brand new project, begin by installing and running the basic pop-create to establish the foundation:

pip install pop-create
pop-create seed -n my_project --directory my_project_root -d pop_create

Next, modify the conf.py of your existing or new project so that it has these values in the SUBCOMMANDS dictionary and the DYNE dictionary:

# my_project_root/my_project/conf.py

# This adds your new project to pop-create's subcommands
SUBCOMMANDS = {"my_project": {
    "help": "Short description of my project", 
    "dyne": "pop_create"
}}

# This should already have been added by the previous pop-create command
# It tells POP that the folder named "my_project_root/my_project/pop_create" extends the "pop_create" dyne on the hub
DYNE = {"pop_create": ["pop_create"]}

With these configurations in place, you're all set to begin writing your plugin.

Designing Your Plugin

pop-create-idem includes code that transforms a CloudSpec dictionary into idem states, tools, and exec modules. Your unique pop_create plugin's purpose is to convert API documentation into the CloudSpec format. pop-create-idem also comes with inbuilt code that attempts to convert OpenAPI3 specs and Swagger specs into idem code, though there are times when API-specific nuances require manual resolution to ensure effectiveness.

Some plugins, such as idem-azure, scrape the documentation from Azure's website and convert it into the CloudSpec format. Likewise, idem-aws parses the boto3 Python library to create a CloudSpec. Before we dive into building that translation plugin, let's first take a look at the CloudSpec Grammar.

Understanding CloudSpec Grammar

CloudSpec Grammar serves as a syntax guide to define your idem project's structure and properties. Here is a brief outline of the CloudSpec Grammar:

CloudSpec = {
    project_name: str,
    service_name: str,
    api_version: str,
    request_format: dict[str, str],
    plugins: dict[str, CloudSpecPlugin],
}

CloudSpecPlugin = {
    doc: str,
    imports: list[str],
    virtual_imports: list[str],
    contracts: list[str],
    func_alias: dict[str, str],
    virtualname: str,
    sub_virtual: bool,
    sub_alias: list[str],
    functions: dict[str, CloudSpecFunction],
    file_vars: dict[str, any],
}

CloudSpecFunction = {
    doc: str,
    return_type: str,
    hardcoded: dict[str, str],
    params: dict[str, CloudSpecParam],
}

CloudSpecParam = {
    name: str,
    required: bool,
    target: str,
    target_type: str,
    member: CloudSpecMember,
    param_type: str,
    doc: str,
    default: any,
    snaked: str,
}

CloudSpecMember = {
    name: str,
    params: dict[str, CloudSpecParam]
}

Descriptions

• CloudSpec
  

  • project_name: Conventionally called "idem_<my_project>".

  • service_name: Typically just "<my_project>".

  • api_version: Used for differentiating between versions of the API you're consuming. Optional, default: 'latest'.

  • request_format: A mapping of function names to a cookiecutter template string that describes how the function should be called. CookieCutter has access to the "ctx" from your plugin. Optional, default: HTTP_REQUEST_FORMAT.

  • plugins: A mapping of a plugin ref's to their CloudSpecPlugin definition. The ref is a dot "." separated string representing the path to a plugin. Optional.

• CloudSpecPlugin
  

  • doc: Docstring for the function.

  • imports: List of Python imports required for this plugin. Optional.

  • virtual_imports: List of virtual imports. The plugin will not load onto the hub if these imports fail. Optional.

  • contracts: List of contracts enforced by this plugin. Optional.

  • func_alias: Dictionary mapping function names that clash with Python keywords to their intended names on the hub (i.e., {"lambda_": "lambda"}). Optional.

  • virtualname: Virtual name for this plugin on the hub if its name clashes with a Python keyword or a standard Python library. Optional.

  • sub_virtual: For "init.py" files specifically. A condition that would prevent all files in the same directory from being loaded onto the hub. Optional.

  • sub_alias: Also for "init.py" files. An alias for its parent directory, to be used if the parent directory clashes with a Python keyword or standard library. Optional.

  • functions: Mapping of function names to a CloudSpecFunction. Optional.

  • file_vars: Any other arbitrary key/values to pass to the plugin spec. Optional.

• CloudSpecFunction
  

  • doc: Description of the function.

  • return_type: Type hint for the return value of the function. Optional, default: 'None'.

  • hardcoded: Mapping of keys to strict values that should be used as-is. Optional.

  • params: Mapping of parameter names to the CloudSpecParam object. This will implicitly extend the function docstring for you. Optional.

• CloudSpecParam
  

  • name: Name of the parameter.

  • required: True if the parameter is required, else False.

  • target: Indicates how this parameter is consumed in the function (i.e., is it part of "kwargs", or does it have an alias?).

  • target_type: Indicates how the parameter should be consumed by create plugins (i.e., "mapping", "arg").

  • member: For dictionary type parameters that need to create a dataclass for their definition to define what should be in that dictionary. Optional.

  • param_type: Type hint for the parameter. Optional.

  • doc: Docstring for the parameter. Optional.

  • default: Default value for the parameter. Technically this is optional, but you know better.

  • snaked: The snake_cased version of the parameter. Some APIs have camel case variables, and this helps convert everything into a consistent, readable format. Optional.

• CloudSpecMember
  

  • name: Name of a dataclass from a param.

  • params: Params object to be used within the new data class.

Creating Your Plugin

Though a pop-create-idem plugin requires only one file at its core, it is advisable to break that file into logical components with descriptive names. Fill the directory my_project_root/my_project/pop_create/ with helper files and functions that facilitate better understanding and maintenance.

Here's a pseudocode outline of how a pop_create plugin could look:

import pathlib

def context(hub, ctx, directory: pathlib.Path):
    # Parse the API and convert it to Cloud Spec
    cloud_spec = {
        "plugins" = {"functions": = {"params": {}}}
    }

    # Attach the cloud_spec to the ctx for pop-create-idem to use
    ctx.cloud_spec = cloud_spec

    # Call the "idem_cloud" subparser with your extended ctx
    hub.pop_create.init.run(directory=directory, subparsers=["idem_cloud"], **ctx)

    # Return ctx to pop-create for cleanup and copying files
    return ctx
    

In this code snippet, the context function is crucial and required by pop-create contracts. It must have the exact arguments, hub, ctx, and directory, and it is obligated to return ctx at the end of its execution. hub is the central object for accessing all pop-loaded systems. ctx is the context object that carries the configuration values throughout the execution process. directory is a path to a temporary directory that allows write permissions, ensuring that any files or data can be safely manipulated during execution.

In the function, the first operation is to parse the API and convert it into a format referred to as CloudSpec. Each key-value pair in the cloud_spec dictionary defines specifications for the cloud-based service, represented by individual plugins. Each plugin usually represents a single resource on the API, such as a compute instance or a storage bucket.

Once the cloud_spec is constructed, it is attached to the ctx object, which is a mandatory step for pop-create-idem to function correctly. Adding cloud_spec to ctx essentially lets pop-create-idem know the cloud resources' structure it needs to work with.

Next, the function calls hub.pop_create.init.run, passing the temporary directory path, the 'idem_cloud' subparser, and the enhanced ctx object. This 'idem_cloud' subparser works with the cloud_spec definition embedded in ctx to generate 'idem' plugins. These plugins play a crucial role in managing cloud resources in a declarative way using Python.

For each plugin or resource defined in the cloud_spec, it is important to note that "create", "delete", "update", "get", and "list" operations need to be defined for it to function properly with idem. These operations facilitate full control over the lifecycle of each resource. You can specify a specific create_plugin on the cli, but the default is "state_modules", which will create CRUD exec modules that can be automatically combined by idem to be used as states without needing to write an explicit present, absent, or describe functions. You can specify any of these other "create" plugins from the CLI with "--create-plugin=autostate" (other options at the time of writing are auto_state, docs, exec_modules, state_modules, templates, tests, tool). In proper POP fashion you could also write your own "create" plugin to extend this functionality in your own project. Finally, run "pip install ." in your new project's root to add it to the Python path and then "pop-create my_project -n my_project" to test your new plugin!

Conclusion

As I reflect on my journey through the realm of software development, I can't help but marvel at the evolution of our tools and approaches, and how they have enabled us to design and implement increasingly complex and powerful systems. My initial years were filled with shell scripting, where every line of code was a direct action with instant results. Then came the era of Salt and Idem, marking a significant step forward in declarative programming.

The creation of pop-create and pop-create-idem represents yet another leap forward in this journey. These tools, the result of the diligent work of a team of exceptional developers I'm fortunate to collaborate with, have greatly streamlined the plugin creation process for Idem. The level of abstraction and automation they provide is incredible and they continue to receive regular improvements and refinements from the community.

Looking to the future, I see further exciting developments on the horizon. With the rapid advancement in AI, it's not hard to imagine these tools becoming even more sophisticated and intelligent. They could potentially analyze API documentation automatically, understand the underlying patterns, and generate highly optimized plugins accordingly. The resulting increase in productivity and efficiency could be monumental.

This vision excites me. It's the essence of what drew me to software development in the first place—the challenge of creating increasingly efficient and powerful tools, tools that empower us to do more with less effort. As I continue my journey, I'm eager to see what further innovations await us in the world of declarative programming and beyond. Enjoy coding!