Mastering Idem's Describe Subcommand with Filtering

If you're new to Idem, the describe subcommand is your best friend. It serves as an entry point for DevOps engineers and SREs, helping you swiftly write out states even if you've never touched Idem before. Its uses don't stop there, it also can be used for enumeration or for drift detection. Whether you're just dipping your toes in Idem or looking to monitor resources, describe serves multiple purposes that adapt to your needs. However, it can be very resource-intensive. In this post, we will discuss how "describe" works in detail, how to use idem describe's innate filtering, and how to optimize that filtering by looking under the hood and making use of describe's underlying "list" exec module calls.

Filtering in Action with idem describe

When you run idem describe, the command usually invokes the resource's list exec module to enumerate resources. After pulling the data, describe formats this output into valid "present" SLS states. The --filter option then comes into play, filtering data post-fetch. This design is crucial because it allows Idem describe's filtering to be compatible with multiple cloud providers, which all may have unique server-side filtering capabilities that we can't predict.

Example:

idem describe aws.s3.bucket --filter="[?resource[?bucket_prefix=='production']]"

Note that this method can be resource-intensive as Idem fetches *all* data before filtering. The --filter option works on pre-processed data, represented as a list of dictionaries. To get a clearer view of the data you are working with to craft a JMESpath, you can run:

idem describe aws.s3.bucket --output=jmespath

Server-Side Filtering with idem exec

Many "list" exec modules expose the underlying API's arguments, allowing for server-side filtering. For instance, with idem-aws, you can call:

idem exec aws.s3.bucket.list filter="buckets[?starts_with(name, 'production')]"

However, this means you'll have to manually transform the output into "present" states. To convert the output into an SLS-friendly format, you can use jq:

idem exec aws.s3.bucket | jq -n 'reduce (inputs | .[] | { (.resource_id): [to_entries | map({(.key): .value})] }) as $item ({}; . * $item)'

Here, "resource_id" becomes the top-level key, and other attributes are arranged as key-value pair dictionaries -- which is the SLS present state format.

Conclusion

Idem's describe subcommand isn't just an enumeration tool; it's a versatile instrument for resource management. While "list" exec modules may sometimes share the same keys as the corresponding present states, this isn't a guarantee. Describe takes the uncertainty out of the equation and always produces a valid SLS present state for the resource, ironing out any quirks. On the other hand, "list" can offer optimization through server-side filtering but requires manual labor to transform into a present state. Your choice between the client-side convenience of "describe" and the server-side efficiency of "list" hinges on your specific needs and how much manual work you're willing to invest.