The Early Days: Birth of an Idea
My journey started back in 2015 when I began working as a junior Linux Systems Administrator while in college. As a Linux newbie, I took my first steps into the world of system administration, which soon led me to encounter SaltStack for the first time. In these early days, my fascination with Linux grew, leading me to organize a Linux club at my college. I also became connected with the broader open-source community through the Utah Open Source organization, an association I'm still a part of today.
I first applied to SaltStack during these formative college years. Although I was not ready to join the team at the time, the experience sparked a drive in me to learn Python and gain professional experience, which ultimately led me to apply again and join the SaltStack team.
During this period, I was also undergoing significant changes in my personal life. I relied heavily on the SaltStack community, DC801 hacker meetups, Salt Lake Python user group meetings, Utah Open Source, and coffee meetups to build my personal communities.
Joining SaltStack
When I finally joined SaltStack, my primary role involved testing Salt Cloud. I gained a lot of experience working with the various clouds and in maintaining a large open source project. Our team had been pushing for Salt to adopt a rolling release structure and improve the testing process.
During this time, Thomas Hatch, the creator of SaltStack, introduced idem at SaltConf 2019. Idem was a reimagined version of Salt's state engine that utilized Plugin Oriented Programming (POP), a concept that struck me as potentially transformative.
The Birth of Idem
The birth of idem was heralded by its first commit on September 23, 2019. At SaltConf 2019, Thomas Hatch introduced Nick Hughes, a prominent member of the salt-cloud community, the salt-cloud working group captain, and CEO of EITR Technology, to idem. Together, they envisioned a future where idem's agentless runtime could be used to manage cloud infrastructure. This meeting led Nick to create idem-azurerm over a single weekend, the first instance of idem being used to its full potential in a cloud environment.
Despite this promising start, the road ahead was not paved for idem. It was an agentless and niche-less tool that was yet to find its place in the broader tech landscape. The question of idem's future application was as open as the cloud-strewn sky. However, a journey halfway across the world to Belgium would soon provide a hint of what was to come.
Just months after the inception of idem, I had the opportunity to present at a cloud conference in Belgium. The event brought together key stakeholders from the cloud and open-source communities across Europe, including the team from SUSE, a German-based multinational open-source software company.
Presenting at this conference offered me the chance to showcase idem to a broader international audience, explaining its capabilities and potential applications. This marked the first significant international exposure for idem and the beginning of its journey into the broader open-source community.
These experiences served to solidify my belief in idem's potential. Yet, idem was still a tool in search of a home, waiting to fully reveal its potential to the world. The road was uncertain, but the wheels were in motion, setting the stage for idem's evolution and growth.
Advocating for Idem
Inspired by idem's potential, I started working on moving salt-grains into idem in my downtime. I created numerous OS-specific plugins for idem, including a cmd.run plugin for any OS, along with plugins for grains collection. This was part of our attempts to find idem's niche, which involved exploring different potential uses for idem, such as using it as a standalone universal package manager or as a distributed "grains" collection tool. However, these attempts didn't gain traction, and idem's potential remained largely untapped.
In this time I also did a twitch stream and created many YouTube Videos to both advocate for idem and to show how to build a distributed open source app. Later on, these videos and stream recordings would provide valuable training on idem's philosophy and structure.
Despite these setbacks, Thomas Hatch and I strongly believed in idem's potential. Thomas was a powerful advocate for idem, introducing it to new users and pushing for its adoption. As for me, my history in the open-source community and strong belief in the Unix philosophy of "doing one thing well" pushed me to ensure that idem remained a distributed application and didn't morph into a monolith.
Working Solo
My passion for idem remained unwavering, even as the project's progression turned into a solitary endeavor due to a lack of resources. Given that idem was not viewed as a principal revenue driver, it was relegated to a lower priority status. This positioning led to a uniquely challenging situation. Despite the constraints, I was unyieldingly committed to fulfilling all my responsibilities on the SaltStack team and dedicating any additional time I had to my passion project, idem.
My tenacity saw me work tirelessly, with many late, sleepless nights outside my regular work hours dedicated to advancing idem. During these demanding periods, I found invaluable support in the mentorship of Thomas Hatch. His vast knowledge and experience provided crucial guidance as I sought to navigate the complexities and expand idem's functionalities.
The taxing nature of single-handedly steering idem was not lost on me. In recognition of this challenge, my colleague Derek Ardolf introduced me to Cookiecutter, a Python templating engine. Inspired by its potential, we collaborated to create pop-create-idem, a ground-breaking tool that harnessed the power of Cookiecutter and OpenAPI3 specifications to automate the generation of new idem plugins.
VMware's Acquisition and Idem's Rapid Growth
The acquisition of SaltStack by VMware marked a turning point in idem's journey. As the ebb and flow of the corporate transition permeated SaltStack, idem found itself adrift in these tumultuous waters. VMware, with its cloud-centric focus, saw the nascent potential in idem as an opportunity to further bolster its cloud strategy.
In the wake of this acquisition, the discourse surrounding idem shifted. The once low-priority side project started garnering attention and seemed to be on the brink of becoming a major player in VMware's software arsenal. A key turning point was when Thomas Hatch got a chance to present idem and its potential to VMware executives. He vouched for its value as a flexible, powerful tool capable of bridging gaps in cloud management with its unique agentless architecture and modularity.
Despite these promising developments, the road to idem's maturation was strewn with hurdles. During the transitional phase, idem found itself in a precarious position. As responsibilities and priorities were being reshuffled, I was constantly moved between teams within the organization. Teams working on Salt and SaltStack Config, burdened by an intense workload, were frustrated by the resources I dedicated to idem. Striking a balance between addressing present demands and advocating for the future potential of idem was a delicate task. Nonetheless, I remained steadfast in my belief in idem, championing its potential whenever the opportunity arose.
The establishment of a dedicated idem team was a journey filled with uncertainty and speculation. There were numerous discussions about how this team should be formed. Proposals ranged from shaping it out of an existing team to setting it up as a branch within the Research and Development department, or possibly as a distinct unit under the Office of the CTO (OCTO).
As these discussions ripened and began to form a consensus, the decision was made to go ahead with the creation of the team. I expected that I might have a chance to lead this team, given my intimate involvement and history with idem. However, that wasn't in the stars for the moment.
VMware made a significant decision to assemble a robust team dedicated to idem's development. This team was an eclectic mix of a seasoned project manager, a dedicated SRE, a community manager, several experienced developers, and many other talented contributors, each contributing their diverse expertise and unique perspectives. Despite being new to idem, the team's capacity to rapidly understand and adopt the project's core principles and potential was remarkable.
This strategic shift marked the dawn of a period of swift growth and maturation for idem. Transitioning from a solo endeavor to a team-driven initiative, idem was now recognized as a key asset within the company. This collective effort sparked a fresh dynamism in the project, with the idem team emerging as a powerhouse, spearheading the integration of idem into other VMware products. Consequently, idem experienced a rapid evolution, establishing itself as a robust and adaptable tool for cloud management.
The Future of Idem
As it stands today, idem has matured significantly and is enjoying growing use within both VMware and the broader open-source community. However, the journey doesn't stop here. As we look ahead, there are exciting visions of idem's evolution and potential to reshape how we interact with cloud infrastructure.
In my vision, I see idem metamorphosing into a user-friendly tool capable of swiftly deploying intricate multi-cloud infrastructure. The concept of "idem formulas" could become the new standard in the field. These "formulas" would allow users to set up everything they need for a cloud application — from load balancers and VPCs to gateways and databases — with a single command line input.
Think of idem formulas as blueprints for various types of infrastructure setups, defined in a 'params' file akin to how Salt's pillars are used in SaltStack Formulas. These formulas could then be used to rapidly deploy complex infrastructures, almost like a ready-to-assemble kit for cloud setups.
This isn't just about convenience, though. By facilitating cloud development in this way, we're looking to make these powerful capabilities accessible to a wider range of users. This could open the door to a new generation of developers and businesses, enabling them to harness the full power of the cloud without needing a deep understanding of the underlying complexities.
This vision is in line with the ever-evolving nature of cloud technology. With an increasing shift towards Software-as-a-Service (SaaS) solutions, idem can play a pivotal role in providing quick and efficient deployment solutions. We're in an era where agility and speed are paramount, and idem, with its potential for rapid deployment of complex cloud environments, is well-positioned to cater to these demands.
As we continue to develop and refine idem, our focus will remain on empowering users to navigate the cloud more effectively, ultimately working towards a future where the power of the cloud is truly harnessed for everyone's benefit.