Created by: Ron Karroll and David Horstman
Last updated: March 29, 2022
When I first started at Clipboard Health a few months ago, I was eager to hit the ground running. I asked our Head of Engineering Mike Cook to recommend a good infrastructure task to get my feet wet. I expected heād suggest something on the scale of updating the linter. As my colleagues noted later, our company has high expectations even for its new hires, and when you ask to be āput to workā, youād better be ready to tackle a real challenge.
The problem posed to me was to develop a system for rapidly scaffolding and deploying new microservices to help our business scale. Our customer base and engineering team both are and were growing rapidly, and our legacy monolithic application was starting to show some cracks.
The fact that we couldnāt independently scale different parts of our application meant we had to choose between placing a high load on our most in-demand services like payment processing or spinning up more instances of our entire application, all just to keep up with the demand for one service. Since the amount of effort required to onboard a new engineer grows with the size of our codebase, this was also beginning to strain our finite training resources.
With the goal of alleviating those concerns in mind, I set out to design a tool to make it so easy to build new features in separate microservices that no engineer would be so much as tempted to add to the complexity of the existing monolith.
Objectives
We had already started migrating to a microservices architecture before I joined the company, but the process was slow. It would typically take several days for an engineer to create a new repository, initialize it with application code, set up a deployment pipeline, and finally make whatever new idea they were testing available to our product team.
Mike believed that the speed of this process was the main obstacle standing in the way of broad microservice adoption. We set reducing this ātime-to-launchā from ~3 days to less than 20 minutes as our first objective.
Our other main objective was to uphold high standards while giving everyone access to industry-standard libraries and frameworks. We decided to pamper our engineers by including all the best tooling in every new microservice we create:
- Typescript accelerates development by enabling editor suggestions and catching type errors at compile-time
- NestJS encourages modularity and provides unparalleled stability and consistency at runtime
- Prisma exposes a declarative interface for defining database schemas and a strongly-typed bridge between SQL and Typescript
- eslint/prettier/cspell ensure all code is linted, formatted, and spell-checked using a consistent ruleset across our organization
- Terraform allows us to define and provision new server resources without leaving the repository
- GitHub actions automates our entire CI/CD pipeline from commit to deployment, removing the need for any manual intervention beyond opening pull requests
Having determined our speed goals and ideal architecture, all that remained was to build the tool!
Introducing First Stage
We decided to name our new tool First Stage, since it allows our developers to easily get their ideas off the ground (the boring part) and focus on propelling their metaphorical rocket out into the stars.
For a variety of reasons (but primarily because CLI applications are objectively the coolest) we decided to design our tool around a command-line interface inspired by the likes of the AWS CLI and kubectl. We built in interactive prompts, colors, and spinners to reflect the serious professional polish weāve applied to all aspects of this tool.
Getting started with the tool is dead-simple. Any of our engineers can clone the repository, build the tool, and launch it with āfirststage boostā (get it?). From there, they answer a series of prompts about the new serviceās name, features, etc. and First Stage takes care of creating the repository, pushing it to remote, provisioning resources in AWS, and creating a deployment. The whole process from running āboostā to having a link to the live deployment only takes about 10 minutes.
Project Internals
Do you wish you could launch a similar tool within your organization? Hereās how we built it.
The backbone of First Stage is commander, a popular javascript framework for developing command-line applications. It provides an easy and intuitive way to define complicated commands with flags and arguments using method chaining. It also automatically generates āhelp documentation to obviate any need for code duplication.
We extended this functionality using Inquirer, a library specifically for prompt construction. It enabled advanced modes of interaction such as lists, input sanitization and transformation. Of course, it also helped make everything look gorgeous.
The last piece of the puzzle is GitHub itself. Our tool uses GitHub CLI to create repositories and GitHub Actions to provision infrastructure and create deployments. It was very fortunate that the DevOps team I joined for this project since I was already well-versed in these tools. Given that, re-configuring our existing terraform files and workflows to be generic enough for this project was surprisingly straightforward.
Second Stage?
Weāve just made our first release to the company as a whole, but First Stage is far from its final form. Weāre running the project as an internal open-source tool so any engineer at Clipboard Health can contribute to increasing the toolās functionality and ease-of-use.
Looking ahead, weāre planning on adding support for:
I hope youāll stay tuned for the progress of this project as it continues to develop!