Two years ago, Yodle began to move from a monolithic weekly deployment out of a monolithic repository to continuous delivery of microservices out of individual repositories. This process has been great, but it has also led to a number of challenges, many of which you can hear about in our CTO’s talk at QCon. We’ve recently released Vantage, a tool we’ve developed to solve some of the problems that have arisen around dependency management, as an open source project (along with an accompanying Gradle plugin).
Vantage is a web application designed to give an organization perspective into the state of their dependencies across their entire estate. It is intended to solve the problems that arise when an organization scales from a manageable number of centralized projects to hundreds or thousands of libraries and applications spread across independent repositories. To illustrate those problems, let’s walk along the path Yodle took from a monolithic repository and build to microservices in individual repositories and see how that has affected dependency management along the way.
The Monorepo Era
For all the faults of a monolithic build out of a single repository, it does tend to make dependency management simpler. As someone who creates or modifies libraries, if I make a change to a subproject, I know that change will roll out to every other project that depends on it. I don’t need to worry about convincing application developers to upgrade their dependencies because there’s a security flaw in our authorization library or a memory leak in our service discovery library. Third party library versions can be centrally managed so that if we want to ensure that everyone is on a new enough version of Spring, it’s a simple as updating a file. We can easily rebuild everything and run our monolithic regression suite to ensure that the change we’re trying to make doesn’t break everything.
That all sounds great. No one needs to worry about library versions. Application developers don’t need to make sure they’re on the latest versions of libraries. Library developers don’t need to worry about haranguing application developers to upgrade. Everyone wins, right? Well, it turns out that like many of the other positive aspects of monorepos and monobuilds, such a system doesn’t scale. Sometimes you need to introduce a backwards incompatible change (unfortunate, but it does happen) and have to face the prospect of updating a number of applications to make your change. Sometimes some edge case in a single webapp prevents you from upgrading to Spring 4. Sometimes you inherit code from an acquisition and it would take months of work to synchronize their dependencies with yours. Often, the person driving the dependency change knows little to nothing about the specifics of the applications being affected by the upgrade. The cost of doing a monolithic build and regression just to test a simple dependency upgrade becomes unbearable. The cost of just having hundreds of projects in your monolithic build starts becoming a problem even when you’re not upgrading dependencies.
Starting the Split
In order to address these scaling problems, we began splitting out libraries into their own repositories and publishing versioned artifacts. Hundreds of subprojects were pulled out of our gradle build. (Many of these were Thrift interface projects, which required us to write a gradle plugin called Griddle to handle Thrift IDL generation with published artifacts). We stopped requiring third party library versions to stay in sync across the entire build. This addressed many of the scalability concerns with having a single monorepo, but it introduced new problems.
Application developers were now primarily responsible for ensuring their dependencies were staying up to date. However they are not the ones with insight into what library versions exist or what versions they should be using. Library developers might send out emails asking people to upgrade because of a bugfix in a shared library, but emails get missed. We had a wiki page describing what versions of libraries people should use, but that page got out of date and many developers didn’t consult it. Overall, there just wasn’t a good channel for keeping developers knowledgeable about their dependencies. Dependency versions began to stagnate and fragment. The one saving grace was that, because all our applications were in one repository, a developer could easily grep one place for in-use library versions. On occasion, for important changes, mass dependency changes had to be made via clever use of xargs and sed.
This state was OK but not great from a dependency management perspective, but we still had a scaling problem irrespective of dependency management. The monolithic build was still slow. The monolithic regression and deploys were still expensive and unreliable. This drove us towards splitting up applications and services into their own repositories with their own build, certification, and deployment processes. This was vital for our ability to scale, but introduced yet more problems in dependency management. Whereas before, we at least had a central place to investigate and update dependency usage, now there were hundreds of repositories. It became difficult to even know what repositories existed or which ones might need to be updated. We used Hound to search the code across many repositories, but that is an incomplete solution. Even if we knew what repositories needed to be updated, it’s not scalable for a library developer to go into dozens of repositories they’ve never touched, update applications they don’t own, and run certification and deploy processes they’ve never used.
So to summarize the problem, we’re in a situation where application developers don’t know what libraries exist, what versions to use, or if the versions they depend on should be upgraded for some reason. Library developers don’t know who is using their libraries or what versions are in use, and don’t have a direct channel to provide information about what dependencies should be upgraded to application developers.
Enter Vantage, Stage Left
The seed for Vantage was planted in our heads at a talk by CoreOS about Clair. The goal of Clair is to analyze Docker containers and their base images for vulnerabilities so that a developer who produces a Docker container that is affected by a security issue would be notified. What we realized is that we wanted that kind of analysis, but for our library dependencies. We came up with a list of three questions we wanted to be able to answer:
- Given an application, does it depend on any library versions with known issues?
- Given a library, what versions of that library are still in use?
- Given a version of a library, what other libraries or applications depend on that version?
At Yodle, we hold internal hackathons a few times a year, and so with those goals in hand, we formed a team and began working on Vantage. We’ve iterated on it off and on and it has recently hit a state that we’re happy sharing with the rest of the world.
How It Works
Vantage’s domain model is primarily based on three entities, Components, Versions and Issues. A Component is essentially ‘a thing that has versions’, i.e. a library or an application. A Version, as you’d expect, represents a particular release of a Component. This may be a particular library version like ’1.2.0′ or a git commit hash representing the commit from which an application was built. Versions can have dependencies on other versions of other components. Issues represent some kind of indication that a particular version or set of versions should not be used for some reason. This may be a bug, a security flaw, an organizational policy, or a gentle encouragement to upgrade to more recent versions of a particular library.
In order to understand how these entities interact, let’s walk through some scenarios. Let’s say you’re an application owner, and you want to look at whether or not your application has any dependencies with known issues. Vantage can display a list of all your versions and flag any that depend on a dependency with a known issue.
You can drill down into the most recent version and get a list of all the dependencies of that application. Dependencies with known issues are floated to the top with a warning icon.
From the opposite perspective, a library developer may be interested in what versions of their library are currently in use. Active versions are called out by a green star, so that you can tell which versions are being used. You can additionally see that some of those active versions have known issues.
The fact that a library with known issues is in use warrants further investigation. Drilling down into that version lets you see details about the issue. It also displays which other components depend on this library version so that you know what needs to be upgraded.
Finally, issues can be created or modified through the UI (or through the REST api).
This functionality solves the three problems we initially set out to solve. However there was a fundamental flaw in the first of our goals: requiring application developers to go out of their way to view this information is still asking too much. Not only are they responsible for a large number of apps, they need to check periodically because new issues could be reported. Furthermore, they would ideally know when they are introducing a bad dependency at the time that they do it, not after the app has already been built and published to Vantage. We iterated on this process by integrating the check for bad dependencies into our build process itself via our vantage-gradle plugin. As part of the gradle build, the plugin will query vantage for issues affecting the project’s dependencies and display those to the user running the build. We even go so far as to allow the plugin to break the build entirely if the severity of the issue is high enough. This strategy gets the relevant information in front of the developers early, and at a stage where it’s easy for them to make the necessary fixes.
If Vantage sounds like something you might be interested in, the best place to start is to clone it from Github and follow the instructions on loading sample data. This will load Vantage with a set of example data and let you explore its functionality. Actually publishing your dependencies to Vantage requires using the REST api. The easiest way to do so is to use the vantage-gradle plugin as it provides the integration out of the box. However there is nothing inherent to Vantage regarding gradle or even java/jvm build tools. Plugins or external scripts could be written for any number of other languages or build tools. We simply started with gradle as that is our primary build tool at Yodle. Vantage requires this self-reporting of dependency information as analyzing the text of the build files or the artifacts of the build process provides an insufficient level of detail. Build tools often have powerful dependency resolution rules which may result in different versions or even entirely different modules being used than are requested as dependencies. Transitive dependencies or test dependencies are often not present in the build artifacts. While this represents a higher barrier of entry, we believe that the additional information is necessary to provide a quality solution.
Switching to a microservices architecture solves a lot of problems, but it introduces a number of new problems around dependency management. As an organization scales, it becomes harder and harder to reason about the state of your organization’s dependencies. Versions fragment, the number of possible dependency combinations grows, and already fixed bugs linger because the right people don’t know that they’ve already been fixed. It starts acting less like one organization with shared strengths and knowledge and more like multiple organizations each having to solve the same problems in isolation.
If there’s one thing to take away from reading this, it’s that dependency management in the face of moving to microservices is a problem you need to have a plan for before you think you need it. These problems may not always cause your system to break outright, but they creep up over months or years and impose a tax on your organization, much like any tech debt does. It’s easy to throw your hands up and accept these problems as a cost of doing business, but we believe that there’s an opportunity for processes and tools like Vantage to fight back against that growing mountain of dependency related tech debt. And we hope that others can find it useful too.