Catching breaking changes with apigenie diff
Andriy Mykulyak · · 6 min read
Why I care about breaking changes
When you work API-first, the description of the API is the contract. Clients are generated from it, mocks answer from it, and other teams plan their work around it. That is the whole point, and it is also the risk: a small edit to the description can break code you have never seen.
The edits that do the damage rarely look dangerous. Someone removes an
operation nobody seems to use. Someone makes a query parameter required, or
renames a Protobuf field, or tightens a GraphQL argument type. In review, these
are one-line changes in a long YAML or .proto file, and they are easy to
approve.
I wanted a tool that reads both versions of the description, tells me exactly
what changed, and says which of those changes will hurt consumers. That tool
is apigenie diff.
What it compares
The command takes two files, the old one and the new one:
apigenie diff [OPTIONS] SOURCE_FILE DEST_FILE
Both files must be of the same kind: both OpenAPI 3.x documents, both Swagger
2.0 documents, both AsyncAPI 3.x documents, both Protobuf files or both
GraphQL files. The differ picks the format from the file itself: a swagger
field at the root means Swagger, an asyncapi field means AsyncAPI, a .proto
extension means Protobuf, and .graphql, .graphqls or .gql means GraphQL.
Each format is compared the way its consumers see it:
- OpenAPI and Swagger. Paths, operations, parameters, request bodies, responses, schemas, security and so on. The severity of a schema change depends on the direction of the data: a change in a request is judged by what it does to callers, a change in a response by what it does to readers. Read-only properties don’t matter in requests.
- AsyncAPI. Servers, channels, operations, messages with their payloads and headers, security schemes, bindings and components. Here the direction is decided by who sends the message. A new required payload property breaks the senders of a message the application receives, while the receivers of a message the application sends can ignore it.
- Protobuf. Messages, enums, services and extend blocks, down to fields, enum values and methods. Fields and enum values are matched by their numbers, because the numbers identify them in the encoded data. So a renamed field is a modified field, and a field that got another number is a removed one plus an added one. Removing a field is non-breaking when the new file reserves its number, which is how Protobuf retires fields.
- GraphQL. The type system: root operation types, types of every kind with
their fields, arguments and enum values, directive definitions and applied
directives. Type extensions are merged into what they extend, so moving a
field into an
extend typeblock is not a change. Breaking changes follow what graphql-js’sfindBreakingChanges()decides.
The diff reference describes the matching rules of each format in detail.
Severities
Every change gets one of three severities:
- breaking: existing consumers stop working, for example a removed operation or a field whose type changed incompatibly;
- non-breaking: the API changes, but existing consumers keep working, for example an added operation or an added optional parameter;
- non-functional: nothing changes for the programs involved, for example descriptions in GraphQL or comments in Protobuf.
The defaults come from the recommended rules of each format. They are
reasonable, but every team has its own contract with its consumers, so the
rules are configurable in the commands.diff block of the
configuration file. A property can be
set to breaking, non_breaking, non_functional, none to ignore the
change, or nested to take the highest severity of the changes inside. Rules
can depend on conditions and can be overridden for specific locations. For
example, this lets deprecated operations be removed without failing the build:
version: '1.0'
commands:
diff:
extends: [recommended]
rules:
pathItem:
operations:
remove_item:
if: deprecated
then: non_breaking
else: breaking
The swagger, asyncapi, protobuf and graphql sections of the same block
configure the other formats.
Reading the output
By default, apigenie diff prints the changelog report with the stylish
formatter. Here is the beginning of what it prints for two versions of the
Petstore example:
$ apigenie diff petstore.source.yaml petstore.dest.yaml
petstore.dest.yaml
1 1 breaking Metadata changed
98 7 non-breaking Added operation PUT /pets/{petId} (updatePet)
87 7 breaking Removed operation DELETE /pets/{petId} (deletePet)
25 7 breaking Modified operation GET /pets (listPets)
25 7 breaking Added operation.parameters['query:status']
...
Each line has a position in the destination file, a severity and a description. Changes inside a modified operation are indented under it. For Protobuf and GraphQL, what has been removed is reported at the closest enclosing declaration that is still there, as in this Protobuf example from the reference:
shop.proto
19 1 breaking Modified message 'Customer'
19 1 non-breaking Removed field 'fax'
25 3 breaking Modified field 'full_name'
25 3 breaking Modified field name
Two options control the output. --report chooses what to print: status,
changelog (the default) or detailed. --formatter chooses how: html,
json, markdown, stylish (the default), unix or visualstudio. A
Markdown changelog is handy for release notes, JSON for further processing:
apigenie diff --report detailed --formatter markdown v1/openapi.yaml v2/openapi.yaml
Failing CI on breaking changes
The exit status is what makes the command useful in a pipeline. It exits with status 1 when there are breaking changes, and 0 otherwise. Any CI system can use that directly:
git show main:api/openapi.yaml > /tmp/openapi.base.yaml
apigenie diff --config apigenie.yaml /tmp/openapi.base.yaml api/openapi.yaml
The exit status considers all changes, whatever the report shows, so choosing a shorter report doesn’t hide a breaking change.
There is also --update-version. Instead of printing a report, it updates
info.version of the destination file to the smallest version compatible with
the changes: a new major version for breaking changes, minor for non-breaking
and patch for non-functional ones. It keeps the rest of the file intact,
prints nothing and exits with 0. Protobuf and GraphQL files have no version,
so the option is rejected for them.
In the GitHub Action
If your API lives on GitHub, the API Genie GitHub Action runs the differ for you:
on: pull_request
jobs:
api:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: mykulyak/api-genie-action@v0
with:
command: diff
files: api/openapi.yaml
With command: diff, each file is compared with its version at the base
commit of the pull request, or the commit before a push, and the step fails on
breaking changes. Set base to compare with another commit, branch or tag, or
base-file to compare two files of the repository.
The changes show up as annotations on the files of the pull request, so the
reviewer sees “breaking” next to the line that caused it, and a report is added
to the job summary. Annotations come from the default unix formatter, so
leave formatter alone if you want them. The report input picks the report,
as --report does on the command line. On pull requests, the action can also
post the report as a comment and update it in place on later runs; that needs
the pull-requests: write permission.
If you would rather decide yourself what a breaking change means for the
build, set fail-on: never and read the status output, which is none,
non-breaking or breaking.
The history view: apigenie changelog
apigenie diff compares two files you give it. Often the question is
different: what happened to this API over the last releases? The
changelog command answers it from git
history. It compares the consecutive committed versions of an OpenAPI, Swagger,
AsyncAPI, Protobuf or GraphQL file and rates each change with the same rules
as diff, including your commands.diff configuration.
apigenie changelog api/openapi.yaml --revisions tags --tag-pattern 'v*' --unreleased --formatter markdown
This prints one entry per release tag, plus the changes made since the last
release, in Markdown that can be published as is. With --by operation, the
changes are grouped by operation instead, so you can see the history of a
single endpoint. Unlike diff, changelog exits with 0 even when there are
breaking changes: it describes history, it doesn’t gate it.
Where to start
Take the last released version of your API description and the current one,
and run apigenie diff on them. If the output surprises you, that is exactly
the change that would have surprised your consumers. Then put the same check
in CI, so the next one is caught in review instead of in production.