ramblings on PHP, SQL, the web, politics, ultimate frisbee and what else is on in my life
back 1  2  »  

API versioning in the real world

We here at Liip are currently building a JSON REST API for a customer. At least initially it will only be used by 3 internal projects. One of which we are building and the 2 others are build by partner companies of the customer. Now we want to define a game plan for how to deal with BC breaks in the API. The first part of the game plan is to define what we actually consider a BC break and therefore requires a new API version. We decided to basically define that only removed fields, renamed fields or existing fields who's content has changed should be a BC break. In other words adding a new field to the response should not be considered a BC break. Furthermore changes in how results are sorted are generally not to be considered a BC break as loading more data or an upgrade of the search server can always result in minor reordering. However we would consider a change in any defaults to be an API increase (f.e. changes in default sort order) or changing the default output from "full" to "minimal" to be a BC break. But I guess we would not consider changing from "minimal" to "full" as a BC break as it would just add more fields by default. That being said, for caching reasons, we try not to work with too many such defaults anyway and rather have more requirement parameters. With these definitions ideally we should only rarely have to bump the API version. But there will be the day were we will have to none the less.

First up we do not want to use the URL to version the resources for the obvious reason that this violates the concepts of REST, as this would imply that "/v1/foo" and "/v2/foo" are not the same resource. Remember the "RE" in REST stands for "REpresentational" which means we are talking about representation of state. Therefore a single URI should be used by resource as the unique identifier. Instead to get different representations we should use media types. There isn't really a universally accepted standard for how to encode version information into a media type. So far so bad. To make things worse it gets a bit iffy to define custom media types (ie. "application/vnd.my_api+v1.1") for different versions as then you would logically also return that as the Content-Type in the respons. This in turn will make generic tools not able to pick up that the response is in fact JSON if its not just "application/json". A convention to at least make it human understandable is to add "+json" to the custom media type and I guess clients like browsers could be made to understand this convention. Also it seems like browsers also sometimes ignore the Content-Type entirely and simply try to guess by looking at the actual content.

Then again there is no standard that defines what a web application is actually supposed to do with an Accept header in the strict sense. Sure there is the "q" parameter which defines the priorities of the different media types in the Accept header. But technically it is up to the server to decide what to make of these priorities and as far a I know it can also choose to respond with any media type it wants to. Meaning a request with "Accept: application/vnd.my_api+json+v1.1" could come back with "Content-Type: application/json". Obviously if the server does not support the requested media type (f.e. it never did or no longer does) it should return a 416 HTTP status code.

However this brings us to the next issue: Should we have separate version numbers for different parts of the API? Having different versions would make sense if we will likely have releases that will touch smaller parts. But this will complicate the life of the API user, who would then need to worry about sending the proper media types for different parts of the API. But a release often approach might still make this quite sensible as long as we then keep older versions supported for a longer time. However if we have bigger releases it might be easier for everyone if we just increment the entire API if there are any BC breaks. Given that we have a small group of API users we might then even force everyone to update to the new API within a defined timeframe. This could be nice because then with every such big release we would remove potentially deprecated code for previous versions.

But this brings up to the topic of caching. Caching really requires that we also include the version in the response. So returning "Conent-Type: application/json" would be a no go. It would be better if we would return the actual version of the returned structure in the response, so we are back at returning "Content-Type: application/vnd.my_api+json+v1.1". This way we could ensure we do not return duplicates into the cache, even if parts of the API got their version number incremented without any actual change in the response. But actually it does not really help us that much to look at the response. For doing the actual cache lookups on a request to be able to determine if we have a response cached we obviously only have the request data: we would need to find a solution so that for example the reverse proxy knows that for specific parts of the API "application/vnd.my_api+json+v1.1" should actually use "application/vnd.my_api+json+v1.0" as the cache key lookup. But this raises the even bigger question, how does Varnish deal with content type negotiation in general? How will varnish figure out how to deal with more complex Accept headers like "Accept: application/vnd.my_api+json+v1.2; q=1, application/vnd.my_api+json+v1.1; q=0.6, application/vnd.my_api+json+v1.0; q=0.5". Effectively one would need to handle the entire content type negotiation inside the reverse proxy in order to do the correct cache lookups. And as stated before, this would even need to be aware of the fact that f.e. "/foo" might already be at "1.2" while "/bar" might still be at "1.0". I guess this issue could be handled via the same trick I employed for authentication, ie. translating requests into HEAD requests to let the web application figure this out, but this will add overhead to every request. So at this point I am scratching my head and wondering how to proceed.

Update: An interesting article on managing API evolution discussing what should trigger a version increment and how to plan ahead to prevent that.

Update 2: Another interesting article, this one about the costs of different API versioning strategies.


Re: API versioning in the real world

One solution I could see at this point would be the following:
We maintain a mapping of version supported by path in Varnish. Ideally this mapping file would be generated from the actual running code to prevent config errors. So we have a mapping like:

/foo => [application/vnd.my_api+json+v1.1: application/vnd.my_api+json+v1.0, application/vnd.my_api+json+v1.0: application/vnd.my_api+json+v1.0]
/bar => [application/vnd.my_api+json+v1.2: application/vnd.my_api+json+v1.2, application/vnd.my_api+json+v1.1: application/vnd.my_api+json+v1.1, application/vnd.my_api+json+v1.0: application/vnd.my_api+json+v1.0]

This would enable us to look at the Accept header and determine if the requested media type is supported for the given path. Then we will pick the highest version. Note that the versions supported are the keys and the values is the version that will be returned.

So for example a request:

GET /foo HTTP/1.1
Host: api.example.com
Content-Type: application/json; charset=utf-8
Accept: application/vnd.my_api+json+v1.1, application/vnd.my_api+json+v1.0

We would in Varnish see that in fact we support both "application/vnd.my_api+json+v1.1" and "application/vnd.my_api+json+v1.0". But as we prefer the higher version we will go with "application/vnd.my_api+json+v1.1". However we then see that it would return the content for "application/vnd.my_api+json+v1.0". So for the cache look up we would consider the request as if it would have been:

GET /foo HTTP/1.1
Host: api.example.com
Content-Type: application/json; charset=utf-8
Accept: application/vnd.my_api+json+v1.0

Re: API versioning in the real world

In case custom media types break for your clients not identifying json properly, I see another option to separate the cache for each version: Send the API version in a different header (X-API-Version for instance) and add it to the Vary list.
It may make things more complex for the cotent negotiation in Varnish itself though.

Unrelated, but the link to Text_Wiki rules above your comment form is broken.

Re: API versioning in the real world

Yeah, a custom header is indeed a possibility. One that we discussed. In theory one could even move the version in the request to a custom header. It makes parsing easier but I feel it is not worth it to invest a new standard here.

BTW the "X-" prefix for custom headers has been deprecated.

Re: API versioning in the real world

Besides feeling icky, you didn't really detail what's wrong with placing the version in the URL. That makes life far easier for client developers since they can easily use their browser to play with the API, is a common practice, and is practically equivalent to using a custom header.

Re: API versioning in the real world

The reason why we use media types to begin with. Users of the API will store the URL for references. If he URL is no a premalink you run into problems where "/v1/foo" would appear to be something different than "/v2/foo". For the same reason one should not use "/foo.json" as the URL because that would appear to be something different than "/foo.xml".

Re: API versioning in the real world

But "/v1/foo" and "/v2/foo" may actually be two different representations, no?
Say I have an entity "foo" with the fields firstname="Peter" and lastname="Smith". "/v1/foo" might return these two fields but "/v2/foo" might actually just return one field fullname="Peter Smith" so while the resource is the same it's *representation* is actually different which would mean having different URLs would actually be fine I think.
In the case of "/foo.json" and "/foo.xml" both the resource and the representation are the same so the URL should indeed be the same (without the extension). The only thing different is the encoding in this case.

Re: API versioning in the real world

No, different representations should not be different URLs. The key thing to understand here is that json vs. xml is no different than v1 vs. v2. Maybe its clearer when comparing png vs. html. All are different representations of the same thing. If I have an address I can have different representations of the address, a map, a picture, a painting, a newer map .. these are representations and should not change the address.

Re: API versioning in the real world

First of all, I want to state that I'm completely with you when it comes to add versioning via HTTP headers. I don't know why people keep coming up with the idea of adding the version to the URI. It's just wrong from a RESTful perspective :-)

So basically it doesn't really matter how we do it as long as we do it with HTTP headers. I thought about the custom "API-Version" header a bit and it turned out that I found it confusing. I really prefer something like "Content-Type: application/vnd.my_api+json+v1.2" because that header best represents what it actually is: The content of this resource is represented in "the JSON format and version 1.2 of my API".
So in my opinion, this is the way to go.

Regarding your caching issues having "/foo" at v1.2 and "/bar" at v1.0 I'm scratching my head too. I guess you'll have to teach Varnish semantic versioning so that it can check for a request on version "1.0" if there's already "1.2" in the cache and return this one if so. HEAD requests would be an alternative, yes. But indeed, feels strange to have a reverse proxy that needs to be tricked only for versioning...

Re: API versioning in the real world

It's funny how such simple-to-solve stuff in traditional RPC approaches makes the often as so-much-simpler glorified REST interfaces look bad.

1  2  »