The risks of using custom extensions with formal standards

Pawel Uchida-Psztyc
5 min readAug 12, 2021

In the following article, I reference RAML and OAS as these are very close to my specialty and the industry I work in. However, the concepts described in this article can be applied to any formal model.

It is expected to see developers using formal standards in their projects and systems. Often, it is the easiest and fastest way to start a project. Take RAML or OAS to describe APIs. We use these standards to describe how systems communicate with each other formally. As a result of using standards, we can build tooling that works with them. This can be API documentation generators, visual designers, auto-generated tests, etc. The problem starts when we hit the wall trying to describe a custom behavior of our system. Both RAML and OAS are missing a few key concepts to describe the semantics of an API adequately. Consider describing the “list“ HTTP operation on a resource collection. Both RAML and OAS do not offer a formal way of describing a GET operation as reading a collection of resources. Using just the formal description, the GET operation on a collection equals the GET operation on a resource. But these are entirely different operations!

GET operation on a collections is usually understood as reading a list of resources with pagination, meaning returning a schema properties that allow you to request another page of results. The GET operation on a resource, however, always returns the resource. At least in a RESTful systems.

In such a situation, we are tempted to extend the formal language by custom properties to describe this custom behavior. In RAML, you could use the annotations. We can annotate a GET operation to inform any application that this HTTP operation has pagination, and it should expect some input parameters (like page and limit). OAS allows extending the description by defining custom properties whose names are starting with x-. That’s how you would describe Amazon’s API gateway authorization type. It’s not uncommon to see a security scheme description for an Amazon gateway like the following:

x-amazon-apigateway-authtype: oauth2
...

Custom domain properties (which is how you would formally call annotations) allow an API designer to specify the custom behavior of the API. These extensions are most likely used somewhere in the organization’s internal systems. But here’s the problem: you can’t expect any application that reads RAML or OAS to process this information correctly. Both specifications state that the processors can ignore these properties. For example, take the API Console by MuleSoft, which generates API documentation from RAML and OAS. It processes extensions but renders this information as text that has no significance to the application. Surely, the authors of this application could implement better support for an extension, but this is not scalable and probably would not happen. There are a few reasons for that:

  • Discoverability — the authors need to be aware of the custom domain property description, as do the developers, who are the target audience.
  • Usage — how many people would use these extensions? Is it worth it to use your available resources to implement a feature that 0.1% of your users use?
  • Environment — would the extension only work in a particular system?

As I mentioned, we model APIs to have a formal description of systems, and also we can use tools built for the language. Once we add custom extensions, we can’t rely on the tooling anymore. This may result in the whole API design process being seen as less critical or helpful.

This leads to even more problems. My organization uses extensions and annotations to add metadata to an API specification. These extensions are supported inside our platform. But it doesn’t mean across the platform. Just because one application starts using an extension doesn’t mean another team developing another product knows about the new extension or has the resources to implement it. This can create a situation where a client can design an API with an extension but cannot use this information to test the API, even inside the same platform. On the other hand, there are external applications. We can’t expect any of them to read the same information as it is vendor-specific. This significantly reduces interoperability between our apps and 3rd party systems. This adds to the technical debt that otherwise could’ve been avoided.

So what can we do instead?

I spent a lot of time thinking about this problem as this is what I deal with almost every day. We need custom domain properties, but standardization is the only path forward. A word that a lot of people are afraid of. Mainly because this means weeks or months of discussions with external teams, and the effect is not guaranteed. But yes, these custom extensions that could benefit other parties should be standardized on the language level, not an extension. So, instead of using an extension to describe pagination in a REST API, we should be able to do this in RAML and OAS. It benefits more than a single organization or a team, so it shouldn’t be considered an optional extension. This way, we allow the tools that process the formal description to use this standard. And as is, they will have to implement support for it. Consequently, a thing described in my application can be used in another application that processes the same language.

As I mentioned before, a way to standardize a formal description is long and painful. Everyone who ever worked on any standard knows that. But we have to start somewhere. This could slow down the development process of any system. So consider the following scenario. We design an application that processes a formal language. Because of a lack of vocabulary in this language, we develop an extension. In the next step, we reach out to the maintainers of the formal language to present a new use case, the problem, and the solution — the prepared and tested in the field extension. We continue development with the extension but keep a roadmap item to replace the extension with the applied patch to the language specification. Given that we presented a solid case for the formal language maintainers, we probably will succeed, and therefore, our initial idea becomes something that others can use.

I am not saying we should not use custom domain properties in formal languages. They are needed and often the only way to achieve our goals. However, we should be very considerate when deciding to use one with the end goal of formalizing these extensions as a standard. Only this way can we be open to external communities and ensure that our applications work with 3rd party applications. This makes our products more attractive to clients and communities.

--

--

Pawel Uchida-Psztyc

Design, APIs, front-end, strategy, product, and educator. I work for big tech but I share my personal views and opinions.