REST API (4): Emperor’s [New?] Clothes
Let’s talk about your motivation to take the REST route. How much thought have you given this? Did you see how you benefit from the approach and whether its limitations and constraints match your reality and as don’t impede it? What alternatives did you consider? Or did you just go with what you observed to be the flow and/or the only option? Are you able to formally justify your decisions yourself with positive outcomes in your specific case, without referencing statements of others, including mine?
What kind of “REST” is it?
Why is that important? Because there appear to be at least three different opinions on this in any room with two different people and we must be able know which arguments apply, as they don’t always. Some examples:
“Original”: A “Representational State Transfer” architectural style defined by the PhD dissertation of Mr. Roy Fielding from 2000, that leaves most details out to the API designer and focuses instead on goals and constraints. Could be but does not need to be HTTP, JSON or otherwise.
“Evolved”: An opinionated and strict extension of (1), applying what we learned since then to specifically HTTP, JSON, HATEOAS, mapping of verbs and nouns that only permits causing effects by clients submitting self-sufficient intended states to server that do not depend on previous requests – i.e., no RPC, no action endpoints.
“Pragmatic”: Just an easy-to-say, single word term for mapping any form of (action) requests to HTTP requests (endpoints) in which inputs/arguments are passed as HTTP headers, query string arguments and request bodies and results are passed as HTTP response bodies in any form. If you can specify it using an Open API specification, it is a “REST API”.
While not comprehensive, the above list does illustrate a continuum of understandings in the wild. It begins to look like the Emperor’s New Clothes story – everyone is pretending they see something they don’t. I dare you to do a bit of research and investigate some “REST” APIs out there to see where they fit, regardless of where they say they do. Here are my observations:
“REST” is a very popular term, though it means other things (vacation, break, remainder, …)
Most libraries, frameworks, and tools with “REST” in their names focus on the understanding (3) and let others/you apply the style and constraints of original and evolved definitions.
Most people laughingly brush off original/evolved “camps” and take the pragmatic/business approach from the get-go. This correlates well to my lack of observation of any “REST” API that does not violate other definitions in one way or another.
There are original/evolved “purists” annoyed by this situation. This is substantiated by countless articles, testimonies and job interview questions designed to check if candidates “truly know what REST is.” Mr. Roy Fielding himself published what could be called an outcry about the misappropriation of the term “REST”.
There’s more on this topic in my dedicated earlier post, in case you’re interested. What’s important there is to recognize that there is a division and confusion about this that’ll come into play soon.
Getting the skills
Suppose that your team does not have the skills needed to design a REST API. What can you do about this? Should you decide to hire already skilled people how would you go about this and what would be the outcome recognizing the division/confusion noted earlier? How many candidates would have the understanding you desire vs. how much would you have to (re)train them? What new skills are they bringing in that case?
Let’s dig deeper. Beyond already knowing concepts that are adjacent to REST but aren’t REST, such as HTTP, use of libraries and frameworks, Open API specifications and similar, what else could (and should) people know or learn about to be effective at creating good REST APIs? If you’re on the extreme pragmatic end the answer could be “very little, perhaps nothing”. It would be easier to find relevant workforce, but your API will also bear little to no similarity to others beyond the basics. Newcomers would have to learn the particulars of your API and that expertise would not be transferable to others.
If you’re in the less pragmatic camp and care about the correct application of the style, you’ll, of course, be looking for more. Your team will have to acquire the ability to correctly break down resources and fight opposing forces. Do check those posts if you haven’t already. Understanding those challenges will be important going forward. The people you want to have will have to know these and be able to reason about them. As we continue, do think about how much an existing experience solving these challenges is transferrable to other projects. How long would it take to hire someone with matching skillset or to train them to think the same way you do?
Designing the API
You now have your star team. They know everything you need them to know about the mechanics and constraints. They are welcomed to design the new, shiny REST API. They press on to figure out solutions to the challenges at hand, namely carving out the resource types and balancing their decisions against the architectural forces. These are not easy problems to solve and there is rarely, if ever, an exact science to them. Even if perfect answers exist, they usually must change if anything else changes – and you know it will, even before the “release”. This makes it a somewhat artistic exercise highly dependent on initial assumptions. The less assumptions they try to make the harder, bigger the problem gets.
How long will it take to complete just the clean yet workable design? I’m not talking only about the initial sparks of ideas, and I do not constrain this to the time before implementation. All follow-up design adjustments should be included.
The answer will depend on how true to your definition of REST you are. Those in the pragmatic camp will spend very little time as they aren’t following any constraints and can keep adding whatever they need, as they need it. Those in the other two camps will point out that this can bite them back later and will make significant investments into thoughtful design. How long does that take? I personally don’t know as I have no evidence that anyone has done it 100% cleanly, so there is nobody I could ask to get better stats beyond my own. Another way to state the same is “longer than the time available”. At some point all the projects resort to various forms of cheating by merely tunneling RPC. Is your experience different? Are you prepared to defend your design against REST scrutiny?
Implementation
Skill is there and the API specification is ready. You’ve got your libraries and frameworks as well and you know how to use them. How much more work do you have to do?
If you’re in the pragmatic camp, you continue your smooth sailing. At worst it is little more than mapping the action (RPC) endpoints to your code, each dedicated to a use case as you need it, as simple or complex as you want it. This continued smooth sailing is one of the most major factors for the popularity of the pragmatic approach. It gets stuff done quickly.
Did you cringe at that? Do you (both) belong to another camp and know this shouldn’t be about action endpoints but about transferring current and intended states? How much time will you invest into efficiently composing the current state to be sent in the response? Better yet, how much effort is needed to deal with validation and application of intended states (a.k.a. “updates” and “creates”) considering the possibility of other concurrent updates, state transition validation, authorization, and more? If it is unclear what I’m talking about or if it seems trivial, do read my earlier post I mentioned already.
How many resource types will you have to do this for? If you have multiple APIs you’re working with (say microservices), include those. Then multiply the two numbers. Here are some from my past: a star developer, after getting fully up to speed, was able to get a resource type covered in about two weeks (adjusted for overtime), for the first iteration, but it took multiple times that for more complex types. This was using the existing business and data access layers, not reimplement them. There were a few hundred resource types. That translated to about a decade of effort needed. How much does that cost both in compensation and lost opportunities to do other things?
Evolution
Your API is out. Performance complaints and demands start pouring in. New features and functionality are requested. You have the following tools in your belt:
Performance issues only:
Improve the implementation of existing “endpoints” without changing inputs or outputs.
Leverage caching, knowing that it only helps with good cache hit rates to authorized or unprotected content. Both clients and servers must be updated to, somehow, manage the risk of working with stale states. It’s not just about showing the obsolete states to users but also about workflows that follow.
Performance and/or functionality:
Enhance the existing endpoints to allow clients to be more specific about what they need, while adversely affecting caching and requiring client code changes. Requiring more input or changing anything in the response will mean a breaking change (expected data missing, client cannot handle the unexpected).
Adress N+1 and other more complex issues by designing and implementing brand new endpoints that overlap in scope with the existing ones, making them non-canonical and further affecting caching, also requiring client code changes.
How long do those take? The second group involves more of the “going back to the whiteboard” and brings on the challenges of. If you need to version your API and you have some experience doing it, you’ll know that to do this safely you’ll either have to version the entire API and expose each endpoint in each version within its life span or resolve other issues. If you haven’t, see the “Evolution vs Compatibility” section of my previous post on REST. How long will it take to accomplish either?
Love affairs or boxing matches?
We live in the API world. Everything we do needs it. Very often our APIs don’t have clients that are fully known. We may be developing and controlling the initial set to serve own web UI and mobile apps, for example. Then come public APIs with custom clients and enterprise customer integrations. You may develop BFFs for the clients you make but not for others if you’re honest (as you can’t really design for their needs on time). Since BFFs (back ends for front ends) are meant to also be BFFs (best friends forever), they need to be designed with their use cases in mind, each yielding its own, optimized, version of the API, requiring separate design, implementation, and evolution efforts. There’s that multiplication again. That decade-worth effort I mentioned was only for the in-house web UI. Be my guest: multiply it by whatever factor not insignificantly greater than 1 that you chose. Then there is a surprise: what is behind those BFFs that do work with the same concepts, data? What do they use to get their stuff done? Who designs and implements that? How much does that cost?
Pragmatists have another option: instead of creating separate BFFs simply merge them and create one API with a variety of endpoints that together can serve the needs of all. Both have the same challenge here, though – they are faced with increased work needed to support all client types while not actually knowing all most of the clients. This yields assumptions and, typically, bloated states that attempt to reduce development effort by covering multiple use cases. The servers keep sending clients ever greater, bloated responses while clients must resort to rapid fire requests due to remaining N+1 issues. In this exchange of punches neither party is happy. Both the client-serve computer and team/developer communication start to look more like a boxing match than a love affair. This is common to all camps, though the work is different.
What if?
What if we reassess the API styles, approaches and constraints and come up with our own? Aspirational goal could be that only one API trip would be needed for a use case, no more than a single trip per click in the UI, while maintaining efficient communication without bloat. As we don’t know the clients upfront, they would need the ability to express their needs more clearly and comprehensively within those single trips and get everything they need done at once. That is, by definition, not possible with the purist view of REST. It is also beyond most pragmatists’ readiness to do things because it requires significant work to come up with flexible ways to express those requests.
That didn’t stop people from doing it. Many have done it, but the work remained proprietary and, hence, didn’t flourish. Some have published their work while continuing to call their approaches “REST” with different levels of success and complexity, e.g., OData and TreeQL (there may be more). Others have abandoned REST altogether, say GraphQL. Have you looked at these? Are you sure you know what they are? My statistics says that there are many misconceptions including that REST clients can’t access those trivially or that they are somehow dangerous. If you are an API designer, you owe it to yourself to see how they compare. Without that, can you list a single benefit of a REST API style as it applies to you and be able to defend it?
Related
In this series
See also
