Blog entries tagged rest :: mwop.net

Notes on GraphQL

2018-07-18T17:05:00-05:00

The last week has been my first foray into GraphQL, using the GitHub GraphQL API endpoints. I now have OpinionsTM. The promise is fantastic: query for everything you need, but nothing more. Get it all in one go. But the reality is somewhat... different. What I found was that you end up with a lot of garbage data structures that you then, on the client side, need to decipher and massage, unpacking edges, nodes, and whatnot. I ended up having to do almost a dozen array_column(), array_map(), and array_reduce() operations on the returned data to get a structure I can actually use. The final data I needed looked like this: [ { "name": "zendframework/zend-expressive", "tags": [ { "name": "3.0.2", "date": "2018-04-10" } ] } ] To fetch it, I needed a query like the following: query showOrganizationInfo( $organization:String! $cursor:String! ) { organization(login:$organization) { repositories(first: 100, after: $cursor) { pageInfo { startCursor hasNextPage endCursor } nodes { nameWithOwner tags:refs(refPrefix: "refs/tags/", first: 100, orderBy:{field:TAG_COMMIT_DATE, direction:DESC}) { edges { tag: node { name target { ... on Commit { pushedDate } ... on Tag { tagger { date } } } } } } } } } } Which gave me data like the following: { "data": { "organization": { "repositories: { "pageInfo": { "startCursor": "...", "hasNextPage": true, "endCursor": "..." }, "nodes": [ { "nameWithOwner": "zendframework/zend-expressive", "tags": { "edges": [ "tag": { "name": "3.0.2", "target": { "tagger": { "date": "2018-04-10" } } } ] } } ] } } } } How did I discover how to create the query? I'd like to say it was by reading the docs. I really would. But these gave me almost zero useful examples, particularly when it came to pagination, ordering results sets, or what those various "nodes" and "edges" bits were, or why they were necessary. (I eventually found the information, but it's still rather opaque as an end-user.) Additionally, see that pageInfo bit? This brings me to my next point: pagination sucks, particularly if it's not at the top-level. You can only fetch 100 items at a time from any given node in the GitHub GraphQL API, which means pagination. And I have yet to find a client that will detect pagination data in results and auto-follow them. Additionally, the "after" property had to be something valid... but there were no examples of what a valid value would be. I had to resort to StackOverflow to find an example, and I still don't understand why it works. I get why clients cannot unfurl pagination, as pagination data could appear anywhere in the query. However, it hit me hard, as I thought I had a complete set of data, only to discover around half of it was missing once I finally got the processing correct. If any items further down the tree also require pagination, you're in for some real headaches, as you then have to fetch paginated sets depth-first. So, while GraphQL promises fewer round trips and exactly the data you need, my experience so far is: I end up having to be very careful about structuring my queries, paying huge attention to pagination potential, and often sending multiple queries ANYWAYS. A well-documented REST API is often far easier to understand and work with immediately. I end up doing MORE work client-side to make the data I receive back USEFUL. This is because the payload structure is based on the query structure and the various permutations you need in order to get at the data you need. Again, a REST API usually has a single, well-documented payload, making consumption far easier. I'm sure I'm probably mis-using GraphQL, or missing a number of features to make this stuff easier, but so far, I'm left wishing I could just have a number of useful REST endpoints that I can hit consistently in order to aggregate the data I need. Before anybody suggests it, yes, I am very aware that GitHub also offers a REST API, and the v3 API has endpoints for most of what I needed. However, I had to rely on tags, not releases, as not all of our tags have associated releases. However, the data returned for tags does not include the commit date; for that, you need to fetch the associated commit, and then the date may be under either the author or the committer. This approach would have meant literally thousands of calls to get the data I need, which would have had me hitting rate limits, and potentially taking hours to complete. My point: perhaps instead of GraphQL, aggregating a bit more data in REST resources (e.g., including commit data with tags), or providing endpoints that allow merging specific resource types could have solved the problem easily. This is where having a developer relations team that finds out what data consumers are needing comes in handy, instead of simply mandating graphql all the things to allow infinite flexibility (and the frustrations of such flexibility, both for the API developer and consumer). Updates 2018-09-19: syntax highlighting fixes. Notes on GraphQL was originally published 18 July 2018 on https://mwop.net by Matthew Weier O'Phinney.

Apigility: Using RPC with HAL

2014-03-26T15:30:00-05:00

A few days ago, we released our first beta of Apigility. We've started our documentation effort now, and one question has arisen a few times that I want to address: How can you use Hypermedia Application Language (HAL) in RPC services? HAL? Hypermedia Application Language is an IETF proposal for how to represent resources and their relations within APIs. Technically, it provides two mediatypes, application/hal+json and application/hal+xml; however, Apigility only provides the JSON variant. The important things to know about HAL are: It provides a standard way of describing relational links. All relational links are under a _links property of the resource. That property is an object. Each property of that object is a link relation; the value of each link relation is an object (or array of such objects) describing the link that must minimally contain an href proerty. The link object itself can contain some additional metadata, such as a mediatype, a name (useful for differentiating between multiple link objects assigned to the same relation). While not required, the specification recommends resources contain a "self" relational link, indicating the canonical location for the resource. This is particularly useful when we consider embedding (the next topic). Sound hard? It's not: { "_links": { "self": { "href": "/blog/2014-03-26-apigility-rpc-with-hal" } } } Besides link relations, HAL also provides a standard way of describing embedded resources. An embedded resource is any other resource you can address via your API, and, as such, would be structured as a HAL resource — in other words, it would have a _links property with relational links. Essentially, any property of the resource you're returning that can itself be addressed via the URI must be embedded in the resource. This is done via the property _embedded. Like _links, _embedded is an object. Each key in the object is the local name by which the resource refers to the embedded resource. The value of such keys can either be HAL resources or arrays of HAL resources; in fact, this is how collections are represented in HAL! As examples: { "_links": { "self": { "href": "/blog/2014-03-26-apigility-rpc-with-hal" } }, "_embedded": { "author": { "_links": { "self": { "href": "/blog/author/matthew" } }, "id": "matthew", "name": "Matthew Weier O'Phinney", "url": "http://mwop.net" }, "tags": [ { "_links": { "self": { "href": "/blog/tag/php" } }, "id": "php" }, { "_links": { "self": { "href": "/blog/tag/rest" } }, "id": "rest" } ] } } The example above shows two embedded resources. The first is the author; the second, a collection of tags. Note that every object under _embedded is a HAL object! You can go quite far with this — you can also have embedded resources inside your embedded resources, arbitrarily deep. RPC? RPC stands for Remote Procedure Call, and, when describing a web API, is usually used to describe a web service that publishes multiple method calls at a single URI using only POST; XML-RPC and SOAP are the usual suspects. In Apigility, we use the term RPC in a much looser sense; we use it to describe one-off services: actions like "authenticate," or "notify," or "register" would all make sense here. They are actions that usually only need to respond to a single HTTP method, and which may or may not describe a "thing", which is what we usually consider a "resource" when discussing REST terminology. That said: what if what we want to return from the RPC call are REST resources? Returning HAL from RPC Services In order to return HAL from RPC services, we need to understand (a) how Content Negotiation works, and (b) what needs to be returned in order for the HAL renderer to be able to create a representation. For purposes of this example, I'm positing a RegisterController as an RPC service that, on success, is returning a User object that I want rendered as a HAL resource. The zf-content-negotiation module takes care of content negotiation for Apigility. It introspects the Accept header in order to determine if we can return a representation, and then, if it can, will cast any ZF\ContentNegotiation\ViewModel returned from a controller to the appropriate view model for the representation. From there, a renderer will pick up the view model and do what needs to be done. So, the first thing we have to do is return ZF\ContentNegotiation\ViewModel instances from our controller. use Zend\Mvc\Controller\AbstractActionController; use ZF\ContentNegotiation\ViewModel; class RegisterController extends AbstractActionController { public function registerAction() { /* ... do some work ... get a user ... */ return new ViewModel(array('user' => $user)); } } The zf-hal module in Apigility creates the actual HAL representations. zf-hal looks for a "payload" variable in the view model, and expects that value to be either a ZF\Hal\Entity (single item) or ZF\Hal\Collection. When creating an Entity object, you need the object being represented, as well as the identifier. So, let's update our return value. use Zend\Mvc\Controller\AbstractActionController; use ZF\ContentNegotiation\ViewModel; use ZF\Hal\Entity; class RegisterController extends AbstractActionController { public function registerAction() { /* ... do some work * ... get a $user * ... assume we have also now have an $id */ return new ViewModel(array('payload' => array( 'user' => new Entity($user, $id), ))); } } zf-hal contains what's called a "metadata map". This is a map of classes to information on how zf-hal should render them: what route to use, what additional relational links to inject, how to serialize the object, what field represents the identifier, etc. In most cases, you will have likely already defined a REST service for the resource you want to return from the RPC service, in which case you will be done. However, if you want, you can go in and manually configure the metadata map in your API module's config/module.config.php file: return array( /* ... */ 'zf-hal' => array( 'metadata_map' => array( 'User' => array( 'route_name' => 'api.rest.user', 'entity_identifier_name' => 'username', 'route_identifier_name' => 'user_id', 'hydrator' => 'Zend\Stdlib\Hydrator\ObjectProperty', ), ), ), ); Finally, we need to make sure that the service is configured to actually return HAL. We can do this in the admin if we want. Find the "Content Negotiation" section of the admin, and the "Content Negotiation Selector" item, and set that to "HalJson"; don't forget to save! Alternately, you can do this manually in the API module's config/module.config.php file, under the zf-content-negotiation section: return array( /* ... */ 'zf-content-negotiation' => array( 'controllers' => array( /* ... */ 'RegisterController' => 'HalJson', ), /* ... */ ), ); Once your changes are complete, when you make a successful request to the URI for your "register" RPC service, you'll receive a HAL response pointing to the canonical URI for the user resource created! Apigility: Using RPC with HAL was originally published 26 March 2014 on https://mwop.net by Matthew Weier O'Phinney.

RESTful APIs with ZF2, Part 3

2013-02-25T06:29:00-06:00

In my previous posts, I covered basics of JSON hypermedia APIs using Hypermedia Application Language (HAL), and methods for reporting errors, including API-Problem and vnd.error. In this post, I'll be covering documenting your API — techniques you can use to indicate what HTTP operations are allowed, as well as convey the full documentation on what endpoints are available, what they accept, and what you can expect them to return. While I will continue covering general aspects of RESTful APIs in this post, I will also finally introduce several ZF2-specific techniques. Why Document? If you're asking this question, you've either never consumed software, or your software is perfect and self-documenting. I frankly don't believe either one. In the case of APIs, those consuming the API need to know how to use it. What endpoints are available? Which operations are available for each endpoint? What does each endpoint expect as a payload during the request? What can you expect as a payload in return? How will errors be communicated? While the promise of hypermedia APIs is that each response tells you the next steps available, you still, somewhere along the way, need more information — what payloads look like, which HTTP verbs should be used, and more. If you're not documenting your API, you're "doing it wrong." Where Should Documentation Live? This is the much bigger question. Of the questions I raised above, detailing what should be documented, there are two specific types. When discussing what operations are available, we have a technical solution in the form of the OPTIONS method and its counterpart, the Allow header. Everything else falls under end-user documentation. OPTIONS The HTTP specification details the OPTIONS method as idempotent, non-cacheable, and for use in detailing what operations are available for the given resource specified by the request URI. It makes specific mention of the Allow header, but does not limit what is returned for requests made via this method. The Allow header details the allowed HTTP methods for the given resource. Used in combination, you make an OPTIONS request to a URI, and it should return a response containing an Allow header; from that header value, you then know what other HTTP methods can be made to that URI. What this tells us is that our RESTful endpoint should do the following: When an OPTIONS request is made, return a response with an Allow header that has a list of the available HTTP methods allowed. For any HTTP method we do not allow, we should return a "405 Not Allowed" response. These are fairly easy to accomplish in ZF2. (See? I promised I'd get to some ZF2 code in this post!) When creating RESTful endpoints in ZF2, I recommend using Zend\Mvc\Controller\AbstractRestfulController. This controller contains an options() method which you can use to respond to an OPTIONS request. As with any ZF2 controller, returning a response object will prevent rendering and bubble out immediately so that the response is returned. namespace My\Controller; use Zend\Mvc\Controller\AbstractRestfulController; class FooController extends AbstractRestfulController { public function options() { $response = $this->getResponse(); $headers = $response->getHeaders(); // If you want to vary based on whether this is a collection or an // individual item in that collection, check if an identifier from // the route is present if ($this->params()->fromRoute('id', false)) { // Allow viewing, partial updating, replacement, and deletion // on individual items $headers->addHeaderLine('Allow', implode(',', array( 'GET', 'PATCH', 'PUT', 'DELETE', ))); return $response; } // Allow only retrieval and creation on collections $headers->addHeaderLine('Allow', implode(',', array( 'GET', 'POST', ))); return $response; } } The next trick is returning the 405 response if an invalid option is used. For this, you can create a listener in your controller, and wire it to listen at higher-than-default priority. As an example: namespace My\Controller; use Zend\EventManager\EventManagerInterface; use Zend\Mvc\Controller\AbstractRestfulController; class FooController extends AbstractRestfulController { protected $allowedCollectionMethods = array( 'GET', 'POST', ); protected $allowedResourceMethods = array( 'GET', 'PATCH', 'PUT', 'DELETE', ); public function setEventManager(EventManagerInterface $events) { parent::setEventManager($events); $events->attach('dispatch', array($this, 'checkOptions'), 10); } public function checkOptions($e) { $matches = $e->getRouteMatch(); $response = $e->getResponse(); $request = $e->getRequest(); $method = $request->getMethod(); // test if we matched an individual resource, and then test // if we allow the particular request method if ($matches->getParam('id', false)) { if (!in_array($method, $this->allowedResourceMethods)) { $response->setStatusCode(405); return $response; } return; } // We matched a collection; test if we allow the particular request // method if (!in_array($method, $this->allowedCollectionMethods)) { $response->setStatusCode(405); return $response; } } } Note that I moved the allowed methods into properties; if I did the above, I'd refactor the options() method to use those properties as well to ensure they are kept in sync. Also note that in the case of an invalid method, I return a response object. This ensures that nothing else needs to execute in the controller; I discover the problem and return early. End-User Documentation Now that we have the technical solution out of the way, we're still left with the bulk of the work left to accomplish: providing end-user documentation detailing the various payloads, errors, etc. I've seen two compelling approaches to this problem. The first builds on the OPTIONS method, and the other uses a hypermedia link in every response to point to documentation. The OPTIONS solution is this: use the body of an OPTIONS response to provide documentation. (Keith Casey gave an excellent short presentation about this at REST Fest 2012). The OPTIONS method allows for you to return a body in the response, and also allows for content negotiation. The theory, then, is that you return media-type-specific documentation that details the methods allowed, and what they specifically accept in the body. While there is no standard for this at this time, the first article I linked suggested including a description, the parameters expected, and one or more example request bodies for each HTTP method allowed; you'd likely also want to detail the responses that can be expected. { "POST": { "description": "Create a new status", "parameters": { "type": { "type": "string", "description": "Status type -- text, image, or url; defaults to text", "required": false }, "text": { "type": "string", "description": "Status text; required for text types, optional for others", "required": false }, "image_url": { "type": "string", "description": "URL of image for image types; required for image types", "required": false }, "link_url": { "type": "string", "description": "URL of image for link types; required for link types", "required": false } }, "responses": [ { "describedBy": "http://example.com/problems/invalid-status", "title": "Submitted status was invalid", "detail": "Missing text field required for text type" }, { "id": "abcdef123456", "type": "text", "text": "This is a status update", "timestamp": "2013-02-22T10:06:05+0:00" } ], "examples": [ { "text": "This is a status update" }, { "type": "image", "text": "This is the image caption", "image_url": "http://example.com/favicon.ico" }, { "type": "link", "text": "This is a description of the link", "link_url": "http://example.com/" }, ] } } If you were to use this methodology, you would alter the options() method such that it does not return a response object, but instead return a view model with the documentation. namespace My\Controller; use Zend\Mvc\Controller\AbstractRestfulController; class FooController extends AbstractRestfulController { protected $viewModelMap = array(/* ... */); public function options() { $response = $this->getResponse(); $headers = $response->getHeaders(); // Get a view model based on Accept types $model = $this->acceptableViewModelSelector($this->viewModelMap); // If you want to vary based on whether this is a collection or an // individual item in that collection, check if an identifier from // the route is present if ($this->params()->fromRoute('id', false)) { // Still set the Allow header $headers->addHeaderLine('Allow', implode( ',', $this->allowedResourceMethods )); // Set documentation specification as variables $model->setVariables($this->getResourceDocumentationSpec()); return $model; } // Allow only retrieval and creation on collections $headers->addHeaderLine('Allow', implode( ',', $this->allowedCollectionMethods )); $model->setVariables($this->getCollectionDocumentationSpec()); return $model; } } I purposely didn't provide the implementations of the getResourceDocumentationSpec() and getCollectionDocumentationSpec() methods, as that will likely be highly specific to your application. Another possibility is to use your view engine for this, and specify a template file that has the fully-populated information. This would require a custom renderer when using JSON or XML, but is a pretty easy solution. However, there's one cautionary tale to tell, something I already mentioned: OPTIONS, per the specification, is non-cacheable. What this means is that everytime somebody makes an OPTIONS request, any cache control headers you provide will be ignored, which means hitting the server for each and every request to the documentation. Considering documentation is static, this is problematic; it has even prompted blog posts urging you not to use OPTIONS for documentation. Which brings us to the second solution for end-user documentation: a static page referenced via a hypermedia link. This solution is insanely easy: you simply provide a Link header in your response, and provide a describedby reference pointing to the documentation page: Link: ; rel="describedby" With ZF2, this is trivially easy to accomplish: create a route and endpoint for your documentation, and then a listener on your controller that adds the Link header to your response. The latter, adding the link header, might look like this: namespace My\Controller; use Zend\EventManager\EventManagerInterface; use Zend\Mvc\Controller\AbstractRestfulController; class FooController extends AbstractRestfulController { public function setEventManager(EventManagerInterface $events) { parent::setEventManager($events); $events->attach('dispatch', array($this, 'injectLinkHeader'), 20); } public function injectLinkHeader($e) { $response = $e->getResponse(); $headers = $response->getHeaders(); $headers->addHeaderLine('Link', sprintf( '<%s>; rel="describedby"', $this->url('documentation-route-name') )); } } If you want to ensure you get a fully qualified URL that includes the schema, hostname, and port, there are a number of ways to do that as well; the above gives you the basic idea. Now, for the route and endpoint, there are tools that will help you simplify that task as well, in the form of a couple of ZF2 modules: PhlySimplePage and Soflomo\Prototype. (Disclosure: I'm the author of PhlySimplePage.) Both essentially allow you to specify a route and the corresponding template name to use, which means all you need to do is provide a little configuration, and a view template. Soflomo\Prototype has slightly simpler configuration, so I'll demonstrate it here: return array( 'soflomo_prototype' => array( 'documentation-route-name' => array( 'route' => '/api/documentation', 'template' => 'api/documentation', ), ), 'view_manager' => array( 'template_map' => array( 'api/documentation' => __DIR__ . '/../view/api/documentation.phtml', ), ), ); I personally have been using the Link header solution, as it's so simple to implement. It does not write the documentation for you, but thinking about it early and implementing it helps ensure you at least start writing the documentation, and, if you open source your project, you may find you have users who will write the documentation for you if they know where it lives. Conclusions Document your API, or either nobody will use it, or all you're hear are complaints from your users about having to guess constantly about how to use it. Include the following information: What endpoint(s) is (are) available. Which operations are available for each endpoint. What payloads are expected by the endpoint. What payloads can a user expect in return. What media types may be used for requests. What media types may be expected in responses. Additionally, make sure that you do the OPTIONS/Allow dance; don't just accept any request method, and report the standard 405 response for methods that you will not allow. Make sure you differentiate these for collections versus individual resources, as you likely may allow replacing or updating an individual resource, but likely will not want to do the same for a whole collection! Next time So far, I've covered the basics of RESTful JSON APIS, specifically recommending Hypermedia Application Language (HAL) for providing hypermedia linking and relations. I've covered error reporting, and provided two potential formats (API-Problem and vnd.error) for use with your APIs. Now, in this article, I've shown a bit about documenting your API both for machine consumption as well as end-users. What's left? In upcoming parts, I'll talk about ZF2's AbstractRestfulController in more detail, as well as how to perform some basic content negotiation. I've also had requests about how one might deal with API versioning, and will attempt to demonstrate some techniques for doing that as well. Finally, expect to see a post showing how I've tied all of this together in a general-purpose ZF2 module so that you can ignore all of these posts and simply start writing APIs. Updates Note: I'll update this post with links to the other posts in the series as I publish them. Part 1 Part 2 RESTful APIs with ZF2, Part 3 was originally published 25 February 2013 on https://mwop.net by Matthew Weier O'Phinney.

RESTful APIs with ZF2, Part 2

2013-02-13T07:40:00-06:00

In my last post, I covered some background on REST and the Richardson Maturity Model, and some emerging standards around hypermedia APIs in JSON; in particular, I outlined aspects of Hypermedia Application Language (HAL), and how it can be used to define a generic structure for JSON resources. In this post, I cover an aspect of RESTful APIs that's often overlooked: reporting problems. Background APIs are useful when they're working. But when they fail, they're only useful if they provide us with meaningful information; if all I get is a status code, and no indication of what caused the issue, or where I might look for more information, I get frustrated. In consuming APIs, I've come to the following conclusions: Error conditions need to provide detailed information as to what went wrong, and what steps I may be able to take next. An error code with no context gives me nothing to go on. Errors need to be reported consistently. Don't report the error one way one time, and another way the next. DO use HTTP status codes to indicate an error happened. Nothing is more irksome than getting back a 200 status with an error payload. Errors should be reported in a format I have indicated I will Accept (as in the HTTP header). Perhaps the only think more irksome than a 200 status code for an error is getting back an HTML page when I expect JSON. Why Status Codes Aren't Enough Since REST leverages and builds on HTTP, an expedient solution for reporting problems is to simply use HTTP status codes. These are well understood by web developers, right? 4xx error codes are errors made by the requestor, and are actually fairly reasonable to use for reporting things such as lack of authorization tokens, incomplete requests, unsupportable operations, or non-supported media types. But what happens when the error is on the server — because something has gone wrong such as inability to reach your persistence layer or credential storage? The 5xx series of status codes is sparse and wholly unsuited to reporting errors of these types — though you'll likely still want to use a 500 status to report the failure. But what do you present to the consumer so that they know whether or not to try again, or what to report to you so that you can fix the issue? A status code simply isn't enough information most of the time. Yes, you want to define standard status codes so that your clients can perform reasonable branching, but you also need a way to communicate details to the end-user, so that they can log the information for themselves, display information to their own end-users, and/or report it back to you so you can do something to resolve the situation. Custom Media Types The first step is to use a custom media type. Media types are typically both a name as well as a structure — and the latter is what we're after when it comes to error reporting. If we return a response using this media type, the client then knows how to parse it, and can then process it, log it, whatever. Sure, you can make up your own format — as long as you are consistent in using it, and you document it. But personally, I don't like inventing new formats when standard formats exist already. Custom formats mean that custom clients are required for working with the services; using a standard format can save effort and time. In the world of JSON, I've come across two error media types that appear to be gaining traction: application/api-problem+json and application/vnd.error+json API-Problem This particular media type is via the IETF. Like HAL, it provides formats in both JSON and XML, making it a nice cross-platform choice. As noted already, the media type is application/api-problem+json. The representation is a single resource, with the following properties: describedBy: a URL to a document describing the error condition (required) title: a brief title for the error condition (required) httpStatus: the HTTP status code for the current request (optional) detail: error details specific to this request (optional) supportId: a URL to the specific problem occurrence (e.g., to a log message) (optional) As an example: HTTP/1.1 500 Internal Error Content-Type: application/api-problem+json { "describedBy": "http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html", "detail": "Status failed validation", "httpStatus": 500, "title": "Internal Server Error" } The specification allows a large amount of flexibility — you can have your own custom error types, so long as you have a description of them to link to. You can provide as little or as much detail as you want, and even decide what information to expose based on environment. I personally like to point to the HTTP status code definitions, and then provide request-specific detail; I find this gives quick and simple results that I can later shape as I add more detail to my API. However, the specification definitely encourages you to have unique error types with discrete URIs that describe them — never a bad thing when creating APIs. vnd.error This is a proposed media type within the HAL community. Like HAL, it provides formats in both JSON and XML, making it a nice cross-platform choice. It differentiates from API-Problem in a few ways. First, it allows, and even encourages, reporting collections of errors. If you consider PHP exceptions and the fact that they support "previous" exceptions, this is a powerful concept; you can report the entire chain of errors that led to the response. Second, it encourages pushing detail out of the web service; errors include a "logRef" property that points to where the error detail lives. This is probably better illustrated than explained. The response payload is an array of objects. Each object has the following members: logRef: a unique identifier for the specific error which can then be used to identify the error within server-side logs (required) message: the error message itself (required) _links: HAL-compatible links. Typically, "help", "describes", and/or "describedBy" relations will be defined here. As an example, let's consider the API-Problem example I had earlier, and provide a vnd.error equivalent: HTTP/1.1 500 Internal Error Content-Type: application/vnd.error+json [ { "logRef": "someSha1HashMostLikely", "message": "Status failed validation", "_links": { "describedBy": {"href": "http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html"} } } ] vnd.error basically begs you to create custom error types, with documentation end-points that detail the source of the error and what you can do about it (this is true of API-Problem as well). The requirement to include a log reference ("logRef") and have it be unique can be a stumbling block to implementation, however, as it requires effort for uniquely identifying requests, and logging. However, both the identification and logging can be automated. Summary Error reporting in APIs is as important as the normal resource payloads themselves. Without good error reporting, when an API raises errors, clients have difficulty understanding what they can do next, and cannot provide you, the API provider, with information that will allow you to debug on the server side. As noted at the beginning of the article, if you follow the rules below, you'll make consumers of your API happier and more productive. DO use appropriate HTTP status codes to indicate an error happened. Report errors in a format I have indicated I will Accept (as in the HTTP header). Report errors consistently. Don't report the error one way one time, and another way the next. Standardize on a specific error-reporting media type . While you can create your own error structure, I recommend using documented, accepted standards. This will make clients more re-usable, and make many of your decisions for you. Provide detailed information as to what went wrong, and what steps I may be able to take next. Provide documentation for each type of error, and link to that documentation from your error payloads. Which brings me to… Next time I realize I still haven't covered anything specific to ZF2, but I'll start next time, when I cover the next topic: documenting your API. An undocumented API is a useless API, so it's good to start baking documentation in immediately. I'll survey some of the possibilities and how they can be implemented in ZF2 in the next installment, and then we can get our hands dirty with actual API development. Updates Note: I'll update this post with links to the other posts in the series as I publish them. Part 1 Part 3 RESTful APIs with ZF2, Part 2 was originally published 13 February 2013 on https://mwop.net by Matthew Weier O'Phinney.

RESTful APIs with ZF2, Part 1

2013-02-12T05:42:00-06:00

RESTful APIs have been an interest of mine for a couple of years, but due to circumstances, I've not had much chance to work with them in any meaningful fashion until recently. Rob Allen and I proposed a workshop for PHP Benelux 2013 covering RESTful APIs with ZF2. When it was accepted, it gave me the perfect opportunity to dive in and start putting the various pieces together. Background I've attended any number of conference sessions on API design, read countless articles, and engaged in quite a number of conversations. Three facts keep cropping up: JSON is fast becoming the preferred exchange format due to the ease with which it de/serializes in almost every language. The "holy grail" is Richardson Maturity Model Level 3. It's really hard to achieve RMM level 3 with JSON. Richardson Maturity Model As a quick review, the Richardson Maturity Model has the following 4 levels: Level 0: "The swamp of POX." Basically, a service that uses TCP for transport, primarily as a form of remote procedure call (RPC). Typically, these are not really leveraging HTTP in any meaningful fashion; most systems will use HTTP POST for all interactions. Also, you will often have a single endpoint for all interactions, regardless of whether or not they are strictly related. XML-RPC, SOAP, and JSON-RPC fall under this category. Level 1: "Resources." In these services, you start breaking the service into multiple services, one per "resource," or, in object oriented terms, per object. This means a distinct URL per object, which means each has its own distinct identity on the web; this often extends not only to the collection of objects, but to individual objects under the collection as well (e.g., /books as well as /books/life-of-pi). The service may still be RPC in nature, however, and, at this level, often is still using a single HTTP method for all interactions with the resource. Level 2: "HTTP Verbs." At this level, we start using HTTP verbs with our services in the way the HTTP specification intends. GET is for safe operations, and should be cacheable; POST is used for creation and/or updating; DELETE can be used to delete a resource; etc. Rather than doing RPC style methods, we leverage HTTP, occasionally passing additional parameters via the query string or request body. Considerations such as HTTP caching and idempotence are taken into account. Level 3: "Hypermedia Controls." Building on the previous level, our resource representations now also include links, which indicate what we can do next. At this level, our API becomes practically self-describing; given a single end-point, we should be able to start crawling it, using the links in a representation to lead us to the next actions. When I first started playing with web services around a decade ago, everything was stuck at Level 0 or Level 1 — usually with Level 1 users downgrading to Level 0 because Level 0 offerred consistency and predictability if you chose to use a service type that had a defined envelope format (such as XML-RPC or SOAP). (I even wrote the XML-RPC server implementation for Zend Framework because I got sick of writing one-off parsers/serializers for custom XML web service implementations. When you're implementing many services, predictability is a huge win.) A few years ago, I started seeing a trend towards Level 2. Web developers like the simplicity of using HTTP verbs, as they map very well to CRUD operations — the bread and butter of web development. Couple this concept with JSON, and it becomes trivially simple to both create a web service, as well as consume it. I'd argue that the majority of web developers are quite happy to be at Level 2 — and have no problem staying there. They're productive, and the concepts are easy — both to understand and to implement. Level 3, though, is where it becomes really interesting. The idea that I can examine the represention alone in order to understand what I can do next is very intriguing and empowering. JSON and Hypermedia With XML, hypermedia basically comes for free. Add some elements to your representation, and you're done — and don't forget the link relations! JSON, however, is another story. Where do the links go? There is no single, defined way to represent a hyperlink in JSON. Fortunately, there are some emerging standards. First is use of the "Link" HTTP header. While the page I linked shows only a single link in the header, you can have multiple links separated by commas. GitHub uses this when providing pagination links in their API. Critics will point out that the HTTP headers are not technically part of the representation, however; strict interpetations of REST and RMM indicate that the hypermedia links should be part of the resource representation. Regardless, having the links in the HTTP headers is useful for pre-traversal of a service, as you can perform HEAD requests only to discover possible actions and workflows. Collection+JSON is interesting, as it describes the entire JSON envelope. My one criticism is that it details too much; whenever I see a format that dictates how to describe types, I think of XML-RPC or SOAP, and get a little twitchy. It's definitely worth a look, though. What's captured my attention of late, however, is Hypertext Application Language, or HAL for short. HAL has very few rules, but succinctly describes both how to provide hypermedia in JSON as well as how to represent embedded resources — the two things that most need standardized structure in JSON. It does this while still providing a generic media type, and also describing a mirror image XML format! HAL Media Types HAL defines two generic media types: application/hal+xml and application/hal+json. You will use these as the response Content-Type, as they describe the response representation; the client can simply request application/json, and the response format remains compatible. HAL and Links HAL provides a very simple structure for JSON hypermedia links. First, all resource representations must contain hypermedia links, and all links are provided in a _links object: { "_links": { } } Second, links are properties of this object. The property name is the link relation, and the value is an object containing minimally an "href" property. { "_links": { "self": {"href": "http://example.com/api/status/1234"} } } If a given relation can have multiple links, you provide instead an array of objects: { "_links": { "self": {"href": "http://example.com/api/status/1234"}, "conversation": [ {"href": "http://example.com/api/status/1237"}, {"href": "http://example.com/api/status/1241"} ] } } Individual links can contain other attributes as desired — I've seen people include the relation again so that it's self-contained in the link object, and it's not uncommon to include a title or name. HAL and Resources HAL imposes no structure over resources other than requiring the hypermedia links; even then, you typically do not include the hypermedia links when making a request of the web service; the hypermedia links are included only in the representations returned by the service. So, as an example, you would POST the following: POST /api/status Host: example.com Accept: application/json Content-Type: application/json { "status": "This is my awesome status update!", "user": "mwop" } And from that request, you'd receive the following: 201 Created Location: http://example.com/api/status/1347 Content-Type: application/hal+json { "_links": { "self": {"href": "http://example.com/api/status/1347"} }, "id": "1347", "timestamp": "2013-02-11 23:33:47", "status": "This is my awesome status update!", "user": "mwop" } HAL and Embedded Resources The other important thing that HAL defines is how to embed resources. Why is this important? If the resource references other resources, you will want to be able to link to them so you can perform operations on them, too. Embedded resources are represented inside an _embedded object of the representation, and, as resources, contain their own _links object as well. Each resource you embed is assigned to a property of that object, and if multiple objects of the same type are returned, an array of resources is assigned. In fact, this latter is how you represent collections in HAL. Let's consider a simple example first. In previous code samples, I have a "user" that's a string; let's make that an embedded resource instead. { "_links": { "self": {"href": "http://example.com/api/status/1347"} }, "id": "1347", "timestamp": "2013-02-11 23:33:47", "status": "This is my awesome status update!", "_embedded": { "user": { "_links": { "self": {"href": "http://example.com/api/user/mwop"} } "id": "mwop", "name": "Matthew Weier O'Phinney", "url": "http://mwop.net" } } } I've moved the "user" out of the representation, and into the _embedded object — because this is where you define embedded resources. Note that the "user" is a standard HAL resource itself — containing hypermedia links. Now let's look at a collection: { "_links": { "self": {"href": "http://example.com/api/status"}, "next": {"href": "http://example.com/api/status?page=2"}, "last": {"href": "http://example.com/api/status?page=100"} }, "count": 2973, "per_page": 30, "page": 1, "_embedded": { "status": [ { "_links": { "self": {"href": "http://example.com/api/status/1347"} }, "id": "1347", "timestamp": "2013-02-11 23:33:47", "status": "This is my awesome status update!", "_embedded": { "user": { "_links": { "self": {"href": "http://example.com/api/user/mwop"} } "id": "mwop", "name": "Matthew Weier O'Phinney", "url": "http://mwop.net" } } } /* ... */ ] } } Note that the "status" property is an array; semantically, all resources under this key are of the same type. Also note that the parent resource has some additional link relations — these are related to pagination, and allow a client to determine what the next and last pages are (and, if we were midway into the collection, previous and first pages). Since the collection is also a resource, it has some interesting metadata — how many resources are in the collection, how many we represent per page, and what the current page is. Also note that you can nest resources — simply include an _embedded object inside an embedded resource, with additional resources, as I've done with the "user" resource inside the status resource shown here. It's turtles all the way down. Next Time The title of this post indicates I'll be talking about building RESTful APIs with ZF2 — but so far, I've not said anything about ZF2. I'll get there. But there's another detour to take: reporting errors. Updates Note: I'll update this post with links to the other posts in the series as I publish them. Part 2 Part 3 RESTful APIs with ZF2, Part 1 was originally published 12 February 2013 on https://mwop.net by Matthew Weier O'Phinney.

Responding to Different Content Types in RESTful ZF Apps

2010-03-04T15:28:07-06:00

In previous articles, I've explored building service endpoints and RESTful services with Zend Framework. With RPC-style services, you get to cheat: the protocol dictates the content type (XML-RPC uses XML, JSON-RPC uses JSON, SOAP uses XML, etc.). With REST, however, you have to make choices: what serialization format will you support? Why not support multiple formats? There's no reason you can't re-use your RESTful web service to support multiple formats. Zend Framework and PHP have plenty of tools to assist you in responding to different format requests, so don't limit yourself. With a small amount of work, you can make your controllers format agnostic, and ensure that you respond appropriately to different requests. Content-Type Detection The first problem to solve is going to be how to retrieve passed parameters. When using XML or JSON as your serialization format, you aren't getting your standard POST variables — you're getting a raw post instead, and you'll need to deserialize the payload. In fact, if you're getting a PUT request, you also have some work to do, as PHP doesn't do anything with PUT requests. I do this via an action helper. The basic algorithm is: Do we have a raw body in the request? If not, nothing more need be done. Determine the Content-Type passed in the request headers, and decode appropriately: If it was JSON, pass the raw request body to json_decode or Zend_Json::decode. If it was XML, I pass the raw request body to the Zend_Config_XML constructor, and then serialize to an arrya using the toArray() method. Yes, it's a hack, but it's effective. Otherwise, I assume I've got a regular PUT-style request, and I pass the data to parse_str(). I keep the values within the action helper, and then retrieve them on demand within my action controller. The helper looks like the following: class Scrummer_Controller_Helper_Params extends Zend_Controller_Action_Helper_Abstract { /** * @var array Parameters detected in raw content body */ protected $_bodyParams = array(); /** * Do detection of content type, and retrieve parameters from raw body if * present * * @return void */ public function init() { $request = $this->getRequest(); $contentType = $request->getHeader('Content-Type'); $rawBody = $request->getRawBody(); if (!$rawBody) { return; } switch (true) { case (strstr($contentType, 'application/json')): $this->setBodyParams(Zend_Json::decode($rawBody)); break; case (strstr($contentType, 'application/xml')): $config = new Zend_Config_Xml($rawBody); $this->setBodyParams($config->toArray()); break; default: if ($request->isPut()) { parse_str($rawBody, $params); $this->setBodyParams($params); } break; } } /** * Set body params * * @param array $params * @return Scrummer_Controller_Action */ public function setBodyParams(array $params) { $this->_bodyParams = $params; return $this; } /** * Retrieve body parameters * * @return array */ public function getBodyParams() { return $this->_bodyParams; } /** * Get body parameter * * @param string $name * @return mixed */ public function getBodyParam($name) { if ($this->hasBodyParam($name)) { return $this->_bodyParams[$name]; } return null; } /** * Is the given body parameter set? * * @param string $name * @return bool */ public function hasBodyParam($name) { if (isset($this->_bodyParams[$name])) { return true; } return false; } /** * Do we have any body parameters? * * @return bool */ public function hasBodyParams() { if (!empty($this->_bodyParams)) { return true; } return false; } /** * Get submit parameters * * @return array */ public function getSubmitParams() { if ($this->hasBodyParams()) { return $this->getBodyParams(); } return $this->getRequest()->getPost(); } public function direct() { return $this->getSubmitParams(); } } This helper is intended to be run on each request, so I register it in my bootstrap: class Bootstrap extends Zend_Application_Bootstrap_Bootstrap { // ... protected function _initActionHelpers() { // ... $params = new Scrummer_Controller_Helper_Params(); Zend_Controller_Action_HelperBroker::addHelper($params); // ... } // ... } Within your action controller, all you need to do is call the helper: $data = $this->params(); In a RESTful controller, you'll only need to use this with your postAction and putAction. The beauty is that your controller can remain ignorant of the Content-Type — you write the same logic to retrieve your parameters regardless. Responding to the client: Context Switching So, the first half of the problem is taken care of: how to handle the request. The second half is responding appropriately. Zend Framework has some built in tooling to help with this. The ContextSwitch and AjaxContext action helpers look for a particular parameter — "format" by default — and, if detected, will render an alternate view script named after the context. As an example, if an "XML" context is detected, it will render /.xml.phtml — note the .xml segment of the script name. Both helpers work in the same basic way (the latter, AjaxContext, will only activate if the request is determined to originate from an XMLHttpRequest): you define which actions in the controller are context sensitive, and then if the context is detected, a new view script will be used. So, the first trick is ensuring that the context is passed. As mentioned before, the helpers look for a "format" parameter in the request object. You can pass this using a query parameter — ?format=xml — but I find that ugly. There's an HTTP header defined for this purpose already: "Accept". Detecting the header and injecting the context into the request is absurdly simple, and can be done in a dispatchLoopStartup plugin: class Scrummer_Controller_Plugin_AcceptHandler extends Zend_Controller_Plugin_Abstract { public function dispatchLoopStartup(Zend_Controller_Request_Abstract $request) { if (!$request instanceof Zend_Controller_Request_Http) { return; } $header = $request->getHeader('Accept'); switch (true) { case (strstr($header, 'application/json')): $request->setParam('format', 'json'); break; case (strstr($header, 'application/xml') && (!strstr($header, 'html'))): $request->setParam('format', 'xml'); break; default: break; } } } The above can be registered in your application configuration: resources.frontController.plugins[] = "Scrummer_Controller_Plugin_AcceptHandler" I like my RESTful controllers to automatically expose their methods as context-aware. To make this happen, I defined a marker interface, Scrummer_Rest_Controller, and created an action helper that checks if the current controller implements it; if it does, I then automatically add contexts for the RESTful actions. class Scrummer_Controller_Helper_RestContexts extends Zend_Controller_Action_Helper_Abstract { protected $_contexts = array( 'xml', 'json', ); public function preDispatch() { $controller = $this->getActionController(); if (!$controller instanceof Scrummer_Rest_Controller) { return; } $this->_initContexts(); // Set a Vary response header based on the Accept header $this->getResponse()->setHeader('Vary', 'Accept'); } protected function _initContexts() { $cs = $this->getActionController()->contextSwitch; $cs->setAutoJsonSerialization(false); foreach ($this->_contexts as $context) { foreach (array('index', 'post', 'get', 'put', 'delete') as $action) { $cs->addActionContext($action, $context); } } $cs->initContext(); } } Register this via the bootstrap as well: class Bootstrap extends Zend_Application_Bootstrap_Bootstrap { // ... protected function _initActionHelpers() { // ... $params = new Scrummer_Controller_Helper_Params(); Zend_Controller_Action_HelperBroker::addHelper($params); $contexts = new Scrummer_Controller_Helper_RestContexts(); Zend_Controller_Action_HelperBroker::addHelper($contexts); // ... } // ... } There are two things to note about this helper. First, you'll see that I specify a "Vary" header. This is to ensure that if the client chooses to cache responses, it will cache separate responses based on the value sent in the "Accept" header. Second, note that I turn off automatic JSON serialization in the ContextSwitch helper. I do this so that I can keep my controller context agnostic; this will require additional view scripts, but the ability to keep my controller logic simple will be worth it. More on that in a moment. We now have the infrastructure in place to respond to different contexts based on the "Accept" header, and can retrieve parameters appropriately based on the "Content-Type" provided us. Now comes the actual response. Responding to the client: Views Recall that ContextSwitch will attach an additional prefix to the specified view script — /.phtml will become /.xml.phtml or /.json.phtml. Basically, for each context we will respond to, we have an additional view script per action. views/ |-- scripts/ | `-- foo/ | |-- delete.phtml | |-- delete.json.phtml | |-- delete.xml.phtml | |-- get.phtml | |-- get.json.phtml | |-- get.xml.phtml | |-- index.phtml | |-- index.json.phtml | |-- index.xml.phtml | |-- post.phtml | |-- post.json.phtml | |-- post.xml.phtml | |-- put.phtml | |-- put.json.phtml | `-- put.xml.phtml This may seem like overkill, but consider the following representative method from my controller: public function postAction() { $data = $this->params(); $service = $this->getService(); $result = $service->add($data); if (!$result) { $this->view->form = $service->getBacklogForm(); return; } $this->view->success = true; $this->view->backlog = $result; } You don't see anything in there about headers, redirects, or XHR requests. Just slinging data to services and views. Real simple. The view scripts then take care of the appropriate display logic. Let's look at two view scripts for the above action, one for plain old HTML, the other for a JSON response: // backlog/post.phtml ?> if ($this->success): $this->response->setRedirect($this->url(array( 'controller' => 'backlog', 'id' => $this->backlog->id, ), 'rest', true)); else: ?>

Create new backlog

$this->form->setAction($this->url()) ->setMethod('post'); echo $this->form; endif ?> // backlog/post.json.phtml ?> if ($this->success) { $url = $this->url(array( 'controller' => 'backlog', 'id' => $this->backlog->id, ), 'rest', true); $this->response->setHeader('Location', $url) ->setHttpResponseCode(201); echo $this->json($this->backlog->toArray()); return; } $form = $this->form; $form->setAction($this->url()) ->setMethod('post'); echo $this->jsonFormErrors($form); A few things to note: I inject my response object into the view. I feel HTTP headers are part of the view, and thus I deal with them there. That also serves the purpose of keeping my controllers thin and agnostic. Additionally, you'll note that I use different response codes for HTML versus JSON — this allows my JSON-REST support to be RESTful, by returning a 201 status code indicating the resource was created; I also return a JSON representation of the object. Finally, you'll note that I have a special view helper for creating JSON representations of validation errors. Closing points This post is far from exhaustive, and I expect it will likely raise at least as many questions as it tries to answer. My main point in this article is to get you, the reader and developer, thinking creatively about how to expose RESTful web services. Hopefully, you're taking the following away: Architect in such a way as to minimize the code in your controllers; keep that code as agnostic as possible in regards to where input comes from and what type of response is required. Use front controller plugins and action helpers to create scaffolding for your services; these are incredibly flexible and re-usable, and help make point 1 that much easier. Offload as much as possible to your views. This will allow you to isolate logic specific to given formats. What are you waiting for? Don't you have an API to expose? Responding to Different Content Types in RESTful ZF Apps was originally published 4 March 2010 on https://mwop.net by Matthew Weier O'Phinney.

Building RESTful Services with Zend Framework

2009-11-09T09:00:00-06:00

As a followup to my previous post, I now turn to RESTful web services. I originally encountered the term when attending php|tropics in 2005, where George Schlossnaggle likened it to simple GET and POST requests. Since then, the architectural style — and developer understanding of the architectural style — has improved a bit, and a more solid definition can be made. At its heart, REST simply dictates that a given resource have a unique address, and that you interact with that resource using HTTP verbs. The standard verbs utilized are: GET: retrieve a list of resources, or, if an identifier is present, view a single resource POST: create a new resource with the data provided in the POST PUT: update an existing resource as specified by an identifier, using the PUT data DELETE: delete an existing resource as specified by an identifier The standard URL structure used is as follows: /resource - GET (list) and POST operations /resource/{identifier} - GET (view), PUT, and DELETE operations What the REST paradigm provides you is a simple, standard way to structure your CRUD (Create-Read-Update-Delete) applications. Due to the large number of REST clients available, it also means that if you follow the rules, you get a ton of interoperability with those clients. As of Zend Framework 1.9.0, it's trivially easy to create RESTful routes for your MVC application, as well as to handle the various REST actions via action controllers. Zend_Rest_Route allows you to define RESTful controllers at several levels: You can make it the default route, meaning that unless you have additional routes, all controllers will be considered REST controllers. You can specify modules that contain RESTful controllers. You can specify specific controllers per module that are RESTful As examples: $front = Zend_Controller_Front::getInstance(); $router = $front->getRouter(); // Specifying all controllers as RESTful: $restRoute = new Zend_Rest_Route($front); $router->addRoute('default', $restRoute); // Specifying the "api" module only as RESTful: $restRoute = new Zend_Rest_Route($front, array(), array( 'api', )); $router->addRoute('rest', $restRoute); // Specifying the "api" module as RESTful, and the "task" controller of the // "backlog" module as RESTful: $restRoute = new Zend_Rest_Route($front, array(), array( 'api', 'backlog' => array('task'), )); $router->addRoute('rest', $restRoute); To define a RESTful action controller, you can either extend Zend_Rest_Controller, or simply define the following methods in a standard controller extending Zend_Controller_Action (you'll need to define them regardless): // Or extend Zend_Rest_Controller class RestController extends Zend_Controller_Action { // Handle GET and return a list of resources public function indexAction() {} // Handle GET and return a specific resource item public function getAction() {} // Handle POST requests to create a new resource item public function postAction() {} // Handle PUT requests to update a specific resource item public function putAction() {} // Handle DELETE requests to delete a specific item public function deleteAction() {} } For those methods that operate on individual resources (getAction(), putAction(), and deleteAction()), you can test for the identifier using the following: if (!$id = $this->_getParam('id', false)) { // report error, redirect, etc. } Responding is an art Many developers are either unaware of or ignore the part of the specification that dictates what the response should look like. For instance, in classic REST, after performing a POST to create a new item, you should do the following: Set the HTTP response code to 201, indicating "Created" Set the Location header to point to the canonical URI for the newly created item: /team/31 Provide a representation of the newly created item Note that there's no redirect, which flies in the face of standard web development (where GET-POST-Redirect is the typical format). This is a common "gotcha" moment. Similarly, with PUT requests, you simply indicate an HTTP 200 status when successful, and show a representation of the updated item. DELETE requests should return an HTTP 204 status (indicating success - no content), with no body content. Note: when building RESTful HTML applications, you may want to still do GET-POST-Redirect to prevent caching issues. The above applies to RESTful web services, which typically use XML or JSON for transactions, and have smart clients for interacting with the service. I'll be writing another article soon showing some tips and tricks for interacting with HTTP headers, both from the request and for the response, as it's a subject lengthy enough for a post of its own. In the meantime, start playing with Zend_Rest_Route and standardizing on it for your CRUD operations! Building RESTful Services with Zend Framework was originally published 9 November 2009 on https://mwop.net by Matthew Weier O'Phinney.