3.0 REST API Updates

Owned by Matthew Marum
Last updated: Feb 14, 2012 by JamesSVersion comment
12 min read

Versioning

See Protocol and Data Model Versioning

Supporting Multiple Groups

There is a need for requests to be able to support requesting and aggregating data from multiple groups.  For example, being able to request Activities from @friends and @all in same request.
https://groups.google.com/forum/?fromgroups#!topic/opensocial-and-gadgets-spec/QWCHPcWfuo4|https://groups.google.com/forum/?fromgroups#!topic/opensocial-and-gadgets-spec/QWCHPcWfuo4

Updating the Core APIs

Proposal for updating the core APIs in 3.0 [James Snell]

The existing OpenSocial specification is at a point in it's lifecycle where theoretical standardization work has transitioned into real world applications. During that process, the OS specification itself has evolved fairly organically and currently carries with it some rather nice parts, but also some rather nasty warts that all of us implementing real world applications have run up against.

Most critically, the OpenSocial API, as it exists currently, is poorly documented, grossly underspecified, and way too complicated to both implement and use in real world applications. Already many of us who work on the spec and implement support for it in our products have had to resort to ugly undocumented hacks to achieve even some of the most basic functionality. It says a lot about a specification if it's own authors cannot take that specification and write a fully compliant, functional real world application addressing real world business needs without resorting to hacks and undocumented tricks. In order to be successful moving forward, OS *MUST* focus on simplification and ease of use, not only with the javascript programming model and gadgets, but with the on-the-wire protocol layer as well.

To address this, knowing full well that significant discussion will be necessary, I decided to take a step back and reevaluate everything, questioning all assumptions that have been made up to this point. No part of the existing protocol was help sacred as being untouchable and unchangeable. The attached document is the initial, *first draft* result of my reevaluation and redesign.

There are several important aspects to my proposal:

  1. The RPC Protocol should be moved to a separate document and eventually deprecated. As it exists today, the RPC Protocol is a poorly designed and poorly executed hack in the same vein as XML-RPC and SOAP circa 2001. Other than batch requests, it brings absolutely no significant value to the table while increasing the overall complexity of the specification. It violates fundamental REST design principles, it does not properly account for standard mechanisms such as Authentication, Caching, Conditional Requests, etc, it supports a grossly-underspecified batch request model, and, when combined with the REST protocol, makes for an extremely confusing and difficult to read specification (http://opensocial-resources.googlecode.com/svn/spec/trunk/Core-API-Server.xml)
  2. The proposal introduces a new model for requesting multiple resources in a single request that adheres to fundamental REST principles, properly addresses idempotency, authentication, caching and conditional request issues, while keeping things as simple to understand as possible.
  3. The proposal introduces support for new mechanisms that have emerged since OpenSocial was conceived, including the new HTTP PATCH method and the HTTP Link and Prefer headers. These can be leveraged in a variety of ways to simplify and generalize the overall API model and make things more compact and easier to understand.
  4. The proposal incorporates the protocol versioning discussion that I posted earlier today. I'm fully aware that this aspect will need to be evaluated in detail separately, but I wanted to show how all the pieces come together into a coherent whole.
  5. Many of the additional details, such as how all this plays out in the javascript API, how many of the other batch request use cases can be handled, etc will be dealt with separately.
  6. The proposal drops OAuth 1.0 support completely in favor of OAuth 2.0
  7. This was written with an eye towards significantly cleaning up the discussion and presentation of the core protocol model. It's easier to read and understand compared to the existing protocol specification and, ultimately will be easier to implement.

The Proposal

An OpenSocial 3.0 Service API consists of collection of Web-accessible resources that can be accessed and modified using the basic set of HTTP request methods as defined by [TODO: HTTP Reference].

A resource made available via a Service API may either represent an individual object (e.g. a person, a document, an event) or a collection of objects (e.g. a friends list, a folder containing multiple files, a stream of events).  Every resource is identified by a distinct URI to which the various HTTP methods are to be sent.

While the exact semantics and operation of each method may vary by individual resource type,

  • the HTTP GET method is used commonly to retrieve a representation of the current state of a given resource,
  • PUT and PATCH are used to modify the current state,
  • DELETE is used to delete the resource,
  • and POST is used to either create new resources or to perform other types of operations that do not fit within the scope of the other core HTTP methods.

For example, to retrieve a user's current profile using the OpenSocial 3.0 Profile Service, a client application would issue a GET request:

GET /api/people/@me/@self HTTP/1.1
Host: api.example.org
Authorization: Bearer hh5s93j4hdidpola

When a client application utilizing a Service API is required to communicate with the server via an intermediary that restricts the use of certain standard and extension HTTP Methods (e.g.  PUT, DELETE, PATCH), the client SHOULD utilize the X-HTTP-Method-Override HTTP Request Header mechanism in a POST request.  This type of request shall be referred to as a "POST Override Request".

For example, the HTTP PATCH request method is still relatively new and not yet fully supported by deployed HTTP infrastructure.  It is therefore possible that PATCH requests could be inadvertently blocked by intervening intermediaries. To address such cases, the PATCH request may be modified by sending a POST request that includes the X-HTTP-Method-Override HTTP header:

POST /api/people/@me/@self HTTP/1.1
Host: api.example.org
Authorization: Bearer hh5s93j4hdidpola
Content-Type: application/json
X-HTTP-Method-Override: PATCH

{...}

Server's implementing the OpenSocial 3.0 Service API SHOULD respond to POST Override Requests as if the method specified by the X-HTTP-Method-Override was the actual operation used by the request, as opposed to POST.

Responses to requests sent to Service API resources will specify an appropriate HTTP Status Code indicating the status of the response. Specifically,

  • A server MUST return a 400 Bad Request status if the request specifies an invalidly formatted request URI, an invalidly formatted HTTP request header or includes an invalidly formatted payload.
  • A server MUST return a 401 Unauthorized status when receiving a request for a protected resource and the request is either missing appropriate authorization credentials as specified in Section 4, or the authorization credentials are present but not authorized to perform the requested action on the identified resource.
  • A server MUST return a 404 Not Found if it is unable to locate the resource identified by the request URI.
  • A server MUST return a 405 Method Not Allowed if the method specified by the request is not supported for the resource identified by the request URI.

Common URI Structure

All Service API resources share a common, basic URI Structure that MAY be extended on a case-by-case basis.  This common structure ensures that all interactions with the various resources is as consistent as possible across all available Service APIs while still allowing individual service-specific behaviors and characteristics to be used.

By convention, the URI Structure is described using the URI Template syntax defined by [TODO: URI Template Reference].  Every URI Template consists of a number of specific replacement expressions containing one or more variables that, when values are provided for each variable, may be used to expand the Template into a specific request URI.

The Common URI Template (Note: line breaks have been added for readability only):

https://{+host}{/rootpath*}{/resource}{/aspect}{/ext*}{?parameters*}
  • {+host} : Represents the DNS Host Name and port of the server hosting the Service API. For instance, "api.example.org:80".
  • {rootpath*} : Represents the root path of the Service API. The value of the rootpath variable is specified as an array of URI path segments, e.g. ['api','people'] that when expanded by the URI Template generates a valid URI Path Segment containing each of the specified values, e.g. "/api/people".
  • {/resource} : Represents the unique identity of the specific Resource to which the request is being addressed. Requests may specify one or more resource ID's within a single Request URI. For instance, specifying the value "@me" for the resource variable would expand to "/@me" when applied to the URI Template. Multiple values, e.g. ['@me','123456','ABCDEF'], would expand to "/@me,123456,ABCDEF". Working with multiple individual resources within a single Request URI is covered in detail below.
  • {/aspect} : Represents a specific aspect of the resource that is being requested. Every resource type defines it's own collection of available aspects and determines for itself whether to limit request URIs to a single aspect or to allow multiple aspects to be specified in the URI. For instance, a user profile resource may define a "@friends" aspect that identifies a collection of other profiles related to a given resource, and an "@activity" aspect that identifies the collection of activities associated with the resource. If the Service API requires that only a single aspect is allowed within a request URI, then specifying the value "@friends" for the aspect variable would expand to "/@friends" when applied to the URI Template. However, if multiple aspects are allowed, e.g. ['@friends','@activity'], the URI template would expand to: "/ (friends,)activity".
  • {/ext*} : Represents additional aspect-specific path information, specified as an array of individual segments, e.g. a value of ['foo', 'bar'] for ext would expand to "/foo/bar".
  • {?parameters*} : Represents the collection of query parameters associated with the request, specified as a collection of name=value pairs. All Service APIs MUST support a common standard set of query parameters, defined in detailed below and MAY introduce additional resource specific parameters.

In only certain specific cases, the URI Structure of individual    Service APIs MAY diverge from this common structure.  Such deviations will be described in detail on a case-by-case basis within the Service APIs specification.

To illustrate how the common URI structure operates, consider the following example.  For each of the URI Template variables, we provide the following values:

host       "api.example.org"
rootpath   ["api","people"]
resource   "@me"
aspect     "@friends"
parameters [
  {"fields"="name,email"},
  {"count"="10"}]

 When applied to the common URI Template, the Request URI Generated becomes: "https://api.example.org/api/people/@me/@friends?fields=name,email&count=10".

Standard Query Parameters

All Service API resources MUST support a common set of URI Query String parameters that can be used to modify the processing of the request and the information returned to the client.

  • fields : The fields parameter allows a client application retrieving information about a resource to specify the individual pieces of information it is interested in receiving as a simple, comma separated list.  For instance, "fields=name,email". The server MAY choose to return a representation of the requested information containing only the requested fields along with any additional information the server determines is required.  If the resource identified by the URI represents a collection of individual objects, the fields parameter identifies the requested properties of those individual objects, rather than the properties of the collection.
  • format  The default data format used to represent all Service API resources is the JSON-based format defined in [TODO: Data Format Spec].  Individual Service API's MAY support additional data formats on a case-by-case basis.  Client applications SHOULD use normal, well-defined HTTP Content Negotiation mechanisms such as the Accept request header (e.g.  "Accept: application/atom+xml") to identify the specific preferred data format.  However, not all client applications are capable of directly accessing HTTP Headers.  For such clients, the "format" parameter MAY be used as an alternative.  The value of the format parameter is a single MIME Media Type, e.g. "format=application/atom+xml".

When the Request URI represents a collection of objects, additional common parameters are available:

TODO: Complete these

  • count
  • filterBy
  • filterOp
  • filterValue
  • sortOrder
  • startIndex

Conditional Requests

All requests to Service API resources support the use of Conditional Request mechanisms as defined in [TODO: HTTP Specification].

When returning the current representation of any resource, whether that representation is for a single object or a collection of objects, the HTTP Response MUST include either a strong Entity Tag as defined in [TODO: HTTP Spec], Section 2.3, a Last-Modified Timestamp as specified in Section 2.2, or both.

For example, the response to a request retrieving the current representation of a person's profile might include both an Entity Tag and a Last-Modified timestamp:

HTTP/1.1 200 OK
Content-Type: application/json
ETag: "ab12cd34ef56"
Last-Modified: Tue, 15 Nov 1994 12:45:26 GMT


{...}

Once a client has the Entity Tag or Last-Modified timestamp for a given resource representation, the client SHOULD use those in all subsequent requests both to optimize communication and to ensure that multiple clients do not inadvertently overwrite each other's data.

For instance, given the Last-Modified Timestamp included in a response to a request to receive a user's profile, a client can request that the server return an updated view of the profile if, and only if the state of the profile has changed since the specified time:

GET /api/people/@me/@self HTTP/1.1
If-Modified-Since: Tue, 15 Nov 1994 12:45:26 GMT

If the profile has not been modified since the date specified, the server would respond with a "304 Not Modified" status.

Given the Entity Tag of a resource representation, a client wishing to modify the resource can request that changes only be applied if the resource has not been subsequently modified since the original representation and Entity Tag were created:

PUT /api/people/@me/@self HTTP/1.1
If-Match: "ab12cd34ef56"
Content-Type: application/json

{...}

If the server is unable to determine that the specified Entity Tag applies to the current state of the resource, it will abort the change request and respond to the request using a "412 Precondition Failed" status.

Note that the entity tag specified in the response is generally specific to the payload included in the response.  If a Service API supports returning multiple representations of a single resource, such as offering multiple data format options or modified views of the resource tailored to the authentication credentials included in the request, the Entity Tag may vary for each specific response, regardless of whether the actual state of the resource on the server has changed. Therefore, for any single resource, multiple Entity Tags MAY potentially represent the current state of the resource.

At a minimum, OpenSocial 3.0 Service API implementations MUST support the If-Match, If-None-Match, If-Modified-Since and If-Unmodified-Since HTTP request headers for Conditional Requests.  All requests that result in the potential modification of the current state of a resource MUST include at least one of these request headers.

Full vs. Partial Modification

The current state of a resource may be modified in part or in full using either the PATCH or PUT HTTP methods, respectively.

Given a URI that represents an resource, the current state of that resource can be modified in full by sending an HTTP PUT request to the URI.  The payload of the PUT request is considered to be a replacement for the identified resource, although the server is free to determine exactly how the resource is to be modified.

For example, suppose we wish to modify an existing profile that contains nothing more than a name and en email address.  First, we need to GET the existing state of the profile so we know what we are working with and can get the Entity Tag necessary for the PUT operation:

GET /api/people/@me/@self HTTP/1.1

The server responds with the profile:

HTTP/1.1 200 OK
Content-Type: application/json
ETag: "ab12cd34ef56"

{"name": {"displayName":"J. Doe"},
 "emails":["john.doe@example.org"]}

We want to completely modify this profile such that the name is expanded from "J. Doe" to "John Doe" and the email address is dropped completely.  We can do so by sending the following PUT request:

PUT /api/people/@me/@self HTTP/1.1
If-Match: "ab12cd34ef56"
Content-Type: application/json

{"name": {"displayName":"John Doe"}}

 Assuming the change is successful, the server would respond with an appropriate 2xx status, and MAY include the updated representation of the resource:

HTTP/1.1 200 OK
Content-Type: application/json
ETag: "ab12cd34ef57"

{"name": {"displayName":"John Doe"}}

Alternatively, we can use a PATCH request to perform a partial modification of the resource.  For instance, suppose we need to add an email address to the previously modified profile:

PATCH /api/people/@me/@self HTTP/1.1
If-Match: "ab12cd34ef57"
Content-Type: application/json-patch

{"add":"emails","value":"j.doe@example.com"}

Assuming the change is successful, the server would respond with an appropriate 2xx status, and MAY include the updated representation of the resource:

HTTP/1.1 200 OK
Content-Type: application/json
ETag: "ab12cd34ef57"

{"name":{"displayName":"John Doe"},"emails":["j.doe@example.com"]}

Support for the PATCH method to perform partial modifications of resource is optional.

Note that, for illustrative purposes only, this example uses the JSON Patch format defined by [TODO: JSON Patch Reference]. Implementations are free, however, to use any format they choose as the payload of PATCH operations.

OpenSocial 3.0 server implementations that support PATCH MUST advertise that support using the Accept-Patch HTTP Response header as defined by [TODO: PATCH METHOD REF], Section 3.1.

For example,

HTTP/1.1 200 OK
Allow: GET, PUT, POST, DELETE, HEAD, PATCH, OPTIONS
Accept-Patch: application/json-patch

Working with Multiple Resources and Aspects in a single request

As indicated by the definition of the common URI Structure for Service APIs, it is possible for a single GET, HEAD or OPTIONS request to operate on multiple resources and aspects. There are, however, a number of important caveats and considerations that need to be taken into account when working with multiple resources within a single operation.

To illustrate the how support for multiple resources works, consider a profile service that contains multiple profiles representing different users.

A client application can access each of these profiles by issuing a HTTP GET request to their respective URIs, requiring three distinct HTTP requests:

GET /api/people/user1/@self HTTP/1.1
GET /api/people/user2/@self HTTP/1.1
GET /api/people/user3/@self HTTP/1.1

Each of these individual requests would return a single profile representation:

HTTP/1.1 200 OK
Content-Type: application/json
ETag: "abcdef123456"

{
  "id":"acct:user1@example.org"
  "name": {"displayName":"John Doe"}
}

Using the Multiple Resource capability of the OpenSocial 3.0 Service API, however, these three distinct HTTP Requests can be rolled up into a single request:

GET /api/people/user1,user2,user3/@self HTTP/1.1

This combined request would return each of the requested profiles within a single response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "items":[
    {
      "id":"acct:user1@example.org"
      "name": {"displayName":"John Doe"},
    },
    {
      "id":"acct:user2@example.org"
      "name": {"displayName":"Jane Doe"},
    },
    {
      "id":"acct:user3@example.org"
      "name": {"displayName":"Sally Jones"},
    }
  ]
}

Note that while this particular example is utilizing the JSON Collection representation defined by [TODO: Data Spec Ref], the specific representation returned by a multiple resource request is specific to each individual Service API definition and to each type of resource.

If one of the requested resources does not exist, the server MAY choose to either reject the entire request with a 404 Not Found or simply omit the missing item from the response optionally with some format specific indication that the specific item could not be found.

For instance, if the profile identified by "user3" in the example above does not exist, the server could choose to return a response that includes only the two found profiles:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "items":[
    {
      "id":"acct:user1@example.org"
      "name": {"displayName":"John Doe"}},
    {
      "id":"acct:user2@example.org"
      "name": {"displayName":"Jane Doe"}}
    }
  ]
}

Query String parameters included in the request URI apply to each of the individually requested resources just as if separate requests were sent to each resource.

Authentication credentials included in the request MUST be valid and authorized for each of the individually requested resources or the entire request MUST be rejected with a 401 Unauthorized response.

Modification of multiple resources with a single operation is unsupported.  PUT, DELETE, PATCH and POST methods sent to a request URI specifying multiple resources MUST fail with a response of 405 Method Not Supported.

Implementations SHOULD place strict limits on the number of distinct resources that may be included within a single request URI.

Conditional Requests

When a request for multiple resources contains either if If-Match or If-None-Match HTTP Request Headers, the header MUST specify the known current Entity Tag for each of the requested resources.

For example,

GET /api/people/user1,user2,user3/@self HTTP/1.1
If-Match: "user1-12345","user2-54321","user3-1234"

Upon receiving such a request, the server MUST determine if any of the supplied Entity Tags match the current state of each of the requested resources and must only execute the request operation if all of the specified Entity Tags match.

When a request for multiple resources contains either a If-Modified-Since or If-Unmodified-Since HTTP request header, the Server MUST first determine if all requested resources adhere to the condition. For instance, if even just one of the requested resources has been modified since the date and time specified by an If-Unmodified-Since request header, the request MUST NOT be processed.

For example,

GET /api/people/user1,user2,user3/@self HTTP/1.1
If-Unmodified-Since: Tue, 15 Nov 1994 12:45:26 GMT

Multiple Aspect Requests

In addition to allowing multiple resources to be specified within the request URI, multiple individual aspects may also be specified.

For instance, expanding on the previous profile example, suppose we wish to retrieve three separate users profiles along with each of their lists of friends:

GET /api/people/user1,user2,user3/<at:var at:name="self," />friends HTTP/1.1

Whether such requests are supported by a specific Service API, as well as the specific response serialization format for such requests is specific to each Service API definition.

Other Related Issues