Core-Gadget - Data Pipelining
This is a DRAFT
<section title="Data Pipelining" anchor="DataPipelining">
<t>Data Pipelining is a declarative mechanism for defining the data a
gadget requires from the container. XML Elements are used to associate
variable keys with specific kinds of data from a variety of different
sources. For instance, the <os:DataRequest> element can be used
to access social data provided by the OpenSocial container, while the
<os:HttpRequest> element can provide access to content from any
HTTP endpoint.</t>
<t>All Data Pipeline elements have a 'key' attribute whose value is
used to uniquely identify the data. Such keys can be used within
Expression Language statements used for Variable Substitution or
with JavaScript API calls. Key names are always case-sensitive.</t>
<figure><preamble>The following illustrates a basic example of
Data Pipelining in a partial gadget specification:</preamble>
<artwork xml:space="preserve"><![CDATA[
<Module xmlns:os="http://ns.opensocial.org/2008/markup">
...
<Data>
<os:DataRequest key="viewer" method="people.get" userId="@viewer" fields="name,birthday" />
<os:HttpRequest key="remote" href="http://example.com/api" />
</Data>
<Content type="html">
The viewer name is ${viewer.name}.
The remote content is "${remote.result.content}".
</Content>
</Module>
]]></artwork></figure>
<t>The Data Pipeline elements can appear:
<list style="symbols">
<t>As children of the <Data> element,</t>
<t>As direct children of the <Content> element when the Proxied
Content model is used, and </t>
<t>Within specially formatted HTML <script> tags included
within HTML Content.</t>
</list>
</t>
<t>The placement of Data Pipeline elements will determine where the
data associatd with the key specified will be made available to the
container while the gadget is being rendered. The considerations
and requirements for each of the options listed above are discussed
in detail in the relevant sections that follow.</t>
<section title="The DataContext" anchor="DataContext">
<t>For every gadget instance, a <spanx style="verb">DataContext</spanx>
object is created that maintains all of the data associated with the
gadget instance's Data Pipeline elements. The information contained
within the <spanx style="verb">DataContext</spanx> is available for
use with Variable Substitution via Expression Language statements
or via the <spanx style="verb">opensocial.data</spanx> APIs; and while
the data contained within the <spanx style="verb">DataContext</spanx>
is generally populated through processing of the Data Pipeline
elements, the <spanx style="verb">opensocial.data</spanx> APIs can
be used to modify values or add additional keys.</t>
</section>
<section title="Data Pipeline Elements and the Data Block">
<t>By default, when Data Pipeline elements appear as children of the
<Data> element, the container will process the elements and
register the resulting data with the <spanx style="verb">DataContext</spanx>
for all views.</t>
<t>Individual Data Pipeline elements, however, MAY specify a "view"
attribute whose value is a comma separated list of specific views
for which the element is associated. Such elements will only be
processed and added to the <spanx style="verb">DataContext</spanx>
when the identified views are being rendered.</t>
<figure><preamble>In the following example, the <os:HttpRequest>
element using key "foo" is registered globally and available within
all views while the <os:HttpRequest> element using key "bar"
is only processed and available when the "example" view is being
rendered:</preamble><artwork><![CDATA[
<Module>
...
<Data>
<os:HttpRequest key="foo" href="http://example.com/api" />
<os:HttpRequest key="bar" href="http://example.com/api" view="example"/>
</Data>
<Content type="html">
...
</Content>
<Content type="html" view="example">
...
</Content>
</Module>
]]></artwork></figure>
<t>Multiple Data Pipeline elements MUST NOT share the same value for
the key attribute unless each is specifically targeted at a different
view. In the following example, the key "foo" would resolve to the
value "abc" when the default view is rendered and to "123" when the
"example" view is rendered:</t>
<figure><artwork><![CDATA[
<Module>
...
<Data>
<os:Var key="foo" value="abc" />
<os:Var key="foo" value="abc" view="example" />
</Data>
<Content type="html">
...
</Content>
<Content type="html" view="example">
...
</Content>
</Module>
]]></artwork></figure>
<t>Note that currently, this specification does not specify
exactly when the container processes the collection of Data Pipeline
elements, nor are containers required to process data elements in
any particular order. Containers that process Data Pipeline elements
to populate the <spanx style="verb">DataContext</spanx> will likely
do so regardless of whether the referenced key is ever actually
used within an expression statement or API call.</t>
</section>
<section title="Data Pipeline Elements within HTML Content">
<t>When using HTML Content, Data Pipeline elements MAY appear
within the HTML inside a special <script> tag.</t>
<figure><preamble>For instance,</preamble><artwork>
<Module>
<ModulePrefs>
<Require feature="opensocial-data" />
</ModulePrefs>
<Content type="html"><![CDATA[
<script xmlns:os="http://ns.opensocial.org/2008/markup" type="text/os-data">
<os:HttpRequest key="foo" href="http://example.com/api" />
<os:Var key="bar" value="abc123"/>
</script>
]]></Content>
</Module>
</artwork></figure>
<t>As illustrated, when Data Pipeline elements are embedded within
HTML Content, the gadget specification MUST require the "opensocial-data"
feature using the <Require> element within the <ModulePrefs>.</t>
<t>The Data Pipeline elements themselves are contained within an
HTML script tag whose type attribute specifies "text/os-data". Also
note that the Data Pipeline XML namespace is declared for the "os:"
prefix. This is required in order for the embedded Data Pipeline
elements to processed properly while the content is being rendered.</t>
<t>Data Pipeline elements defined in this manner are scoped specifically
to the <Content> element in which they are embedded, meaning that
the data each provides to the <spanx style="verb">DataContext</spanx>
will be available for expression statements and API calls that are
also in the same <Content> element.</t>
<t>If an embedded Data Pipeline element specifies the same key as a
Data Pipeline element contained within the <Data> element,
the value provided by the embedded elements takes precedence. In the
following example, the variable "foo" will resolve to "123" when
the default view is rendered:</t>
<figure><artwork>
<Module>
<ModulePrefs>
<Require feature="opensocial-data" />
</ModulePrefs>
<Data>
<os:Var key="foo" value="abc"/>
</Data>
<Content type="html"><![CDATA[
<script xmlns:os="http://ns.opensocial.org/2008/markup" type="text/os-data">
<os:Var key="foo" value="123"/>
</script>
]]></Content>
</Module>
</artwork></figure>
</section>
<section title="Data Pipeline Elements and Proxied Content">
<t>When using Proxied Content to render a view for a gadget,
the container typically sends an HTTP GET request to the IRI
provided by the <Content> element's href attribute.
In such cases, any information the remote endpoint needs to
generate the necessary content in response is provided within
the IRI as query string parameters. There are times, however,
when the endpoint requires significantly more information
than what can be adequately conveyed by the limited nature
of query string parameters. To address such cases, Data Pipeline
elements can be used directly within a <Content> element
to define a collection of data that is to be sent to the
remote IRI using an HTTP POST request instead of GET.</t>
<figure><preamble>For instance, with typical, non-pipelined
Proxy content:</preamble><artwork><![CDATA[
<Content type="html" href="http://example.org/remote-server?id=${ViewParams.id}" />
]]></artwork></figure>
<figure><preamble>The ${ViewParams.id} expression would be replaced with
an appropriate value drawn from the View Parameters passed into the
gadget, and HTTP request sent to the remote endpoint by the
container would use HTTP GET:</preamble><![CDATA[
GET /remote-server?id=abc123 HTTP/1.1
Host: example.org
Accept: application/json
]]></figure>
<figure><preamble>However, when using Proxied Content with the
Data Pipeline elements, a developer can pass complex structured
data to remote endpoint:</preamble><artwork><![CDATA[
<Content type="html" href="http://example.org/remote-server">
<os:Var key="id" value="${ViewParams.id}" />
<os:Var key="name">
{
"givenName": "John",
"familyName": "Doe"
}
</os:Var>
</Content>
]]></artwork></figure>
<figure><preamble>The request is transformed into an HTTP POST with
a JSON-formatted payload containing the data specified by the
pipeline elements:</preamble><artwork><![CDATA[
POST /remote-server HTTP/1.1
Host: example.org
Content-Type: application-json
{
"id" : "abc123",
"name": {
"givenName": "John",
"familyName": "Doe"
}
}
]]></artwork></figure>
<t>As illustrated by the example, each of the keys defined by the
Data Pipeline elements contained within the <Content> are
used as the property names in the JSON object sent to the remote
endpoint. The value of each property is the value returned by
processing the Data Pipeline element.</t>
<t><Content< element's of this type can contain any number
of Data Pipeline elements, each of which MUST have a corresponding
property in the generated JSON object that is sent to the server,
regardless of whether the attempt to process a particular Data Pipeline
element results in an error or not. The value will be included within
the JSON exactly as it is provided by the <spanx style="verb">DataContext</spanx>.
Only pipelined data contained within the <Content> will be
included in the generated JSON object.</t>
<figure><preamble>In the follow example, assume that the processing
of the <os:DataRequest> element fails and error details are
provided by the <spanx style="verb">DataContext</spanx>:</preamble>
<artwork><![CDATA[
<Content type="html" href="http://example.org">
<os:Var key="key1" value="abc" />
<os:DataRequest key="key2" method="people.get" userId="@viewer" groupId="@frields" />
</Content>
]]></artwork></figure>
<figure><preamble>The JSON data posted to the remote server would
include the value of "key1" and the error information for "key2":</preamble>
<artwork><![CDATA[
POST / HTTP/1.1
Host: example.org
Content-Type: application/json
{
"key1" : "abc",
"key2" : {
"error" : {
"message" : "Processing Error",
"code" : 500
}
}
}
]]></artwork></figure>
<t>When a gadget specification uses unrecognized Data Pipeline tags,
the container is given the choice of either ignoring it or specifying
a 404 (Not Found) error within the JSON using the basic error structure
defined in [TODO: Reference to Core API spec].</t>
<figure><preamble>In the example below, the gadget specification uses
an undefined, hypothetical Data Pipeline extension:</preamble><artwork><![CDATA[
<Content type="html" href="http://example.org">
<os:Var key="key1" value="abc" />
<osx:MyExtension key="key2" attr="val" />
</Content>
]]></artwork></figure>
<figure><preamble>The JSON generated for the HTTP POST would either
omit "key2" entirely:</preamble>
<artwork><![CDATA[
POST / HTTP/1.1
Host: example.org
Content-Type: application/json
{
"key1" : "abc"
}
]]></artwork></figure>
<figure><preamble>Or indicate that key2 could not be resolved:</preamble>
<artwork><![CDATA[
POST / HTTP/1.1
Host: example.org
Content-Type: application/json
{
"key1" : "abc",
"key2" : {
"error: {
"message": "Not Found",
"code": 404
}
}
}
]]></artwork></figure>
<t>In response to the HTTP POST request, the remote server SHOULD
return HTML content suitable for rendering by the container. Such
content will be processed in the same way as type "html" <Content>
elements and MAY contain embedded Data Pipeline elements using the
customized HTML <script> tag.</t>
</section>
<section title="The Data Pipeline Elements">
<t>This specification currently defines three Data Pipeline elements:
<list style="hanging">
<t hangText="os:DataRequest">Used to access social data provided
by the container.</t>
<t hangText="os:HttpRequest">Used to access data provided by
a remote endpoint.</t>
<t hangText="os:Var">Used to define static, literal variables.</t>
</list>
</t>
<t>The XML Namespace for each of the Data Pipeline elements is
"http://ns.opensocial.org/2008/markup". By convention, this
specification uses the namespace prefix "os:" for the Data
Pipeline Namespace. Note, however, that the choice of prefix
is arbitrary and not semantically equivalent.</t>
<t>Implementations MAY define additional Data Pipeline elements
that use an XML Namespace other than "http://ns.opensocial.org/2008/markup".
Such extensions MUST contain a "key" attribute whose value specifies
the name that will be used to associate that element's resolved value
within the <spanx style="verb">DataContext</spanx>. Containers are
recommended to ignore any Data Pipeline elements they do not support or
do not understand.</t>
<figure><artwork>
namespace os = "http://ns.opensocial.org/2008/markup"
BaseDataPipeline = {
attribute key { text },
undefinedAttribute*
}
</artwork></figure>
<section title="The <os:DataRequest> Element" anchor="DataRequest">
<t>The <os:DataRequest></t> element is used to access social
data provided by the container. It is generally equivalent to the
JavaScript API methods provided to the gadget by the container to
access information such as the identity of the current viewer,
a particular users group of friends, an activity stream, and so forth.
the <os:DataRequest> element can only be used to access content
and cannot be used to modify social data stored by the container.</t>
<figure><artwork>
DataRequest = element os:DataRequest {
BaseDataPipeline,
attribute method { text }
}
</artwork></figure>
<t>In addition to the "key" attribute required for all Data Pipeline
elements, the <os:DataRequest> element MUST contain a "method"
attribute whose value identifies the data retrieval operation that
is to be performed as defined by [TODO: Reference to new core data/api
spec]. All additional attributes contained by the <os:DataRequest>
are to be mapped directly to the identically named input parameters
for the data retrieval operation specified by the "method".</t>
<t>Child elements or any additional content provided within the
<os:DataRequest> element SHOULD be ignored.</t>
<t>For example, the "people.get" operation defined by
[TODO: Core Data/API Spec] allows a gadget to retrieve information
about people known to the container. It can, for instance, be used
to retrieve basic profile details about an individual or retrieve a
listing of associated profiles (e.g. a "friends" list).</t>
<figure><preamble>When invoked using the JavaScript API to retrieve the listing
of profiles associated with the gadget's current viewer, the
method is invoked as:</preamble><artwork>
var params = {"userId": "@viewer", "groupId": "@friends"};
osapi.people.get(params).execute(callback);
function callback(friends) { ... }
</artwork></figure>
<figure><preamble>The equivalent operation using the <os:DataRequest>
element would be:</preamble><artwork><![CDATA[
<os:DataRequest key="friends" method="people.get" userId="@viewer" groupId="@friends" />
]]></artwork></figure>
<t>Assuming the method identified was processed successfully, the
"friends" variable will be added to the <spanx style="verb">DataContext</spanx>
with a value identical to that provided as input to the callback
function in the previous JavaScript API.</t>
</section>
<section title="The <os:HttpRequest> Element" anchor="HttpRequest">
<t>The <os:HttpRequest></t> element is used to access arbitrary
content from a remote HTTP endpoint. It is generally equivalent to the
<spanx style="verb">osapi.http</spanx> JavaScript API methods provided
to the gadget by the container. Currently, only HTTP GET and POST
methods are supported.</t>
<figure><artwork>
DataRequest = element os:DataRequest {
BaseDataPipeline,
SignedFetch,
attribute method { "get" | "post" },
attribute href { IRI }
}
</artwork></figure>
<t>In addition to the "key" attribute required for all Data Pipeline
elements, the <os:HttpRequest> element MUST contain a "method"
attribute whose value is either "get" or "post" indicating the HTTP
request method to use for the request, and an "href" attribute whose
value specifies a dereferenceable IRI to which the request will be
sent. If not specified, the value of the "method" attribute will be
assumed to be "get". All additional attributes contained by the
<os:HttpRequest> are to be mapped directly to the identically
named input parameters defined for corresponding
<spanx style="verb">osapi.http.get</spanx> or <spanx style="verb">osapi.http.post</spanx>
JavaScript APIs.</t>
<t>Child elements or any additional content provided within the
<os:HttpRequest> element SHOULD be ignored.</t>
<figure><preamble>For example, the <spanx style="verb">osapi.http.get</spanx>
operation can be used to retrieve content from a remote endpoint
located a "http://example.org/foo":</preamble><artwork>
var params = {"format":"json", "authz":"signed"};
osapi.http.get(params).execute(callback);
function callback(results) { ... }
</artwork></figure>
<figure><preamble>The equivalent operation using the <os:HttpRequest>
element would be:</preamble><artwork><![CDATA[
<os:HttpRequest key="results" format="json" authz="signed" />
]]></artwork></figure>
<t>Assuming the method identified was processed successfully, the
"results" variable will be added to the <spanx style="verb">DataContext</spanx>
with a value identical to that provided as input to the callback
function in the previous JavaScript API.</t>
<t>The exact structure of the value stored within the <spanx style="verb">DataContext</spanx>
will depend on the type of response returned by the remote endpoint.
Currently, the only supported options are for the server to return
JSON-formatted data, textual content (e.g. plain text or HTML) or
an error response. Binary content is currently not supported.</t>
<figure><preamble>If the server returns a JSON formatted response
such as:</preamble><artwork>
HTTP/1.1 200 OK
Content-Type: application/json
{ "data" : "xyz" }
</artwork></figure>
<figure><preamble>The value stored within the <spanx style="verb">DataContext</spanx>
would be:</preamble><artwork>
{
"result" : {
"content" : {
"data" : "xyz"
},
"status" : 200,
"headers" : {
"Content-Type" : [
"application/json"
]
}
}
}
</artwork></figure>
<figure><preamble>If the server turns textual content such as:</preamble>
<artwork><![CDATA[
HTTP/1.1 200 OK
Content-Type: text/html
<html><head>...</head><body>Hello World</body></html>
]]></artwork></figure>
<figure><preamble>The value stored within the <spanx style="verb">DataContext</spanx>
would be:</preamble><artwork><![CDATA[
{
"result" : {
"content" : "<html><head>...</head><body>Hello World</body></html>",
"result": 200,
"headers" : {
"Content-Type" : [
"text/html"
]
}
}
}
]]></artwork></figure>
<figure><preamble>If the server returns an error, or if the content
returned by the server is neither text-based or valid JSON, the value
stored by the <spanx style="verb">DataContext</spanx> will specify an
error with an appropriate value indicating the nature of the error.
Within the value of the "error" property, the "data" property contains
the content actually returned by the server, if any.</preamble>
<artwork><![CDATA[
{
"error" : {
"code" : 404,
"message" : "Not Found",
"data" : {
"content" : "<html><body>File not found</body></html>",
"headers" : {
"Content-Type" : [
"text/html"
]
}
}
}
}
]]></artwork></figure>
</section>
<section title="The <os:Var> Element" anchor="Var">
<t>The <os:Var></t> element is used to declare a string
JSON object, or JavaScript array as a literal value or associate
the results of evaluating an Expression Statement with a key.</t>
<figure><artwork>
SimpleVar = element os:Var {
BaseDataPipeline,
value { text }
}
JsonVar = element os:Var {
BaseDataPipeline,
text
}
Var = SimpleVar | JsonVar
</artwork></figure>
<t>In addition to the "key" attribute required for all
Data Pipeline elements, the <os:Var> element can either
contain a "value" attribute or contain textual content.</t>
<figure><preamble>The following <os:Var> examples are equivalent:</preamble>
<artwork><![CDATA[
<os:Var key="key1" value="ABC" />
<os:Var key="key1">ABC</os:Var>
]]></artwork></figure>
<t>When determining the value to store within the <spanx style="verb">DataContext</spanx>,
the container will:
<list style="symbols">
<t>First apply Variable Substitution to replace all Expression
Language statements appearing within the value,</t>
<t>Then, attempt to parse the string as a JavaScript Array. For
instance, if the value is "[1,2,3,4,5]", then a JavaScript Array
containing the values 1, 2, 3, 4 and 5 will be created. If
successfully parsed, the Array will be stored into the
<spanx style="verb">DataContext</spanx> and processing will end.</t>
<t>If unable to parse as an array, the container will attempt
to parse the value as a JSON Object. If successfully parsed, the
object will be stored into the <spanx style="verb">DataContext</spanx>
and processing will end.</t>
<t>If the value is still unable to be parsed, it will be stored
within the <spanx style="verb">DataContext</spanx> as a literal
string value.</t>
</list>
</t>
<figure><preamble>Given the example,</preamble><artwork><![CDATA[
<os:Var key="key1">
[{"data": ${1+1}}]
<os:Var>
]]></artwork></figure>
<figure><preamble>The value stored within the <spanx style="verb">DataContext</spanx>
will be a JavaScript array containing a JSON object with a single
property "data" whose value is the result of processing the Expression
Language statement "${1+1}":</preamble><artwork><![CDATA[
[
{ "data":2 }
]
]]></artwork></figure>
</section>
<section title="Expression Statements within Data Pipeline Attributes">
<t>The attributes used with the <os:DataRequest>
and <os:HttpRequest> elements often require values that
vary depending on a range of criteria including input parameters
or the results returned by other Data Pipeline elements.</t>
<t>For instance, when the gadget needs to display the next
set of results from a paged-collection, the view
parameters specifying the current page can be passed into
the <os:HttpRequest> or <os:DataRequest>
using Expression Language statements:</t>
<figure><artwork><![CDATA[
<os:DataRequest key="PagedFriends" method="people.get"
userId="@owner" groupId="@friends"
startIndex="${ViewParams.first}"
count="20"/>
]]></artwork></figure>
<t>Likewise, the results of one Data Pipeline element can
be passed as attribute values in another. For instance,
we can reference the stored value for the "PagedFriends"
key generated by processing the previous example within
an <os:HttpRequest>:</t>
<figure><artwork><![CDATA[
<os:HttpRequest href="http://example.com/api?ids=${PagedFriends.ids}"/>
]]></artwork></figure>
<t>When referencing the results of other Data Pipeline elements
in this way, Expression Language statements can access the input
parameters and additional metadata about the referenced results by using the special reserved
"Request" property on the referenced key, e.g. "${PagedFriends.Request.userId}"</t>
<t>The metadata properties provided by the "Request" property include:
<list style="symbols">
<t>All of the attributes defined on the referenced Data Pipeline
element (in the previous example, these would include "key", "method",
"userId", "groupId", "startIndex", and "count"),
<t>The properties "totalResults", "count" and "startIndex" if
the value stored by referenced key is a Collection object
as defined by [TODO: Reference Core Data]; the values of which
map to the corresponding values of the stored collection.</t>
</list>
</t>
<t>These properties are determined dynamically by the container
and can be used within any Expression Language statement:</t>
<figure><artwork><![CDATA[
<os:DataRequest key="Page1" method="people.get"
userId="@owner" groupId="@friends"
startIndex="${ViewParams.first}" count="20"/>
<os:DataRequest key="Page2" method="people.get"
userId="${Page1.Request.userId}" groupId="@friends"
startIndex="${Page1.Request.startIndex + Page1.Request.count}"
count="${Page1.Request.count}"/>
]]></artwork></figure>
</section>
</section>
</section>
<!-- END DATA PIPELINING -->
a