Core-Gadget - Data Pipelining
This is a DRAFT
  <section title="Data Pipelining" anchor="DataPipelining">       <t>Data Pipelining is a declarative mechanism for defining the data a    gadget requires from the container. XML Elements are used to associate    variable keys with specific kinds of data from a variety of different    sources. For instance, the <os:DataRequest> element can be used    to access social data provided by the OpenSocial container, while the    <os:HttpRequest> element can provide access to content from any    HTTP endpoint.</t>        <t>All Data Pipeline elements have a 'key' attribute whose value is    used to uniquely identify the data. Such keys can be used within    Expression Language statements used for Variable Substitution or    with JavaScript API calls. Key names are always case-sensitive.</t>    <figure><preamble>The following illustrates a basic example of    Data Pipelining in a partial gadget specification:</preamble>    <artwork xml:space="preserve"><![CDATA[  <Module xmlns:os="http://ns.opensocial.org/2008/markup">   ...   <Data>    <os:DataRequest key="viewer" method="people.get" userId="@viewer" fields="name,birthday" />    <os:HttpRequest key="remote" href="http://example.com/api" />   </Data>   <Content type="html">    The viewer name is ${viewer.name}.        The remote content is "${remote.result.content}".   </Content>  </Module>    ]]></artwork></figure>        <t>The Data Pipeline elements can appear:     <list style="symbols">      <t>As children of the <Data> element,</t>      <t>As direct children of the <Content> element when the Proxied      Content model is used, and </t>      <t>Within specially formatted HTML <script> tags included      within HTML Content.</t>     </list>    </t>        <t>The placement of Data Pipeline elements will determine where the    data associatd with the key specified will be made available to the    container while the gadget is being rendered. The considerations    and requirements for each of the options listed above are discussed    in detail in the relevant sections that follow.</t>        <section title="The DataContext" anchor="DataContext">     <t>For every gadget instance, a <spanx style="verb">DataContext</spanx>     object is created that maintains all of the data associated with the     gadget instance's Data Pipeline elements. The information contained     within the <spanx style="verb">DataContext</spanx> is available for     use with Variable Substitution via Expression Language statements     or via the <spanx style="verb">opensocial.data</spanx> APIs; and while     the data contained within the <spanx style="verb">DataContext</spanx>     is generally populated through processing of the Data Pipeline     elements, the <spanx style="verb">opensocial.data</spanx> APIs can     be used to modify values or add additional keys.</t>    </section>        <section title="Data Pipeline Elements and the Data Block">         <t>By default, when Data Pipeline elements appear as children of the     <Data> element, the container will process the elements and     register the resulting data with the <spanx style="verb">DataContext</spanx>     for all views.</t>          <t>Individual Data Pipeline elements, however, MAY specify a "view"     attribute whose value is a comma separated list of specific views     for which the element is associated. Such elements will only be     processed and added to the <spanx style="verb">DataContext</spanx>     when the identified views are being rendered.</t>          <figure><preamble>In the following example, the <os:HttpRequest>     element using key "foo" is registered globally and available within     all views while the <os:HttpRequest> element using key "bar"     is only processed and available when the "example" view is being     rendered:</preamble><artwork><![CDATA[  <Module>   ...   <Data>    <os:HttpRequest key="foo" href="http://example.com/api" />    <os:HttpRequest key="bar" href="http://example.com/api" view="example"/>   </Data>   <Content type="html">    ...   </Content>   <Content type="html" view="example">    ...   </Content>  </Module>     ]]></artwork></figure>          <t>Multiple Data Pipeline elements MUST NOT share the same value for     the key attribute unless each is specifically targeted at a different     view. In the following example, the key "foo" would resolve to the     value "abc" when the default view is rendered and to "123" when the     "example" view is rendered:</t>          <figure><artwork><![CDATA[  <Module>   ...   <Data>    <os:Var key="foo" value="abc" />    <os:Var key="foo" value="abc" view="example" />   </Data>   <Content type="html">    ...   </Content>   <Content type="html" view="example">    ...   </Content>  </Module>         ]]></artwork></figure>          <t>Note that currently, this specification does not specify     exactly when the container processes the collection of Data Pipeline     elements, nor are containers required to process data elements in     any particular order. Containers that process Data Pipeline elements     to populate the <spanx style="verb">DataContext</spanx> will likely     do so regardless of whether the referenced key is ever actually     used within an expression statement or API call.</t>        </section>        <section title="Data Pipeline Elements within HTML Content">         <t>When using HTML Content, Data Pipeline elements MAY appear     within the HTML inside a special <script> tag.</t>          <figure><preamble>For instance,</preamble><artwork>  <Module>   <ModulePrefs>    <Require feature="opensocial-data" />   </ModulePrefs>   <Content type="html"><![CDATA[    <script xmlns:os="http://ns.opensocial.org/2008/markup" type="text/os-data">     <os:HttpRequest key="foo" href="http://example.com/api" />     <os:Var key="bar" value="abc123"/>    </script>   ]]></Content>  </Module>     </artwork></figure>          <t>As illustrated, when Data Pipeline elements are embedded within     HTML Content, the gadget specification MUST require the "opensocial-data"     feature using the <Require> element within the <ModulePrefs>.</t>          <t>The Data Pipeline elements themselves are contained within an     HTML script tag whose type attribute specifies "text/os-data". Also     note that the Data Pipeline XML namespace is declared for the "os:"     prefix. This is required in order for the embedded Data Pipeline     elements to processed properly while the content is being rendered.</t>         <t>Data Pipeline elements defined in this manner are scoped specifically     to the <Content> element in which they are embedded, meaning that     the data each provides to the <spanx style="verb">DataContext</spanx>     will be available for expression statements and API calls that are     also in the same <Content> element.</t>          <t>If an embedded Data Pipeline element specifies the same key as a     Data Pipeline element contained within the <Data> element,     the value provided by the embedded elements takes precedence. In the     following example, the variable "foo" will resolve to "123" when     the default view is rendered:</t>         <figure><artwork>  <Module>   <ModulePrefs>    <Require feature="opensocial-data" />   </ModulePrefs>   <Data>    <os:Var key="foo" value="abc"/>   </Data>   <Content type="html"><![CDATA[    <script xmlns:os="http://ns.opensocial.org/2008/markup" type="text/os-data">     <os:Var key="foo" value="123"/>    </script>   ]]></Content>  </Module>     </artwork></figure>    </section>        <section title="Data Pipeline Elements and Proxied Content">         <t>When using Proxied Content to render a view for a gadget,     the container typically sends an HTTP GET request to the IRI     provided by the <Content> element's href attribute.     In such cases, any information the remote endpoint needs to     generate the necessary content in response is provided within     the IRI as query string parameters. There are times, however,     when the endpoint requires significantly more information     than what can be adequately conveyed by the limited nature     of query string parameters. To address such cases, Data Pipeline     elements can be used directly within a <Content> element     to define a collection of data that is to be sent to the     remote IRI using an HTTP POST request instead of GET.</t>          <figure><preamble>For instance, with typical, non-pipelined     Proxy content:</preamble><artwork><![CDATA[  <Content type="html" href="http://example.org/remote-server?id=${ViewParams.id}" />     ]]></artwork></figure>          <figure><preamble>The ${ViewParams.id} expression would be replaced with     an appropriate value drawn from the View Parameters passed into the     gadget, and HTTP request sent to the remote endpoint by the     container would use HTTP GET:</preamble><![CDATA[  GET /remote-server?id=abc123 HTTP/1.1  Host: example.org  Accept: application/json     ]]></figure>          <figure><preamble>However, when using Proxied Content with the     Data Pipeline elements, a developer can pass complex structured     data to remote endpoint:</preamble><artwork><![CDATA[  <Content type="html" href="http://example.org/remote-server">   <os:Var key="id" value="${ViewParams.id}" />   <os:Var key="name">    {     "givenName": "John",     "familyName": "Doe"    }   </os:Var>  </Content>     ]]></artwork></figure>          <figure><preamble>The request is transformed into an HTTP POST with     a JSON-formatted payload containing the data specified by the     pipeline elements:</preamble><artwork><![CDATA[  POST /remote-server HTTP/1.1  Host: example.org  Content-Type: application-json    {   "id" : "abc123",   "name": {    "givenName": "John",    "familyName": "Doe"   }  }     ]]></artwork></figure>          <t>As illustrated by the example, each of the keys defined by the     Data Pipeline elements contained within the <Content> are     used as the property names in the JSON object sent to the remote     endpoint. The value of each property is the value returned by     processing the Data Pipeline element.</t>          <t><Content< element's of this type can contain any number     of Data Pipeline elements, each of which MUST have a corresponding     property in the generated JSON object that is sent to the server,     regardless of whether the attempt to process a particular Data Pipeline     element results in an error or not. The value will be included within     the JSON exactly as it is provided by the <spanx style="verb">DataContext</spanx>.     Only pipelined data contained within the <Content> will be     included in the generated JSON object.</t>          <figure><preamble>In the follow example, assume that the processing     of the <os:DataRequest> element fails and error details are     provided by the <spanx style="verb">DataContext</spanx>:</preamble>     <artwork><![CDATA[  <Content type="html" href="http://example.org">   <os:Var key="key1" value="abc" />   <os:DataRequest key="key2" method="people.get" userId="@viewer" groupId="@frields" />  </Content>     ]]></artwork></figure>          <figure><preamble>The JSON data posted to the remote server would     include the value of "key1" and the error information for "key2":</preamble>     <artwork><![CDATA[  POST / HTTP/1.1  Host: example.org  Content-Type: application/json    {   "key1" : "abc",   "key2" : {    "error" : {     "message" : "Processing Error",     "code" : 500    }   }  }     ]]></artwork></figure>          <t>When a gadget specification uses unrecognized Data Pipeline tags,     the container is given the choice of either ignoring it or specifying     a 404 (Not Found) error within the JSON using the basic error structure     defined in [TODO: Reference to Core API spec].</t>          <figure><preamble>In the example below, the gadget specification uses     an undefined, hypothetical Data Pipeline extension:</preamble><artwork><![CDATA[  <Content type="html" href="http://example.org">   <os:Var key="key1" value="abc" />   <osx:MyExtension key="key2" attr="val" />  </Content>     ]]></artwork></figure>          <figure><preamble>The JSON generated for the HTTP POST would either     omit "key2" entirely:</preamble>     <artwork><![CDATA[  POST / HTTP/1.1  Host: example.org  Content-Type: application/json    {   "key1" : "abc"  }     ]]></artwork></figure>     <figure><preamble>Or indicate that key2 could not be resolved:</preamble>     <artwork><![CDATA[  POST / HTTP/1.1  Host: example.org  Content-Type: application/json    {   "key1" : "abc",   "key2" : {    "error: {     "message": "Not Found",     "code": 404    }   }  }     ]]></artwork></figure>     <t>In response to the HTTP POST request, the remote server SHOULD     return HTML content suitable for rendering by the container. Such     content will be processed in the same way as type "html" <Content>     elements and MAY contain embedded Data Pipeline elements using the     customized HTML <script> tag.</t>    </section>        <section title="The Data Pipeline Elements">         <t>This specification currently defines three Data Pipeline elements:      <list style="hanging">       <t hangText="os:DataRequest">Used to access social data provided       by the container.</t>       <t hangText="os:HttpRequest">Used to access data provided by       a remote endpoint.</t>       <t hangText="os:Var">Used to define static, literal variables.</t>      </list>     </t>          <t>The XML Namespace for each of the Data Pipeline elements is     "http://ns.opensocial.org/2008/markup". By convention, this     specification uses the namespace prefix "os:" for the Data     Pipeline Namespace. Note, however, that the choice of prefix     is arbitrary and not semantically equivalent.</t>          <t>Implementations MAY define additional Data Pipeline elements     that use an XML Namespace other than "http://ns.opensocial.org/2008/markup".     Such extensions MUST contain a "key" attribute whose value specifies     the name that will be used to associate that element's resolved value     within the <spanx style="verb">DataContext</spanx>. Containers are     recommended to ignore any Data Pipeline elements they do not support or     do not understand.</t>         <figure><artwork>  namespace os = "http://ns.opensocial.org/2008/markup"       BaseDataPipeline = {   attribute key { text },   undefinedAttribute*  }     </artwork></figure>         <section title="The <os:DataRequest> Element" anchor="DataRequest">           <t>The <os:DataRequest></t> element is used to access social      data provided by the container. It is generally equivalent to the      JavaScript API methods provided to the gadget by the container to      access information such as the identity of the current viewer,      a particular users group of friends, an activity stream, and so forth.      the <os:DataRequest> element can only be used to access content      and cannot be used to modify social data stored by the container.</t>      <figure><artwork>  DataRequest = element os:DataRequest {   BaseDataPipeline,   attribute method { text }  }      </artwork></figure>      <t>In addition to the "key" attribute required for all Data Pipeline      elements, the <os:DataRequest> element MUST contain a "method"      attribute whose value identifies the data retrieval operation that      is to be performed as defined by [TODO: Reference to new core data/api      spec]. All additional attributes contained by the <os:DataRequest>      are to be mapped directly to the identically named input parameters      for the data retrieval operation specified by the "method".</t>            <t>Child elements or any additional content provided within the      <os:DataRequest> element SHOULD be ignored.</t>            <t>For example, the "people.get" operation defined by      [TODO: Core Data/API Spec] allows a gadget to retrieve information      about people known to the container. It can, for instance, be used      to retrieve basic profile details about an individual or retrieve a      listing of associated profiles (e.g. a "friends" list).</t>            <figure><preamble>When invoked using the JavaScript API to retrieve the listing      of profiles associated with the gadget's current viewer, the      method is invoked as:</preamble><artwork>  var params = {"userId": "@viewer", "groupId": "@friends"};  osapi.people.get(params).execute(callback);    function callback(friends) { ... }      </artwork></figure>            <figure><preamble>The equivalent operation using the <os:DataRequest>      element would be:</preamble><artwork><![CDATA[  <os:DataRequest key="friends" method="people.get" userId="@viewer" groupId="@friends" />      ]]></artwork></figure>            <t>Assuming the method identified was processed successfully, the      "friends" variable will be added to the <spanx style="verb">DataContext</spanx>      with a value identical to that provided as input to the callback      function in the previous JavaScript API.</t>          </section>          <section title="The <os:HttpRequest> Element" anchor="HttpRequest">           <t>The <os:HttpRequest></t> element is used to access arbitrary      content from a remote HTTP endpoint. It is generally equivalent to the      <spanx style="verb">osapi.http</spanx> JavaScript API methods provided      to the gadget by the container. Currently, only HTTP GET and POST      methods are supported.</t>            <figure><artwork>  DataRequest = element os:DataRequest {   BaseDataPipeline,   SignedFetch,   attribute method { "get" | "post" },   attribute href { IRI }  }      </artwork></figure>           <t>In addition to the "key" attribute required for all Data Pipeline      elements, the <os:HttpRequest> element MUST contain a "method"      attribute whose value is either "get" or "post" indicating the HTTP      request method to use for the request, and an "href" attribute whose      value specifies a dereferenceable IRI to which the request will be      sent. If not specified, the value of the "method" attribute will be      assumed to be "get". All additional attributes contained by the      <os:HttpRequest> are to be mapped directly to the identically      named input parameters defined for corresponding      <spanx style="verb">osapi.http.get</spanx> or <spanx style="verb">osapi.http.post</spanx>      JavaScript APIs.</t>            <t>Child elements or any additional content provided within the      <os:HttpRequest> element SHOULD be ignored.</t>            <figure><preamble>For example, the <spanx style="verb">osapi.http.get</spanx>      operation can be used to retrieve content from a remote endpoint      located a "http://example.org/foo":</preamble><artwork>  var params = {"format":"json", "authz":"signed"};  osapi.http.get(params).execute(callback);    function callback(results) { ... }      </artwork></figure>            <figure><preamble>The equivalent operation using the <os:HttpRequest>      element would be:</preamble><artwork><![CDATA[  <os:HttpRequest key="results" format="json" authz="signed" />      ]]></artwork></figure>            <t>Assuming the method identified was processed successfully, the      "results" variable will be added to the <spanx style="verb">DataContext</spanx>      with a value identical to that provided as input to the callback      function in the previous JavaScript API.</t>            <t>The exact structure of the value stored within the <spanx style="verb">DataContext</spanx>      will depend on the type of response returned by the remote endpoint.      Currently, the only supported options are for the server to return      JSON-formatted data, textual content (e.g. plain text or HTML) or      an error response. Binary content is currently not supported.</t>            <figure><preamble>If the server returns a JSON formatted response      such as:</preamble><artwork>  HTTP/1.1 200 OK  Content-Type: application/json    { "data" : "xyz" }      </artwork></figure>           <figure><preamble>The value stored within the <spanx style="verb">DataContext</spanx>      would be:</preamble><artwork>  {   "result" : {    "content" : {     "data" : "xyz"    },    "status" : 200,    "headers" : {     "Content-Type" : [      "application/json"     ]    }   }  }      </artwork></figure>           <figure><preamble>If the server turns textual content such as:</preamble>      <artwork><![CDATA[  HTTP/1.1 200 OK  Content-Type: text/html    <html><head>...</head><body>Hello World</body></html>      ]]></artwork></figure>           <figure><preamble>The value stored within the <spanx style="verb">DataContext</spanx>      would be:</preamble><artwork><![CDATA[  {   "result" : {    "content" : "<html><head>...</head><body>Hello World</body></html>",    "result": 200,    "headers" : {     "Content-Type" : [      "text/html"     ]    }   }  }      ]]></artwork></figure>           <figure><preamble>If the server returns an error, or if the content      returned by the server is neither text-based or valid JSON, the value      stored by the <spanx style="verb">DataContext</spanx> will specify an      error with an appropriate value indicating the nature of the error.      Within the value of the "error" property, the "data" property contains      the content actually returned by the server, if any.</preamble>      <artwork><![CDATA[  {   "error" : {    "code" : 404,    "message" : "Not Found",    "data" : {     "content" : "<html><body>File not found</body></html>",     "headers" : {      "Content-Type" : [       "text/html"      ]     }    }   }  }      ]]></artwork></figure>          </section>          <section title="The <os:Var> Element" anchor="Var">           <t>The <os:Var></t> element is used to declare a string      JSON object, or JavaScript array as a literal value or associate      the results of evaluating an Expression Statement with a key.</t>            <figure><artwork>  SimpleVar = element os:Var {   BaseDataPipeline,   value { text }  }    JsonVar = element os:Var {   BaseDataPipeline,   text  }    Var = SimpleVar | JsonVar      </artwork></figure>           <t>In addition to the "key" attribute required for all      Data Pipeline elements, the <os:Var> element can either      contain a "value" attribute or contain textual content.</t>            <figure><preamble>The following <os:Var> examples are equivalent:</preamble>      <artwork><![CDATA[  <os:Var key="key1" value="ABC" />    <os:Var key="key1">ABC</os:Var>      ]]></artwork></figure>        <t>When determining the value to store within the <spanx style="verb">DataContext</spanx>,      the container will:       <list style="symbols">        <t>First apply Variable Substitution to replace all Expression        Language statements appearing within the value,</t>        <t>Then, attempt to parse the string as a JavaScript Array. For        instance, if the value is "[1,2,3,4,5]", then a JavaScript Array        containing the values 1, 2, 3, 4 and 5 will be created. If        successfully parsed, the Array will be stored into the        <spanx style="verb">DataContext</spanx> and processing will end.</t>        <t>If unable to parse as an array, the container will attempt        to parse the value as a JSON Object. If successfully parsed, the        object will be stored into the <spanx style="verb">DataContext</spanx>        and processing will end.</t>        <t>If the value is still unable to be parsed, it will be stored        within the <spanx style="verb">DataContext</spanx> as a literal        string value.</t>       </list>      </t>            <figure><preamble>Given the example,</preamble><artwork><![CDATA[  <os:Var key="key1">   [{"data": ${1+1}}]  <os:Var>      ]]></artwork></figure>           <figure><preamble>The value stored within the <spanx style="verb">DataContext</spanx>      will be a JavaScript array containing a JSON object with a single      property "data" whose value is the result of processing the Expression      Language statement "${1+1}":</preamble><artwork><![CDATA[  [   { "data":2 }  ]      ]]></artwork></figure>          </section>          <section title="Expression Statements within Data Pipeline Attributes">           <t>The attributes used with the <os:DataRequest>      and <os:HttpRequest> elements often require values that      vary depending on a range of criteria including input parameters      or the results returned by other Data Pipeline elements.</t>            <t>For instance, when the gadget needs to display the next      set of results from a paged-collection, the view      parameters specifying the current page can be passed into      the <os:HttpRequest> or <os:DataRequest>      using Expression Language statements:</t>            <figure><artwork><![CDATA[  <os:DataRequest key="PagedFriends" method="people.get"   userId="@owner" groupId="@friends"   startIndex="${ViewParams.first}"   count="20"/>      ]]></artwork></figure>            <t>Likewise, the results of one Data Pipeline element can      be passed as attribute values in another. For instance,      we can reference the stored value for the "PagedFriends"      key generated by processing the previous example within      an <os:HttpRequest>:</t>            <figure><artwork><![CDATA[  <os:HttpRequest href="http://example.com/api?ids=${PagedFriends.ids}"/>      ]]></artwork></figure>            <t>When referencing the results of other Data Pipeline elements      in this way, Expression Language statements can access the input      parameters and additional metadata about the referenced results by using the special reserved      "Request" property on the referenced key, e.g. "${PagedFriends.Request.userId}"</t>            <t>The metadata properties provided by the "Request" property include:       <list style="symbols">        <t>All of the attributes defined on the referenced Data Pipeline        element (in the previous example, these would include "key", "method",        "userId", "groupId", "startIndex", and "count"),        <t>The properties "totalResults", "count" and "startIndex" if        the value stored by referenced key is a Collection object        as defined by [TODO: Reference Core Data]; the values of which        map to the corresponding values of the stored collection.</t>       </list>      </t>            <t>These properties are determined dynamically by the container      and can be used within any Expression Language statement:</t>            <figure><artwork><![CDATA[  <os:DataRequest key="Page1" method="people.get"   userId="@owner" groupId="@friends"   startIndex="${ViewParams.first}" count="20"/>  <os:DataRequest key="Page2" method="people.get"   userId="${Page1.Request.userId}" groupId="@friends"   startIndex="${Page1.Request.startIndex + Page1.Request.count}"   count="${Page1.Request.count}"/>           ]]></artwork></figure>          </section>    </section>  </section>   <!--  END DATA PIPELINING -->
a