Embracing JSON? Of course, but how?

Eric van der Vlist

vdv@dyomedea.com, @evlist (identica, twitter, skype, ...)

--Wikimedia

Introduction

The beauty of XML

The beauty of XML is that it can be self-documenting and easy to comprehend by humans.

--Aaron Swartz

Aaron Swartz in 2001

--Wikimedia

Why JSON?

JSON can be found at all good XML conferences.

--Robin Berjon

Too late?

Sorry, I disagree on the timing of this submission ... I think JSON has already been embraced ... we've moved on, for all the reasons that you state.

--Assigned_Reviewer_3

Maybe not

It remains an important topic and this proposed paper seems to make a useful contribution to the debate.

--Assigned_Reviewer_4

Take care!

By accepting it I think we're putting a lot of trust in the author to create something where the detail is worthwhile.

--Assigned_Reviewer_4

I'll do my best

But we have only 30 minutes...

My challenge

Change, even slightly, your perspective on JSON and XML.

--Eric van der Vlist

Fasten your seatbelt

100 slides steps to see in 30 minutes!

--sam_lens

Data models

JSON and XML

--Wikimedia

XML and JSON

<?xml version="1.0" encoding="UTF-8" ?>
<person>
    <firstName>John</firstName>
    <lastName>Smith</lastName>
    <age>25</age>
    <address>
		<streetAddress>21 2nd Street</streetAddress>
		<city>New York</city>
		<state>NY</state>
		<postalCode>10021</postalCode>
	</address>
    <phoneNumber>
        <type>home</type>
        <number>212 555-1234</number>
    </phoneNumber>
    <phoneNumber>
        <type>fax</type>
        <number>646 555-4567</number>
    </phoneNumber>
</person>

{
  "person": {
    "firstName": "John",
    "lastName": "Smith",
    "age": "25",
    "address": {
      "streetAddress": "21 2nd Street",
      "city": "New York",
      "state": "NY",
      "postalCode": "10021"
    },
    "phoneNumber": [
      {
        "type": "home",
        "number": "212 555-1234"
      },
      {
        "type": "fax",
        "number": "646 555-4567"
      }
    ]
  }
}

They look similar

Like these

--Wikimedia

They look similar

When they describe the same object...

Similar?

Their data models are different

XML

--Wikimedia

XML Data Model

Hmmm... Which one?

XML + Namespaces in XML 1.0|1.1

No data model there...

XML Information Set

Its purpose is to provide a consistent set of definitions for use in other specifications that need to refer to the information in a well-formed XML document

--W3C

PSVI

the augmented infoset which results from conformant processing as defined in this [XML Schema] specification

--W3C

XDM

Subset of XML Infoset + PSVI + other "kinds" of "items" used by XPath/XSLT/XQuery
One flavour per spec and version (8 up to now...)

Tip number one

When you hear the words "XML Data Model", always ask "which one?"

Tip number two

When you hear the word "XDM", always ask "which version? which flavor?"

Our XML Data Model

The XML Data Model which we'll be considering here is the subset of the XML Infoset used by the various versions of flavors of the XDM.

7 kind of nodes

Document
Element
Attribute
Text
Namespace
Comment
Processing instruction

Elements are the backbone

They have

A name
A map of attributes
An array of children nodes

Names

Namespace "qualified"
Local parts follow the "NCName production"

Map of attributes

Key/value pairs
Keys are (qualified) names
Values are textual

Array of children nodes

Array of elements, text, comment and PI nodes
"Text Nodes that would be adjacent must be combined into a single Text Node"

Legal characters

Legal characters are tab, carriage return, line feed, and the legal characters of Unicode and ISO/IEC 10646

This excludes "control" characters below #x20 other than tab, CR and LF.

JSON the fat free alternative

--Wikimedia

JSON Data Model

JSON can represent four primitive types (strings, numbers, booleans, and null) and two structured types (objects and arrays).

--RFC4627

Objects are maps

An object is an unordered collection of zero or more name/value pairs, where a name is a string and a value is a string, number, boolean, null, object, or array.

--RFC4627

Arrays are arrays

An array is an ordered sequence of zero or more values.

--RFC4627

No lexical restrictions

A string is a sequence of zero or more Unicode characters.

.../... a name is a string.

--RFC4627

JSON vs XML

--Wikimedia

Generic / specialized

JSON -> generic maps and arrays
XML -> elements package a name, a map (with limitations) and an array (with limitations)

Data types

JSON -> 4 primitive types (strings, numbers, booleans & null)
XML alone -> no datatype
XML + schema -> many datatypes

Lexical space

JSON -> no limitation
XML -> no control characters

Encodings

JSON -> UTF-8 only
XML -> any supported character encoding

Not a data model issue...

So what?

Solutions will depend on requirements

Direction
Scope
Loss
Round trips
Style
Query-ability

Different approaches

Bridges
Polyglotism
Serialization
Consolidation

oh, no!

Oh, no, not yet another JSON to XML mapping proposal!

--Michael Kay listening Steve Pemberton present JSON support in XForms 2.0 at XML Prague 2012

Use cases

Case 1: ACME Anvils

--Wikimedia

From XML to JSON

Acme Corporation has developed a set of RESTfull web services which sends and receives data oriented XML documents. Due to popular demand, they want to support JSON in addition to XML.

Or maybe the other way round

Acme Web Design has developed a set of RESTFull web services which sends and receives JSON documents. Their business users require the same information in XML.

ACME care about anvils

They don't care about XML and JSON data models
They just want to describe anvils

A good case for bridges

Bridging XML and JSON

Bridges can be ready to wear

--Wikimedia

XML to JSON

<?xml version="1.0" encoding="UTF-8"?>
<anvil reference="acme-5103">
    <weight unit="pound">9.5</weight>
    <composition>best wrought iron</composition>
    <price currency="USD">.15</price>
</anvil>

{
  "anvil": {
    "-reference": "acme-5103",
    "weight": {
      "-unit": "pound",
      "#text": "9.5"
    },
    "composition": "best wrought iron",
    "price": {
      "-currency": "USD",
      "#text": ".15"
    }
  }
}

Using http://jsontoxml.utilities-online.info/

JSON to XML

{
  "anvil": {
    "reference": "acme-5103",
    "weight": {
      "unit": "pound",
      "value": 9.5
    },
    "composition": "best wrought iron",
    "prices": {
        "usd": 0.15,
        "eur": 0.12
    }
  }
}

<?xml version="1.0" encoding="UTF-8" ?>
<anvil>
    <reference>acme-5103</reference>
    <weight>
        <unit>pound</unit>
        <value>9.5</value>
    </weight>
    <composition>best wrought iron</composition>
    <prices>
        <usd>0.15</usd>
        <eur>0.12</eur>
    </prices>
</anvil>

Using http://jsontoxml.utilities-online.info/

Ready to wear bridges

Work... most of the time
Do not like mixed content
Do not really address lexical restrictions
Results do not look native

Haute Couture Bridges

--Wikimedia

XML <-> JSON

<?xml version="1.0" encoding="UTF-8"?>
<anvil reference="acme-5103">
    <weight unit="pound">9.5</weight>
    <composition>best wrought iron</composition>
    <price currency="USD">.15</price>
</anvil>

{
  "anvil": {
    "reference": "acme-5103",
    "weight": {
      "unit": "pound",
      "value": 9.5
    },
    "composition": "best wrought iron",
    "prices": {
        "usd": 0.15,
        "eur": 0.12
    }
  }
}

Easy for a custom bridge (polyglotism required, see next use case)

Tailored Bridges

--Wikimedia

Schema driven

David Lee presenting JXSON at Balisage 2011

Case 2: Polyglotism

ACME is developing an application which gets anvil descriptions in XML and prices lists in JSON.

XML and JSON together

--Wikimedia

XML + JSON

Description and price of acme 5103?

<?xml version="1.0" encoding="UTF-8"?>
<anvils>
    <anvil reference="acme 5103">
        <weight unit="pound">9.5</weight>
        <composition>best wrought iron</composition>
    </anvil>
    <anvil reference="acme 5104">
        <weight unit="pound">15</weight>
        <composition>ultimate best wrought iron</composition>
    </anvil>
</anvils>

{
"acme 5103": 
    {
        "usd": 0.15,
        "eur": 0.12
    },
"acme 5104": 
    {
        "usd": 0.20,
        "eur": 0.17
    }
}

Polyglotism

JavaScript, E4X

var descriptions = <anvils>
    <anvil reference="acme 5103">
        <weight unit="pound">9.5</weight>
        <composition>best wrought iron</composition>
    </anvil>
    <anvil reference="acme 5104">
        <weight unit="pound">15</weight>
        <composition>ultimate best wrought iron</composition>
    </anvil>
</anvils>;

var prices = {
"acme 5103": 
    {
        "usd": 0.15,
        "eur": 0.12
    },
"acme 5104": 
    {
        "usd": 0.20,
        "eur": 0.17
    }
};

var anvil = "acme 5103";

print("Weight: " 
    + descriptions.anvil.(@reference == anvil).weight 
    + ", price: $" + prices[anvil].usd);

Python, ElementTree

import json

prices = json.loads('''
{
"acme 5103": 
    {
        "usd": 0.15,
        "eur": 0.12
    },
"acme 5104": 
    {
        "usd": 0.20,
        "eur": 0.17
    }
}''')

import xml.etree.ElementTree as ET

descriptions = ET.fromstring('''<?xml version="1.0" encoding="UTF-8"?>
<anvils>
    <anvil reference="acme 5103">
        <weight unit="pound">9.5</weight>
        <composition>best wrought iron</composition>
    </anvil>
    <anvil reference="acme 5104">
        <weight unit="pound">15</weight>
        <composition>ultimate best wrought iron</composition>
    </anvil>
</anvils>''')

anvil = "acme 5103"

print "Weight: " \
    + descriptions.findall("anvil[@reference='" + anvil + "']/weight")[0].text \
    + ", price: $%.2f" % prices[anvil]['usd']

Polyglotism?

As simple as...
... learning two data models

Case 3: Serialization

Acme software needs to store JSON documents in their big XML database. These are arbitrary JSON documents which can be tough to map into XML. There is still a need to query the corpus and storing them as plain text in a root element is not an option either.

XML to represent JSON

--Wikimedia

JSON (reminder)

{
  "anvil": {
    "reference": "acme-5103",
    "weight": {
      "unit": "pound",
      "value": 9.5
    },
    "composition": "best wrought iron",
    "prices": {
        "usd": 0.15,
        "eur": 0.12
    }
  }
}

Be prepared for a verbose next slide using χίμαιραλ (chimeral)

JSON serialized in XML

<?xml version="1.0" encoding="UTF-8"?>
<χ:data-model xmlns:χ="http://χίμαιραλ.com#">
   <χ:map>
      <χ:entry key="anvil" keyType="string">
         <χ:map>
            <χ:entry key="weight" keyType="string">
               <χ:map>
                  <χ:entry key="unit" keyType="string">
                     <χ:atomic-value type="string">pound</χ:atomic-value>
                  </χ:entry>
                  <χ:entry key="value" keyType="string">
                     <χ:atomic-value type="number">9.5</χ:atomic-value>
                  </χ:entry>
               </χ:map>
            </χ:entry>
            <χ:entry key="composition" keyType="string">
               <χ:atomic-value type="string">best wrought iron</χ:atomic-value>
            </χ:entry>
            <χ:entry key="prices" keyType="string">
               <χ:map>
                  <χ:entry key="eur" keyType="string">
                     <χ:atomic-value type="number">0.12</χ:atomic-value>
                  </χ:entry>
                  <χ:entry key="usd" keyType="string">
                     <χ:atomic-value type="number">0.15</χ:atomic-value>
                  </χ:entry>
               </χ:map>
            </χ:entry>
            <χ:entry key="reference" keyType="string">
               <χ:atomic-value type="string">acme-5103</χ:atomic-value>
            </χ:entry>
         </χ:map>
      </χ:entry>
   </χ:map>
</χ:data-model>

Serialization?

Verbose to read
Verbose to query
Still has its use cases

Case 4: JSON support in XSLT

JSON is a popular format for exchange of structured data on the web: it is specified in [JSON]. This section describes facilities allowing JSON data to be processed using XSLT.

--XSL Transformations (XSLT) Version 3.0 - Working Draft 10 July 2012

Convert JSON to maps, not XML

Mike Kay at XML Prague 2012

XSLT 3.0

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    exclude-result-prefixes="xs" version="3.0">

    <xsl:output method="text"/>

    <xsl:variable name="descriptions">
        <anvils>
            <anvil reference="acme 5103">
                <weight unit="pound">9.5</weight>
                <composition>best wrought iron</composition>
            </anvil>
            <anvil reference="acme 5104">
                <weight unit="pound">15</weight>
                <composition>ultimate best wrought iron</composition>
            </anvil>
        </anvils>
    </xsl:variable>

    <xsl:variable name="json-prices" as="xs:string"><![CDATA[
 {
"acme 5103": 
    {
        "usd": 0.15,
        "eur": 0.12
    },
"acme 5104": 
    {
        "usd": 0.20,
        "eur": 0.17
    }
}
    </xsl:variable>

    <xsl:variable name="prices" select="parse-json($json-prices)"/>

    <xsl:variable name="anvil">acme 5103</xsl:variable>

    <xsl:template match="/">
        <xsl:text>Weight: </xsl:text>
        <xsl:value-of select="$descriptions/anvils/anvil[@reference = $anvil]/weight"/>
        <xsl:text>, price: $</xsl:text>
        <xsl:value-of select="$prices($anvil)('usd')"/>
    </xsl:template>

</xsl:stylesheet>

JSON support in XSLT?

A polyglot approach.

Case 5: JSON support in XQuery

XQuery is well suited for hierarchical semi-structured data. Many implementations exist, and can easily be adapted to add JSON support.

--Jonathan Robie XML Prague 2012

Jonathan Robie

Jonathan Robie at XML Prague 2012

JSONiq

a small and simple set of extensions to XQuery that add support for JSON

--JSONiq

JSONiq?

Differs from XSLT 3.0
Polyglot approach

In time?

Case 6: JSON support in XForms

XForms allows the initialization, processing and serialization of instances whose data come from a JSON source by transforming the JSON value into an XML instance, and serializing it back out as JSON. The XML version of JSON has been designed to be round-trippable, and to allow XPath selectors that resemble the equivalent Javascript selectors.

--W3C XForms wiki

Treating JSON as a subset of XML

Steven Pemberton at XML Prague 2012

JSON support in XForms?

A bridge

Case 7: Consolidation

If XML is not as universal as it looked a few years ago – might we extend the language (no pun intended), restoring the universality?

--Hans-Jürgen Rennau Balisage 2012

A unified document language

Hans-Jürgen Rennau at Balisage 2012

Another approach?

Bridges are not perfect
Polyglotism is an ugly chimera
Serialisation is verbose

UDL

Elements get 2 new properties

model = map|sequence
key (only for children of maps)

UDL elements

An element can be either:

a conventional XML element
or a map
or an array

UDL?

Fromage ou dessert?

--Wikimedia

Fromage et dessert!

If we take the pain to update the XML data model we deserve "fromage et dessert"!

χίμαιραλ/superset

That's the goal of χίμαιραλ (chimeral) / superset

Limitations are removed so that an element can be used as both as:

a conventional element
and a map
and an array

χίμαιραλ/superset data model

Elements can be anonymous (JSON objects and arrays do not have names).
Attribute names can be QNames, but also strings, numbers, booleans and null (to be used as JSON object keys).
Attributes can have children attributes and nodes (to be used as JSON object values).