| Method | Description | Requirement in availableOutputs() | Scale |
|---|---|---|---|
| availableOutputs() | Returns a list that helps you understand which output methods you can call. The strings in this list can be Local, RDD, DataFrame, or PUL. | - | |
| json() | Returns the results as a tuple containing dicts, lists, strs, ints, floats, booleans, Nones. | Local | Sequence length below the materialization cap. The default is 200 but it can be increased in the RumbleDB configuration. |
| df() | Returns the results as a pyspark data frame | DataFrame (i.e., RumbleDB was able to infer an output schema) | No limitation, but beyond a billion items, you should use a Spark cluster. |
| pdf() | Returns the results as a pandas data frame | DataFrame (i.e., RumbleDB was able to infer an output schema) | Should fit in your computer's memory. |
| rdd() | Returns the results as an RDD containing dicts, lists, strs, ints, floats, booleans, Nones (experimental) | RDD | No limitation, but beyond a billion items, you should use a Spark cluster. |
| items() | Returns the results as a list containing Java Item objects that can be queried with the RumbleDB Item API. Will contain more information and more accurate typing. | Local | Sequence length below the materialization cap. The default is 200 but it can be increased in the RumbleDB configuration. |
| open(), hasNext(), nextJSON(), close() | Allows streaming (with no limitation of length) through individuals items as dicts, lists, strs, ints, floats, booleans, Nones. | Local | No limitation, as long as you go through the stream without saving all past items. |
| open(), hasNext(), next(), close() | Allows streaming (with no limitation of length) through individuals items as Java Item objects that can be queried with the RumbleDB Item API. Will contain more information and more accurate typing. | Local | No limitation, as long as you go through the stream without saving all past items. |
| applyPUL() | Persists the Pending Update List produced by the query (to the Delta Lake or a table registered in the Hive metastore). | PUL | - |
```python df.createTempView("myview") df2 = spark.sql("SELECT * FROM myview").toDF("name"); df2.show(); ``` A DataFrame output by Spark SQL can be reused as input to a JSONiq query. ```python rumble.bind('$b', df2); seq2 = rumble.jsoniq("for $i in 1 to 5 return $b"); df3 = seq2.df(); df3.show(); ``` And a DataFrame output by JSONiq can be reused as input to another JSONiq query. ```python rumble.bind('$b', df3); seq3 = rumble.jsoniq("$b[position() lt 3]"); df4 = seq3.df(); df4.show(); ``` # More advanced output retrieval methods This section shares more techniques for advanced users who want to make the most of RumbleDB in Python. ## RumbleDB Item API JSONiq has a rich type system, and the [conversion to JSON values](/writing-jsoniq-queries-in-python/type-mapping) can lose type information. An alternative consists of retrieving the sequence as a tuple of native items, which can be accessed with the [RumbleDB Item API](https://github.com/RumbleDB/rumble/blob/ccd9b91829510d7fbddf919d6d9ef7e45aa74905/src/main/java/org/rumbledb/api/Item.java). ```python list = res.items(); for result in list: print(result.getStringValue()) ``` ## Streaming through the items Sometimes, a sequence can be very long, and materializing it to a tuple of JSON values or a tuple of native items can fail because of the materialization cap. While it can be changed in the configuration to allow for larger tuples, this does not scale. Another way to retrieve a sequence of arbitrary length is to use the iterator API to stream through the items one by one. If you do not keep previous values in memory, there is no limit to the sequence size than can be retrieved in this way (but it may take more time than using RDDs or DataFrames, which benefit from parallelism). This is how to stream through the items converted to JSON one by one: ```python res.open(); while(res.hasNext()): print(res.nextJSON()); res.close(); ``` This is how to stream through the native items, using the Item API: ```python res.open(); while (res.hasNext()): print(res.next().getStringValue()); res.close(); ``` ## Getting unstructured output as an RDD Sometimes, it is not possible to retrieve the output sequence as a (pandas or pyspark) DataFrame because no schema could be inferred. This is notably the case if the output sequence is heterogeneous (such as a sequence of items mixing atomics, objects of various structures, arrays, etc). And materializing or streaming may not be an option either if there are billions of items. In this case, it is possible to obtain the output as an RDD instead. This gets an RDD of JSON values that can be processed by Python (using the [type mapping](/writing-jsoniq-queries-in-python/type-mapping)). The rdd() method is experimental because we had to reverse-engineer how pyspark encodes RDDs for the Java Virtual Machine (pickling). ```python rdd = res.rdd(); print(rdd.count()); for str in rdd.take(10): print(str); ``` # Writing queries directly in Jupyter notebook cells The Python edition of RumbleDB comes out of the box with a JSONiq magic. If you are in a Jupyter notebook and have installed the jsoniq pip package, you can activate the jsoniq magic with: ``` %load_ext jsoniqmagic ``` Then, you can run JSONiq in standalone cells and see the results: ``` %%jsoniq {"foobar":1} ``` Of course, you can still continue to use rumble.jsoniq() calls and process the outputs as you see fit. An example of the magic in action is available in our [upgraded online sandbox](https://colab.research.google.com/github/RumbleDB/rumble/blob/master/RumbleSandbox.ipynb). Note: This is a different magic than the magic that works with the RumbleDB HTTP server. It is more modern and running a server is no longer needed with this different magic. It suffices to install the jsoniq Python package. ## Change the behavior to output DataFrames By default, the output will be in the form of serialized JSON values. If the output is structured, then you can change this default behavior to show it in the form of a DataFrame instead. For a pandas DataFrame: ```notebook-python %%jsoniq -pdf for $i in 1 to 10000000 return { "foobar" : $i} ``` For a pyspark DataFrame: ``` %%jsoniq -df for $i in 1 to 10000000 return { "foobar" : $i} ``` Note that it will not work in all cases. If the output is not fully structured or RumbleDB is unable to infer a DataFrame schema, you can specify the schema yourself. The schema language is called JSound and you will find a [tutorial here](https://github.com/ghislainfourny/jsound-tutorial). ``` %%jsoniq -pdf declare type local:mytype as { "product" : "string", "store-number" : "int", "quantity" : "decimal" }; validate type local:mytype* { for $product in json-lines("http://rumbledb.org/samples/products-small.json", 10) where $product.quantity ge 995 return $product } ``` It is possible to measure the response time with the -t parameter: ``` %%jsoniq -t for $i in 1 to 10000000 return { "foobar" : $i} ``` # Advanced configuration ## RumbleDB's specific configuration It is possible to access RumbleDB's advanced configuration parameters with ``` conf = rumble.getRumbleConf() ``` Then, you can change the value of some parameters. For example, you can increase the number of JSON values that you can retrieve with a json() call: ``` conf.setResultSizeCap(1000) ``` You can also configure RumbleDB to output verbose information about the internal query plan, type and mode detection, and optimizations. This can be of interest to data engineers or researchers to understand how RumbleDB works. ``` conf.setPrintIteratorTree(True) ``` The complete API for configuring RumbleDB is accessible in our [JavaDoc](https://rumbledb.org/docs/latest/api/org/rumbledb/config/RumbleRuntimeConfiguration.html) pages. These methods are also callable in Python. Warning: some of the configuration methods do not make sense in Python and are specific to the command line edition of RumbleDB (such as setting the query content or an output path and input/output format). Also, setting external variables in Python should not be done via the configuration, but with the bind() and unbind() functions or extra parameters in jsoniq() calls. ## Allocating more memory If you get an out-of-memory error, it is possible to allocate memory when you build the Rumble session with a config() call. This is exactly the same way it is done when building a Spark session. The config() call can of course be used in combination with any other method calls that are part of the builder chain (withDelta(), appName(), config(), etc). {% hint style="info" %} This will only have an effect when done the first time the session is created. Spark and Java are not able to adjust the memory automatically after the session has been created. Subsequent calls of getOrCreate() do not create a new session, they only get the existing one. In Jupyter, you can restart the Kernel before you create the session, to force a new session. {% endhint %} For example: ``` rumble = RumbleSession.builder .config("spark.driver.memory", "10g") .getOrCreate() ``` # Write back to the disk (or data lake) Generally, it is possible to write output by to disk using the pandas DataFrame API, the pyspark DataFrame API, or Python's library to write JSON values to disk. For convenience, we provide a way to also directly do so with the sequence object output by the query. it is possible to write the output to a file locally or on a cluster. The API is similar to that of Spark dataframes. Note that it creates a directory and stores the (potentially very large) output in a sharded directory. RumbleDB was already tested with up to 64 AWS machines and 100s of TBs of data. Of course the examples below are so small that it makes more sense to process the results locally with Python, but this shows how GBs or TBs of data obtained from JSONiq can be written back to disk. ```python seq = rumble.jsoniq("$a.Name"); seq.write().mode("overwrite").json("outputjson"); seq.write().mode("overwrite").parquet("outputparquet"); seq = rumble.jsoniq("1+1"); seq.write().mode("overwrite").text("outputtext"); ``` # Data sources and formats RumbleDB is able to read a variety of formats from a variety of file systems and database management systems. We support functions to read JSON, JSON Lines, XML, Parquet, CSV, Text, ROOT, Delta files from various storage layers such as S3 and HDFS, Azure blob storage. We run most of our tests on Amazon EMR with S3 or HDFS, as well as locally on the local file system, but we welcome feedback on other setups. We also support some ETL-based systems such as PostgreSQL, MongoDB and the Hive metastore. ## Supported formats ### JSON A JSON file containing a single JSON object (or value) can be read with json-doc(). It will not spread access in any way, so that the files should be reasonably small. json-doc() can read JSON files even if the object or value is spread over multiple lines. ``` json-doc("file.json") ``` returns the (single) JSON value read from the supplied JSON file. This will also work for structures spread over multiple lines, as the read is local and not sharded. json-doc() also works with an HTTP URI. ### JSON Lines JSON Lines files are files that have one JSON object (or value) per line. Such files can thus become very large, up to billions or even trillions of JSON objects. JSON Lines files are read with the json-lines() function (formerly called json-file()). json-lines() exists in unary and binary. The first parameter specifies the JSON file (or set of JSON files) to read. The second, optional parameter specifies the minimum number of partitions. It is recommended to use it in a local setup, as the default is only one partition, which does not fully use the parallelism. If the input is on HDFS, then blocks are taken as splits by default. This is also similar to Spark's textFile(). json-lines() also works with an HTTP URI, however, it will download the file completely and then parallelize, because HTTP does not support blocks. As a consequence, it can only be used for reasonable sizes. Example of usage: ``` for $my-json in json-lines("hdfs://host:port/directory/file.json") where $my-json.property eq "some value" return $my-json ``` If a default host and port are set in the Hadoop configuration, you can directly specify an absolute path without host and port: ``` for $my-json in json-lines("/absolute/directory/file.json") where $my-json.property eq "some value" return $my-json ``` For a set of files: ``` for $my-json in json-lines("/absolute/directory/file-*.json") where $my-json.property eq "some value" return $my-json ``` If a working directory is set: ``` for $my-json in json-lines("file.json") where $my-json.property eq "some value" return $my-json ``` Several files or whole directories can be read with the same pattern syntax as in Spark. ``` for $my-json in json-lines("*.json") where $my-json.property eq "some value" return $my-json ``` In some cases, JSON Lines files are highly structured, meaning that all objects have the same fields and these fields are associated with values with the same types. In this case, RumbleDB will be faster navigating such files if you open them with the function structured-json-lines(). structured-json-lines() parses one or more json files that follow [JSON-lines](http://jsonlines.org/) format and returns a sequence of objects. This enables better performance with fully structured data and is recommended to use only when such data is available. Warning: when the data has multiple types for the same field, this field and contained values will be treated as strings. This is also similar to Spark's spark.read.json(). Example of usage: ``` for $my-structured-json in structured-json-lines("hdfs://host:port/directory/structured-file.json") where $my-structured-json.property eq "some value" return $my-structured-json ``` ### XML XML files can be read into RumbleDB using the doc() function. The parameter specifies the XML file to read and return as a document node. Example of usage: ``` doc("path/to/file.xml") ``` Additionally, RumbeDB provides the xml-files() function to read many XML files at once. xml-files() exists in unary and binary. The first parameter specifies the directory of XML files to read. The second, optional parameter specifies the minimum number of partitions. It is recommended to use it in a local setup, as the default is only one partition. Example of usage: ``` xml-files("path/to/directory/*.xml", 10) ``` ### Text Text files can be read into a sequence of string items, one string per line. RumbleDB can open files that have billions or potentially even trillions of lines with the function text-file(). text-file() exists in unary and binary. The first parameter specifies the text file (or set of text files) to read and return as a sequence of strings. The second, optional parameter specifies the minimum number of partitions. It is recommended to use it in a local setup, as the default is only one partition, which does not fully use the parallelism. If the input is on HDFS, then blocks are taken as splits by default. This is also similar to Spark's textFile(). Example of usage: ``` count( for $my-string in text-file("hdfs://host:port/directory/file.txt") for $token in tokenize($my-string, ";") where $token eq "some value" return $token ) ``` Several files or whole directories can be read with the same pattern syntax as in Spark. (Also see examples for json-lines for host and port, sets of files and working directory). There is also a function local-text-file() that reads locally, without parallelism. RumbleDB can stream through the file efficiently. ``` count( for $my-string in local-text-file("file:///home/me/file.txt") for $token in tokenize($my-string, ";") where $token eq "some value" return $token ) ``` RumbleDB supports also the W3C-standard functions unparsed-text and unparsed-text-lines. The output of the latter is automatically parallelized as a potentially large sequence of strings. ``` count( for $my-string in unparsed-text-lines("file:///home/me/file.txt") for $token in tokenize($my-string, ";") where $token eq "some value" return $token ) ``` ``` count( let $text := unparsed-text("file:///home/me/file.txt") for $my-string in tokenize($text, "\n") for $token in tokenize($my-string, ";") where $token eq "some value" return $token ) ``` ### Parquet Parquet files can be opened with the function parquet-file(). Parses one or more parquet files and returns a sequence of objects. This is also similar to Spark's spark.read.parquet() ``` for $my-object in parquet-file("file.parquet") where $my-object.property eq "some value" return $my-json ``` Several files or whole directories can be read with the same pattern syntax as in Spark. ``` for $my-object in parquet-file("*.parquet") where $my-object.property eq "some value" return $my-json ``` ### CSV CSV files can be opened with the function csv-file(). Parses one or more csv files and returns a sequence of objects. This is also similar to Spark's spark.read.csv() ``` for $i in csv-file("file.csv") where $i._c0 eq "some value" return $i ``` Several files or whole directories can be read with the same pattern syntax as in Spark. ``` for $i in csv-file("*.csv") where $i._c0 eq "some value" return $i ``` Options can be given in the form of a JSON object. All available options can be found in the [Spark documentation](https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameReader.html#csv-java.lang.String...-) ``` for $i in csv-file("file.csv", {"header": true, "inferSchema": true}) where $i.key eq "some value" return $i ``` ### PostgreSQL {% hint style="info" %} This functionality is currently only available in the Python edition (pip install jsoniq) as of 2.0.1+. {% endhint %} PostgreSQL tables can be opened with the function postgresql-table(). PostgreSQL is an OLTP system with its own storage system. Thus, unlike most other functions on this page, it uses a connection string rather than a path on a data lake. It opens one table and returns it as a sequence of objects. The first argument is the connection string in the JDBC format, containing host, port, username, password, and database. The second argument is the name of the table to read. ``` for $i in postgresql-table("jdbc:postgresql://servername/dbname?user=postgres&password=example", "tablename") where $i.attribute eq "some value" return $i ``` The third parameter can be used to control the number of partitions. ``` for $i in postgresql-table("jdbc:postgresql://servername/dbname?user=postgres&password=example", "tablename", 10) where $i.attribute eq "some value" return $i ``` ### MongoDB {% hint style="info" %} This functionality is currently only available in the Python edition (pip install jsoniq) as of 2.0.2+. {% endhint %} MongoDB collections can be opened with the function mongodb-collection(). MongoDB is an OLTP system with its own storage system. Thus, unlike most other functions on this page, it uses a connection string rather than a path on a data lake. It opens one collection and returns it as a sequence of objects. The first argument is the connection string in the MongoDB format, containing host, port, database, collection, username, password. The second argument is the name of the collection to read. ``` for $i in mongodb-collection("mongodb://servername/dbname", "collection") where $i.attribute eq "some value" return $i ``` The third parameter can be used to control the number of partitions. ``` for $i in mongodb-collection("mongodb://servername/dbname", "collection", 10) where $i.attribute eq "some value" return $i ``` MongoDB does not work "out of the box" but requires some configuration as indicated on the MongoDB Spark connector website. In the Python edition, we simplified the process and all that is needed is to add withMongo() on the session building chain: ```python RumbleSession.builder.withMongo().getOrCreate(); ``` ### Hive metastore RumbleDB can connect to a table registered in the Hive metastore with the function table(). The Hive metastore manages its own storage system. Thus, unlike most other functions on this page, it uses a simple name rather than a path on a data lake. ``` for $i in table("mytable") where $i.attribute eq "some value" return $i ``` RumbleDB can also modify data in a Hive metastore table with the [JSONiq Update Facility](/the-jsoniq-language/jsoniq-update-facility). ### Delta files Delta files, part of the [Delta Lake framework](https://delta.io/), can be opened with the function delta-file(). ``` for $i in delta-file("hdfs://path/to/my/delta-file") where $i.attribute eq "some value" return $i ``` RumbleDB can also modify data in a delta file with the [JSONiq Update Facility](/the-jsoniq-language/jsoniq-update-facility). Delta files do not work "out of the box" but require some configuration as indicated on the Delta Lake website (importing packages, configuring some parameters). In the Python edition, we simplified the process and all that is needed is to add withDelta() on the session building chain: ```python RumbleSession.builder.withDelta().getOrCreate(); ``` ### AVRO Avro files can be opened with the function avro-file(). Parses one or more avro files and returns a sequence of objects. This is similar to Spark's `spark.read().format("avro").load()` ``` for $i in avro-file("file.avro") where $i._col1 eq "some value" return $i ``` Several files or whole directories can be read with the same pattern syntax as in Spark. ``` for $i in avro-file("*.avro") where $i._col1 eq "some value" return $i ``` Options can be given in the form of a JSON object. All available options relevant for reading in avro data can be found in the [Spark documentation](https://spark.apache.org/docs/latest/sql-data-sources-avro.html#data-source-option) ``` for $i in avro-file("file.avro", {"ignoreExtension": true, "avroSchema": "/path/to/schema.avsc"}) where $i._col1 eq "some value" return $i ``` ### libSVM libSVM files can be opened with the function libsvm-file(). Parses one or more libsvm files and returns a sequence of objects. This is similar to Spark's `spark.read().format("libsvm").load()` ``` for $i in libsvm-file("file.txt") where $i._col1 eq "some value" return $i ``` Several files or whole directories can be read with the same pattern syntax as in Spark. ``` for $i in libsvm-file("*.txt") where $i._col1 eq "some value" return $i ``` ### ROOT ROOT files can be open with the function root-file(). The second parameter specifies the path within the ROOT files (a ROOT file is like a mini-file system of its own). It is often `Events` or `tree`. ``` for $i in root-file("events.root", "Events") where $i._c0 eq "some value" return $i ``` ## Creating your own big sequence The function parallelize() can be used to create, on the fly, a big sequence of items in such a way that RumbleDB can spread its querying across cores and machines. This function behaves like the Spark parallelize() you are familiar with and sends a large sequence to the cluster. The rest of the FLWOR expression is then evaluated with Spark transformations on the cluster. ``` for $i in parallelize(1 to 1000000) where $i mod 1000 eq 0 return $i ``` There is also be a second, optional parameter that specifies the minimum number of partitions. ``` for $i in parallelize(1 to 1000000, 100) where $i mod 1000 eq 0 return $i ``` ## Supported file systems As a general rule of thumb, RumbleDB can read from any file system that Spark can read from. The file system is inferred from the scheme used in the path used in any of the functions described above, with the exception of MongoDB, the Hive metastore, and PostgreSQL, which are ETL-based. Note that the scheme is optional, in which case the default file system as configured in Hadoop and Spark is used. A relative path can also be provided, in which case the working directory (including its file system) as configured is used. ### Local file system The scheme for the local file system is `file://`. Pay attention to the fact that for reading an absolute path, a third slash will follow the scheme. Example: ``` file:///home/user/file.json ``` Warning! If you try to open a file from the local file system on a cluster of several machines, this might fail as the file is only on the machine that you are connected to. You need to pass additional parameters to `spark-submit` to make sure that any files read locally will be copied over to all machines. If you use `spark-submit` locally, however, this will work out of the box, but we recommend specifying a number of partitions to avoid reading the file as a single partition. For Windows, you need to use forward slashes, and if the local file system is set up as the default and you omit the file scheme, you still need a forward slash in front of the drive letter to not confuse it with a URI scheme: ``` file:///C:/Users/hadoop/file.json file:/C:/Users/hadoop/file.json /C:/Users/hadoop/file.json ``` In particular, the following will *not* work: ``` file://C:/Users/hadoop/file.json C:/Users/hadoop/file.json C:\Users\hadoop\file.json file://C:\Users\hadoop\file.json ``` ### HDFS The scheme for the Hadoop Distributed File System is `hdfs://`. A host and port should also be specified, as this is required by Hadoop. Example: ``` hdfs://www.example.com:8021/user/hadoop/file.json ``` If HDFS is already set up as the default file system as is often the case in managed Spark clusters, an absolute path suffices: ``` /user/hadoop/file.json ``` The following will *not* work: ``` hdfs:///user/hadoop/file.json hdfs://user/hadoop/file.json hdfs:/user/hadoop/file.json ``` ### S3 There are three schemes for reading from S3: `s3://`, `s3n://` and `s3a://`. Examples: ``` s3://my-bucket/directory/file.json s3n://my-bucket/directory/file.json s3a://my-bucket/directory/file.json ``` If you are on an Amazon EMR cluster, `s3://` is straightforward to use and will automatically authenticate. For more details on how to set up your environment to read from S3 and which scheme is most appropriate, we refer to the Amazon S3 documentation. ### Azure blob storage The scheme for Azure blob storage is `wasb://`. Example: ``` wasb://mycontainer@myaccount.blob.core.windows.net/directory/file.json ``` # The JSONiq language JSONiq is a query and processing language specifically designed for the popular JSON data model. The main ideas behind JSONiq are based on lessons learned in more than 30 years of relational query systems and more than 15 years of experience with designing and implementing query languages for semi-structured data like XML and RDF. The main source of inspiration behind JSONiq is XQuery, which has been proven so far a successful and productive query language for semi-structured data (in particular XML). JSONiq borrowed a large numbers of ideas from XQuery, like the structure and semantics of a FLWOR construct, the functional aspect of the language, the semantics of comparisons in the face of data heterogeneity, the declarative, snapshot-based updates. However, unlike XQuery, JSON is not concerned with the peculiarities of XML, like mixed content, ordered children, the confusion between attributes and elements, the complexities of namespaces and QNames, or the complexities of XML Schema, and so on. The power of the XQuery's FLWOR construct and the functional aspect, combined with the simplicity of the JSON data model result in a clean, sleek and easy to understand data processing language. As a matter of fact, JSONiq is a language that can do more than queries: it can describe powerful data processing programs, from transformations, selections, joins of heterogeneous data sets, data enrichment, information extraction, information cleaning, and so on. Technically, the main characteristics of JSONiq (and XQuery) are the following: * It is a *set-oriented language*. While most programming languages are designed to manipulate one object at a time, JSONiq is designed to process sets (actually, sequences) of data objects. * It is a *functional language*. A JSONiq program is an expression; the result of the program is the result of the evaluation of the expression. Expressions have fundamental role in the language: every language construct is an expression, and expressions are fully composable. * It is a *declarative language*. A program specifies what is the result being calculated, and does not specify low level algorithms like the sort algorithm, the fact that an algorithm is executed in main memory or is external, on a single machine or parallelized on several machines, or what access patterns (aka indexes) are being used during the evaluation of the program. Such implementation decisions should be taken automatically, by an optimizer, based on the physical characteristics of the data, and of the hardware environment. Just like a traditional database would do. The language has been designed from day one with optimizability in mind. * It is designed for *nested, heterogeneous, semi-structured data*. Data structures in JSON can be nested with arbitrary depth, do not have a specific type pattern (i.e. are heterogeneous), and may or may not have one or more schemas that describe the data. Even in the case of a schema, such a schema can be open, and/or simply partially describe the data. Unlike SQL, which is designed to query tabular, flat, homogeneous structures. JSONiq has been designed from scratch as a query for nested and heterogeneous data. # JSONiq 1.0 JSONiq 1.0 is the first version of the JSONiq language, currently in use. It is a cousin of the XQuery 3.0 language and was developed by W3C XML Query Working Group members as a proposal of how to integrate JSON support into the language, while making it appealing to the JSON community, and making it easy for an existing XQuery engine to implement. # Introduction In this specification, we detail the JSONiq language in version 1.0. Historically, JSONiq was first created as an extension to XQuery. Later, a separate core syntax was created which makes it 100% tailored for JSON. It is the JSONiq core syntax that is detailed in this document. The functionality directly inherited from XQuery is described on a higher level and we explicitly refer for more in-depth details to the [W3C specification](https://www.w3.org/TR/xquery-30). ### Structure of a JSONiq program. A JSONiq program can either be a main module, which contains a query that can be executed, or a library module, which defines functions and variables that can be used in other modules. A main or library module can be optionally prefixed with a JSONiq declaration with a version (currently 1.0) and an encoding. Module  ### Main modules A JSONiq main module is made of two parts: an optional prolog, and an expression, which is the main query. MainModule  The result of the main JSONiq program is the result of its main query. In the prolog, it is possible to declare global variables and functions. Mostly, you will recognize a prolog declaration by the semi-colon it ends with. The main query does not contain semi-colons (at least in core JSONiq). Global variables and functions can use and call each other arbitrarily, even if the dependency is further down in the prolog. If there a cycle, an error is thrown. JSONiq largely follows the W3C standard regarding modules. The detailed specification is found [here](https://www.w3.org/TR/xquery-30/#id-query-prolog). ### Library modules Library modules do not contain any main query, just global variables and functions. They can be imported by other modules. A library module is introduced with a module declaration, followed by the prolog containing its variables and functions. LibraryModule  ### Feature matrix JSONiq is 99% reliant on XQuery, a W3C standard. For everything taken over from the W3C standard, a brief, non-normative explanation is provided with a link to the corresponding part in the W3C specification. | Feature | Specification status | | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | JSONiq Data Model | | | Atomic items | W3C-conformant | | Structured items | JSONiq-specific | | Function items | W3C-conformant | | Node items (XML) | Omitted (optional support by some engines) | | JSONiq Type System | | | Atomic types | W3C-conformant, but support for xs:ID, xs:IDREF, xs:IDREFS, xs:Name, xs:NCName, xs:ENTITY, xs:ENTITIES, xs:NOTATION omitted (except for engines also supporting XML) | | js:null type | JSONiq-specific | | js:item, js:atomic types | JSONiq-specific synonyms for item() and xs:anyAtomicType | | Structured types | JSONiq-specific | | Function types | W3C-conformant | | Empty sequence type | JSONiq-specific notation () for empty-sequence() | | XML node types | Omitted (optional support by engines supporting XML) | | Concepts | | | Effective boolean value | W3C-conformant, extended with object, array and null semantics | | Atomization | Omitted (optional support by engines supporting XML) | | Expressions | | | Numeric literals | W3C-conformant | | String literals | W3C-conformant, but escape is done with \ not with & | | Boolean and null literals | JSONiq-specific | | Variable reference | W3C-conformant | | Parenthesized expressions | W3C-conformant | | Context item expressions | W3C-conformant but $$ syntax instead of . | | Static function calls | W3C-conformant | | Named function reference | W3C-conformant | | Inline function expressions | W3C-conformant | | Filter expressions | W3C-conformant | | Dynamic function calls | W3C-conformant | | Path expressions (XML) | Omitted (optional support by engines supporting XML, but relative paths must start with ./) | | Object lookup | JSONiq-specific | | Array lookup | JSONiq-specific | | Array unboxing | JSONiq-specific | | Sequence expressions | W3C-conformant | | Arithmetic expressions | W3C-conformant, no atomization needed (except for engines also supporting XML) | | String concatenation expressions | W3C-conformant | | Comparison expressions | W3C-conformant, no need to atomize or convert from untyped and untypedAtomic (except for engines also supporting XML) | | Logical expressions | W3C-conformant | | XML constructors | Omitted (optional support by engines supporting XML) | | JSON (object and array) constructors | JSONiq-specific | | FLWOR expressions | W3C-conformant | | Unordered and ordered expressions | W3C-conformant | | Conditional expressions | W3C-conformant | | Switch expressions | W3C-conformant | | Quantified expressions | W3C-conformant | | Try-catch expressions | W3C-conformant | | Instance-of expressions | W3C-conformant | | Typeswitch expressions | W3C-conformant | | Cast expressions | W3C-conformant | | Castable expressions | W3C-conformant | | Constructor functions | W3C-conformant, additional constructor function for null() | | Treat expressions | W3C-conformant | | Simple map operator | W3C-conformant | | Validate expressions | Omitted (optional support by engines supporting XML) | | Extension expressions | W3C-conformant | | Static context | | | XPath 1.0 compatibility mode | Omitted (optional support by engines supporting XML) | | Statically known namespaces | W3C-conformant | | Default element/type namespace | W3C-conformant, strong recommendation for implementations to overwrite with the proxy namespace *
# JSON update primitives A Pending Update List is an unordered list of update primitives. Update primitives are internal and do not appear in the syntax. Each kind of update primitive models one individual update to an object or an array. A Pending Update List can by analogy be seen as the diff between two git revisions, and a single update primitive can be seen, with this same analogy, as the difference between two single lines of code. Thus, the JSONiq Update Facility is to trees what git is to lines of text: a "tree diff" language. JSONiq adds the following new update primitives, specific to JSON. They are similar to those defined by the XQuery Update Facility for XML. Update primitives within a PUL are applied with strict snapshot semantics. For examples, the positions are resolved against the array before the updates. Names are resolved on the object before the updates. ### Update primitive for objects and arrays (in collections or in memory)
| Update primitive | Description |
|---|---|
jupd:insert-into-object( $target as object(), $content as object()) | Inserts all pairs of the object $content into the object $target. |
jupd:insert-into-array( $target as array(), $position as xs:integer, $content as item()*) | Inserts all items in the sequence $content before position $position into the array $target. |
jupd:delete-from-object( $target as object(), $keys as xs:string*) | Removes the pairs the names of which appear in $keys from the object $target. |
jupd:delete-from-array( $target as array(), $position as xs:integer) | Removes the item at position $position from the array $target (causes all following items in the array to move one position to the left). |
jupd:replace-in-array( $target as array(), $position as xs:integer, $content as item()) | Replaces the item at position $position in the array $target with the item $content (do nothing if $position is not comprised between 1 and jdm:size($target)). |
jupd:replace-in-object( $target as object(), $key as xs:string, $content as item()) | Replaces the value of the pair named $key in the object $target with the item $content (do nothing if there is no such pair). |
jupd:rename-in-object( $target as object(), $key as xs:string, $content as xs:string) | Renames the pair originally named $key in the object $target as $content (do nothing if there is no such pair). |
| Update primitive | Description |
|---|---|
jupd:create-collection( $name as string, $mode as string, $content as item()*) | Creates a collection initialized with the provided items. Mode determines the kind of collection (e.g., a Hive metastore table, a delta lake file, etc). |
jupd:truncate-collection( $name as string, $mode as string) | Deletes the specified collection. |
jupd:edit( $target as item(), $content as item()) | Modifies an item in a collection into another item, preserving its identity and location. |
jupd:delete-in-collection( $target as item()) | Deletes the provided item from its collection. |
jupd:insert-first-into-collection( $name as string, $mode as string, $content as item()*) | Inserts the provided items at the very beginning of the specified collection. |
jupd:insert-last-into-collection( $name as string, $mode as string, $content as item()*) | Inserts the provided items at the very end of the specified collection. |
jupd:insert-before-into-collection( $target as item, $content as item()*) | Inserts the provided items before the specified item in its collection. |
jupd:insert-last-into-collection( $target as item, $content as item()*) | Inserts the provided items after the specified item in its collection. |
# Merging updates In the middle of a program, several PULs can be produced against the same snapshot. They are then merged with upd:mergeUpdates (part of the XQuery Update Facility standard), which is extended as follows. * Several deletes on the same object are replaced with a unique delete on that object, with a list of all selectors (names) to be deleted, where duplicates have been eliminated. * Several deletes on the same array and selector (position) are replaced with a unique delete on that array and with that selector. * Several inserts on the same array and selector (position) are equivalent to a unique insert on that array and selector with the content of those original inserts appended in an implementation-dependent order (like XQUF). * Several inserts on the same object are equivalent to a unique insert where the objects containing the pairs to insert are merged. An error jerr:JNUP0005 is raised if a collision occurs. * Several replaces on the same object or array and with the same selector raise an error jerr:JNUP0009. * Several renames on the same object and with the same selector raise an error jerr:JNUP0010. * If there is a replace and a delete on the same object or array and with the same selector, the replace is omitted in the merged PUL. * If there is a rename and a delete on the same object or array and with the same selector, the rename is omitted in the merged PUL.
# Applying updates At the end of an updating program, the resulting PUL is applied with upd:applyUpdates (part of the XQuery Update Facility standard), which is extended as follows: * First, before applying any update, each update primitive (except the jupd:insert-into-object primitives, which do not have any target) locks onto its target by resolving the selector on the object or array it updates. If the selector is resolved to the empty sequence, the update primitive is ignored in step 2. After this operation, each of these update primitives will contain a reference to either the pair (for an object) or the value (for an array) on or relatively to which it operates. * Then each update primitive is applied, using the target references that were resolved at step 1. The order in which they are applied is not relevant and does not affect the final instance of the data model. After applying all updates, an error jerr:JNUP0006 is raised upon pair name collision within the same object. # The transform expression
In the remainder of this chapter, we showcase the individual updating expressions one by one, inside a copy-modify-return expression. Update expressions can also appear outside of a copy-modify-return expression, in which case they propagate and/or persist directly to their targets, to the extent that the context makes it meaningful and possible.
# Primary updating expressions Update expressions are the visible part of JSONiq Updates in the language. Each primary updating expression contributes an update primitive to the Pending Update List being built. ## Nested updates (memory or persistent) These expressions may appear in a copy-modify-return (transform) expression (for in-memory updates on cloned values), or outside (for persistent updates to an underlying storage). ### **Inserting values into an object or array**
### **Renaming a key**
### **Replacing a value in an object or array.**
