If you really want to start writing queries right now, there is a public sandbox here that will just work and guide you. You only need to have a Google account to be able to execute them, as this exposes our Jupyter notebook via the Colab environment. You are also free to download and use this notebook with any other provider or even your own local Jupyter and it will work just the same: the queries are all shipped to our own, small public backend no matter what. However, this may require a bit of configuration (JAVA_HOME pointing to Java 17 or 21, and if you have conflicting Spark installations in addition to pyspark, SPARK_HOME pointing to a Spark 4.0 installation).

If you do not have a Google account, you can also use our simpler sandbox page without Jupyter, here where you can type small queries and see the results.

With the sandboxes above, you can only inline your data in the query or access a dataset with an HTTP URL.

Once you want to take it to the next level and query your own data on your laptop, you will find instructions below to use RumbleDB on your own computer manually, which among others will allow you to query any files stored on your local disk. And then, you can take a leap of faith and use RumbleDB on a large cluster (Amazon EMR, your company's cluster, etc).

It is also possible to use RumbleDB with brew, however there is currently no way to adjust memory usage. To install RumbleDB with brew, type the commands:

You can test that it works with:

Then, launch a JSONiq shell with:

The RumbleDB shell appears:

You can now start typing simple queries like the following few examples. Press three times the return key to execute a query.

Any expression in JSONiq returns a sequence of items. Any variable in JSONiq is bound to a sequence of items. Items can be objects, arrays, or atomic values (strings, integers, booleans, nulls, dates, binary, durations, doubles, decimal numbers, etc). A sequence of items can be a sequence of just one item, but it can also be empty, or it can be as large as to contain millions, billions or even trillions of items. Obviously, for sequence longer than a billion items, it is a better idea to use a cluster than a laptop. A relational table (or more generally a data frame) corresponds to a sequence of object items sharing the same schema. However, sequences of items are more general than tables or data frames and support heterogeneity seamlessly.

When passing Python values to JSONiq or getting them from a JSONiq queries, the mapping to and from Python is as follows:

You can use RumbleDB from within Python programmes by running

Java version

Important note: since the jsoniq package depends on pyspark 4, Java 17 or Java 21 is a requirement. If another version of Java is installed, the execution of a Python program attempting to create a RumbleSession will lead to an error message on stderr that contains explanations.

You can control your Java version with:

Information about how this package is used can be found .

RumbleDB can also be used as a maven dependency. You can find it here.

The JavaDoc documentation is accessible here.

JSONiq 1.0 is the first version of the JSONiq language, currently in use.

It is a cousin of the XQuery 3.0 language and was developed by W3C XML Query Working Group members as a proposal of how to integrate JSON support into the language, while making it appealing to the JSON community, and making it easy for an existing XQuery engine to implement.

You can use RumbleDB from within Python programmes by running

Java version

Important note: since the jsoniq package depends on pyspark 4, Java 17 or Java 21 is a requirement. If another version of Java is installed, the execution of a Python program attempting to create a RumbleSession will lead to an error message on stderr that contains explanations.

You can control your Java version with:

Information about how this package is used can be found in this section.

Common issue: colliding Spark version

Some advanced users who have already configured a Spark installation on their machine may encounter a version issue if SPARK_HOME points to this alternate installation, and it is a different version of Spark (e.g., 3.5 or 3.4). The jsoniq package requires Spark 4.0.

If this happens, RumbleDB should output an informative error message. They are two ways to fix such conflicts:

• The easiest is remove the SPARK_HOME environment variable completely. This will have RumbleDB fall back to the Spark 4.0 installation that ships with its pyspark dependency.

• Or you can instead change the value of SPARK_HOME to point to a Spark 4.0 installation, if you have one. This would be for more advanced users who know what they are doing.

If you have another working Spark installation on your machine, you can see which version it is with

The above command is of course expected not to work for first-time users who only installed the jsoniq package and never installed Spark additionally on their machine.

High-level information on the library

A RumbleSession is a wrapper around a SparkSession that additionally makes sure the RumbleDB environment is in scope.

JSONiq queries are invoked with rumble.jsoniq() in a way similar to the way Spark SQL queries are invoked with spark.sql().

JSONiq variables can be bound to lists of JSON values (str, int, float, True, False, None, dict, list) or to Pyspark DataFrames. A JSONiq query can use as many variables as needed (for example, it can join between different collections).

It will later also be possible to read tables registered in the Hive metastore, similar to spark.sql(). Alternatively, the JSONiq query can also read many files of many different formats from many places (local drive, HTTP, S3, HDFS, ...) directly with simple such as json-lines(), text-file(), parquet-file(), csv-file(), etc.

The resulting sequence of items can be retrieved as a list of JSON values, as a Pyspark DataFrame, or, for advanced users, as an RDD or with a streaming iteration over the items using the .

It is also possible to write the sequence of items to the local disk, to HDFS, to S3, etc in a way similar to how DataFrames are written back by Pyspark.

The design goal is that it is possible to chain DataFrames between JSONiq and Spark SQL queries seamlessly. For example, JSONiq can be used to clean up very messy data and turn it into a clean DataFrame, which can then be processed with Spark SQL, spark.ml, etc.

Any feedback or error reports are very welcome.

There are many ways to install and use RumbleDB. For example:

• By simply using one of our online sandboxes (Jupyter notebook or simple sandbox page)

• Our newest library: by installing a pip package (pip install jsoniq)

• By running the standalone RumbleDB jar with Java on your laptop

• By installing with homebrew

• By installing Spark yourself on your laptop (for more control on Spark parameters) and use a small RumbleDB jar with spark-submit

• By using our docker image on your laptop (go to the "Run with docker" section on the left menu)

• By uploading the small RumbleDB jar to an existing Spark cluster (such as AWS EMR)

• By running RumbleDB as an HTTP server in the background and connecting to it in a Jupyter notebook with the %%jsoniq magic.

• By installing it manually on your machine.

Further steps

After installing RumbleDB, further steps could involve:

• Learning JSONiq. More details can be found in the JSONiq section of this documentation and in the and .

• Storing some data on S3, creating a Spark cluster on Amazon EMR (or Azure blob storage and Azure, etc), and querying the data with RumbleDB. More details are found in the cluster section of this documentation.

• Using RumbleDB with Jupyter notebooks. For this, you can run RumbleDB as a server with a simple command, and get started by downloading the and just clicking your way through it. More details are found in the Jupyter notebook section of this documentation. Jupyter notebooks work both locally and on a cluster.

Java version (important)

You need to make sure that you have Java 11 or 17 and that, if you have several versions installed, JAVA_HOME correctly points to Java 11 or 17.

RumbleDB works with both Java 11 and Java 17. You can check the Java version that is configured on your machine with:

If you do not have Java, you can download version 11 or 17 from .

Do make sure it is not Java 8, which will not work.

The Python edition of Rumble can be used to directly write JSONiq queries in Jupyter notebook cells. This is explained . You first need to install the library as described .

JSONiq is a query and processing language specifically designed for the popular JSON data model. The main ideas behind JSONiq are based on lessons learned in more than 30 years of relational query systems and more than 15 years of experience with designing and implementing query languages for semi-structured data like XML and RDF.

The main source of inspiration behind JSONiq is XQuery, which has been proven so far a successful and productive query language for semi-structured data (in particular XML). JSONiq borrowed a large numbers of ideas from XQuery, like the structure and semantics of a FLWOR construct, the functional aspect of the language, the semantics of comparisons in the face of data heterogeneity, the declarative, snapshot-based updates. However, unlike XQuery, JSON is not concerned with the peculiarities of XML, like mixed content, ordered children, the confusion between attributes and elements, the complexities of namespaces and QNames, or the complexities of XML Schema, and so on.

The power of the XQuery's FLWOR construct and the functional aspect, combined with the simplicity of the JSON data model result in a clean, sleek and easy to understand data processing language. As a matter of fact, JSONiq is a language that can do more than queries: it can describe powerful data processing programs, from transformations, selections, joins of heterogeneous data sets, data enrichment, information extraction, information cleaning, and so on.

Technically, the main characteristics of JSONiq (and XQuery) are the following:

• It is a set-oriented language. While most programming languages are designed to manipulate one object at a time, JSONiq is designed to process sets (actually, sequences) of data objects.

• It is a functional language. A JSONiq program is an expression; the result of the program is the result of the evaluation of the expression. Expressions have fundamental role in the language: every language construct is an expression, and expressions are fully composable.

• It is a declarative language. A program specifies what is the result being calculated, and does not specify low level algorithms like the sort algorithm, the fact that an algorithm is executed in main memory or is external, on a single machine or parallelized on several machines, or what access patterns (aka indexes) are being used during the evaluation of the program. Such implementation decisions should be taken automatically, by an optimizer, based on the physical characteristics of the data, and of the hardware environment. Just like a traditional database would do. The language has been designed from day one with optimizability in mind.

• It is designed for nested, heterogeneous, semi-structured data. Data structures in JSON can be nested with arbitrary depth, do not have a specific type pattern (i.e. are heterogeneous), and may or may not have one or more schemas that describe the data. Even in the case of a schema, such a schema can be open, and/or simply partially describe the data. Unlike SQL, which is designed to query tabular, flat, homogeneous structures. JSONiq has been designed from scratch as a query for nested and heterogeneous data.

RumbleDB is a querying engine that allows you to query your large, messy datasets with ease and productivity. It covers the entire data pipeline: clean up, structure, normalize, validate, convert to an efficient binary format, and feed it right into Machine Learning estimators and models, all within the JSONiq language.

RumbleDB supports JSON-like datasets including JSON, JSON Lines, Parquet, Avro, SVM, CSV, ROOT as well as text files, of any size from kB to at least the two-digit TB range (we have not found the limit yet).

RumbleDB is both good at handling small amounts of data on your laptop (in which case it simply runs locally and efficiently in a single-thread) as well as large amounts of data by spreading computations on your laptop cores, or onto a large cluster (in which case it leverages Spark automagically).

RumbleDB can also be used to easily and efficiently convert data from a format to another, including from JSON to Parquet thanks to JSound validation.

It runs on many local or distributed filesystems such as HDFS, S3, Azure blob storage, and HTTP (read-only), and of course your local drive as well. You can use any of these file systems to store your datasets, but also to store and share your queries and functions as library modules with other users, worldwide or within your institution, who can import them with just one line of code. You can also output the results of your query or the log to these filesystems (as long as you have write access).

With RumbleDB, queries can be written in the tailor-made and expressive JSONiq language. Users can write their queries declaratively and start with just a few lines. No need for complex JSON parsing machinery as JSONiq supports the JSON data model natively.

The core of RumbleDB lies in JSONiq's FLWOR expressions, the semantics of which map beautifully to DataFrames and Spark SQL. Likewise expression semantics is seamlessly translated to transformations on RDDs or DataFrames, depending on whether a structure is recognized or not. Transformations are not exposed as function calls, but are completely hidden behind JSONiq queries, giving the user the simplicity of an SQL-like language and the flexibility needed to query heterogeneous, tree-like data that does not fit in DataFrames.

This documentation provides you with instructions on how to get started, examples of data sets and queries that can be executed locally or on a cluster, links to JSONiq reference and tutorials, notes on the function library implemented so far, and instructions on how to compile RumbleDB from scratch.

Please note that this is a (maturing) beta version. We welcome bug reports in the GitHub issues section.

At the end of an updating program, the resulting PUL is applied with upd:applyUpdates (part of the XQuery Update Facility standard), which is extended as follows:

• First, before applying any update, each update primitive (except the jupd:insert-into-object primitives, which do not have any target) locks onto its target by resolving the selector on the object or array it updates. If the selector is resolved to the empty sequence, the update primitive is ignored in step 2. After this operation, each of these update primitives will contain a reference to either the pair (for an object) or the value (for an array) on or relatively to which it operates.

• Then each update primitive is applied, using the target references that were resolved at step 1. The order in which they are applied is not relevant and does not affect the final instance of the data model. After applying all updates, an error jerr:JNUP0006 is raised upon pair name collision within the same object.

This method gives you more control about the Spark configuration than the experimental standalone jar, in particular you can increase the memory used, change the number of cores, and so on.

If you use Linux, Florian Kellner also kindly contributed an installation script for Linux users that roughly takes care of what is described below for you.

Install Spark (if you do not have installed already)

RumbleDB requires an Apache Spark installation on Linux, Mac or Windows. Important note: it needs to be either Spark 4, or the Scala 2.13 build of Spark 3.5.

It is straightforward to directly , unpack it and put it at a location of your choosing. We recommend to pick Spark 4.0.0.

SPARK_HOME and PATH (you need to check even if you already have an existing installation)

You then need to point the SPARK_HOME environment variable to this directory, and to additionally add the subdirectory "bin" within the unpacked directory to the PATH variable. On macOS this is done by adding.

(with SPARK_HOME appropriately set to match your unzipped Spark directory) to the file .zshrc in your home directory, then making sure to force the change with

in the shell. In Windows, changing the PATH variable is done in the control panel. In Linux, it is similar to macOS.

As an alternative, users who love the command line can also install Spark with a package management system instead, such as brew (on macOS) or apt-get (on Ubuntu). However, these might be less predictable than a raw download.

You can test that Spark was correctly installed with:

Java version (important)

You need to make sure that you have Java 11 (for Spark 3.5) or 17 (for Spark 3.5 or 4.0) or 21 (for Spark 4.0) and that, if you have several versions installed, JAVA_HOME correctly points to the correct Java installation. Spark only supports Java 11 or 17 or 21 depending on the version.

Spark 4+ is documented to work with both Java 17 and Java 21. If there is an issue with the Java version, RumbleDB will inform you with an appropriate error message. You can check the Java version that is configured on your machine with:

Download the small version of the RumbleDB jar

Like Spark, RumbleDB is just a download and no installation is required.

In order to run RumbleDB, you simply need to download one of the small .jar files from the and put it in a directory of your choice, for example, right besides your data.

If you use Spark 3.5, use rumbledb-2.0.0-for-spark-3.5-scala-2.13.jar.

If you use Spark 4.0, use rumbledb-2.0.0-for-spark-4.0.jar.

These jars do not embed Spark, since you chose to set it up separately. They will work with your Spark installation with the spark-submit command.

Make sure to use the corresponding jar name accordingly in all our instructions in lieu of rumbledb.jar, replacing rumbledb.jar with the actual name of the jar file you downloaded.

In a shell, from the directory where the RumbleDB .jar lies, type, all on one line:

replacing rumbledb.jar with the actual name of the jar file you downloaded.

The RumbleDB shell appears:

You can now start typing simple queries like the following few examples. Press three times the return key to execute a query.

or

or

Starting the HTTP server

RumbleDB can be run as an HTTP server that listens for queries. In order to do so, you can use the --server and --port parameters:

This command will not return until you force it to (Ctrl+C on Linux and Mac). This is because the server has to run permanently to listen to incoming requests.

Most users will not have to do anything beyond running the above command. For most of them, the next step would be to open a Jupyter notebook that connects to this server automatically.

This HTTP server is built as a basic server for the single user use case, i.e., the user runs their own RumbleDB server on their laptop or cluster, and connects to it via their Jupyter notebook, one query at a time. Some of our users have more advanced needs, or have a larger user base, and typically prefer to implement their own HTTP server, lauching RumbleDB queries either via the public RumbleDB Java API (like the basic HTTP server does -- so its code can serve as a demo of the Java API) or via the RumbleDB CLI.

Caution! Launching a server always has consequences on security, especially as RumbleDB can read from and write to your disk; So make sure you activate your firewall. In later versions, we may support authentication tokens.

Testing that it works (not necessary for most end users)

The HTTP server is meant not to be used directly by end users, but instead to make it possible to integrate RumbleDB in other languages and environments, such as Python and Jupyter notebooks.

To test that the server is running, you can try the following address in your browser, assuming you have a query stored locally at /tmp/query.jq. All queries have to go to the /jsoniq path.

The request returns a JSON object, and the resulting sequence of items is in the values array.

Almost all parameters from the command line are exposed as HTTP parameters.

A query can also be submitted in the request body:

Use with Jupyter notebooks

With the HTTP server running, if you have installed Python and Jupyter notebooks (for example with the Anaconda data science package that does all of it automatically), you can create a RumbleDB magic by just executing the following code in a cell:

Where, of course, you need to adapt the port (8001) to the one you picked previously.

Then, you can execute queries in subsequent cells with:

or on multiple lines:

Use with clusters

You can also let RumbleDB run as an HTTP server on the master node of a cluster, e.g. on Amazon EMR or Azure. You just need to:

• Create the cluster (it is usually just the push of a few buttons in Amazon or Azure)

• Wait for a few minutes

• Make sure that your own IP has incoming access to EMR machines by configuring the security group properly. You usually only need to do so the first time you set up a cluster (if your IP address remains the same), because the security group configuration will be reused for future EMR clusters.

Then there are two options

With SSH tunneling

• Connect to the master with SSH with an extra parameter for securely tunneling the HTTP connection (for example -L 8001:localhost:8001 or any port of your choosing)

• Download the RumbleDB jar to the master node

  wget https://github.com/RumbleDB/rumble/releases/download/v1.24.0/rumbledb-1.24.0.jar

• Launch the HTTP server on the master node (it will be accessible under http://localhost:8001/jsoniq).

With the EC2 hostname

There is also another way that does not need any tunnelling: you can specify the hostname of your EC2 machine (copied over from the EC2 dashboard) with the --host parameter. For example, with the placeholder :

You also need to make sure in your EMR security group that the chosen port (e.g., 8001) is accessible from the machine in which you run your Jupyter notebook. Then, you can point your Jupyter notebook on this machine to http://<ec2-hostname>:8001/jsoniq.

Be careful not to open this port to the whole world, as queries can be sent that read and write to the EC2 machine and anything it has access to (like S3).

There are several ways to get back the output of the JSONiq query. There are many examples of use further down this page.

RumbleDB can work out of the box with pandas DataFrames, both as input and (when the output has a schema) as output.

Binding JSONiq variables to pandas DataFrames

bind() also accepts pandas dataframes

data = {'Name': ['Alice', 'Bob', 'Charlie'],
        'Age': [30,25,35]};
pdf = pd.DataFrame(data);

rumble.bind('$a',pdf);
seq = rumble.jsoniq('$a.Name')

The same goes for extra named parameters.

data = {'Name': ['Alice', 'Bob', 'Charlie'],
        'Age': [30,25,35]};
pdf = pd.DataFrame(data);

seq = rumble.jsoniq('$a.Name', a=pdf)

Getting the results as a pandas DataFrame

It is also possible to get the results back as a pandas dataframe with pdf() (if the output has a schema, which you can check by calling availableOutputs() and seeing if "DataFrame" is in the returned list).

We show here how to install RumbleDB from the GitHub repository and build it yourself if you wish to do so (for example, to use the latest master). However, the easiest way to use RumbleDB is to simply download the already compiled .jar files.

Requirements

The following software is required:

• : the version of Java is important, as RumbleDB only works with Java 11 (Standalone or Spark 3.5), 17 (Standalone or Spark 3.5 or Spark 4 or Python) or 21 (Spark 4 or Python). The current master branch corresponds to Spark 4.0, meaning that Java 17 or 21 is required.

The syntax to start a session is similar to that of Spark. A RumbleSession is a SparkSession that additionally knows about RumbleDB. All attributes and methods of SparkSession are also available on RumbleSession.

Even though RumbleDB uses Spark internally, it can be used without any knowledge of Spark.

Executing a query is done with rumble.jsoniq() like so.

A query returns a sequence of items, here the sequence with just the integer item 2.

There are several ways to retrieve the results of the query. Calling the json() is just one of them. It retrieves the sequence of as a tuple of JSON values that Python can process. The detailed . Other methods for .

RumbleDB can work out of the box with pyspark DataFrames, both as input and (when the output has a schema) as output.

Using Pyspark DataFrames with JSONiq

The power users can also interface our library with pyspark DataFrames. JSONiq sequences of items can have billions of items, and our library supports this out of the box: it can also run on clusters on AWS Elastic MapReduce for example. But your laptop is just fine, too: it will spread the computations on your cores. You can bind a DataFrame to a JSONiq variable. JSONiq will recognize this DataFrame as a sequence of object items.

Creating a data frame also similar to Spark (but using the rumble object).

This is how to bind a JSONiq variable to a dataframe. You can bind as many variables as you want.

It is possible to bind a JSONiq variable to a tuple of native Python values and then use it in a query. JSONiq, variables are bound to sequences of items, just like the results of JSONiq queries are sequence of items. A Python tuple will be seamlessly converted to a sequence of items by the library. Currently we only support strs, ints, floats, booleans, None, and (recursively) lists and dicts. But if you need more (like date, bytes, etc) we will add them without any problem. JSONiq has a rich type system.

Values can be passed with extra named parameters, like so.

It is also possible to bind variables more durably (across multiple jsoniq() calls) with bind().

It is possible to bind only one value. The it must be provided as a singleton tuple. This is because in JSONiq, an item is the same a sequence of one item.

For convenience and code readability, you can also use bindOne().

A variable that was durably bound with bind() or bindOne() can be unbound with unbind().

The Python edition of RumbleDB comes out of the box with a JSONiq magic.

If you are in a Jupyter notebook and have installed the jsoniq pip package, you can activate the jsoniq magic with:

Then, you can run JSONiq in standalone cells and see the results:

Of course, you can still continue to use rumble.jsoniq() calls and process the outputs as you see fit.

An example of the magic in action is available in our .

Note: This is a different magic than the magic that works with the RumbleDB HTTP server. It is more modern and running a server is no longer needed with this different magic. It suffices to install the jsoniq Python package.

Generally, it is possible to write output by to disk using the pandas DataFrame API, the pyspark DataFrame API, or Python's library to write JSON values to disk.

For convenience, we provide a way to also directly do so with the sequence object output by the query.

it is possible to write the output to a file locally or on a cluster. The API is similar to that of Spark dataframes. Note that it creates a directory and stores the (potentially very large) output in a sharded directory. RumbleDB was already tested with up to 64 AWS machines and 100s of TBs of data.

Of course the examples below are so small that it makes more sense to process the results locally with Python, but this shows how GBs or TBs of data obtained from JSONiq can be written back to disk.

Updates can be applied to a clone of an existing instance with the expression.

The content of the modify clause may build a complex Pending Update List with multiple updates. Remember that, with snapshot semantics, each update is applied against the initial snapshot, and updates do not see each other's effects.

Updating expression can also be combined with conditional expressions (in the then and else clauses), switch expressions (in the return clauses), FLWOR expressions (in the return clause), etc for more powerful queries based on patterns in the available data (from any source visible to the JSONiq query).

The updates generated inside the modify clause may only target the cloned object, i.e., the variable specified in the copy clause.

Example 191. JSON copy-modify-return expression

Result: { "foo" : true, "bar" : 123, "foobar" : [ true, false ] }

RumbleDB's specific configuration

It is possible to access RumbleDB's advanced configuration parameters with

Then, you can change the value of some parameters. For example, you can increase the number of JSON values that you can retrieve with a json() call:

You can also configure RumbleDB to output verbose information about the internal query plan, type and mode detection, and optimizations. This can be of interest to data engineers or researchers to understand how RumbleDB works.

The complete API for configuring RumbleDB is accessible in our pages. These methods are also callable in Python.

Warning: some of the configuration methods do not make sense in Python and are specific to the command line edition of RumbleDB (such as setting the query content or an output path and input/output format). Also, setting external variables in Python should not be done via the configuration, but with the bind() and unbind() functions or extra parameters in jsoniq() calls.

RumbleDB uses the following software:

• ANTLR v4 Framework - BSD License

• Apache Commons Text - Apache License

• Apache Commons Lang - Apache License

JSONiq follows the and introduces update primitives and update expressions specific to JSON data.

In JSONiq, updates are not immediately applied. Rather, a snapshot of the current data is taken, and a list of updates, called the Pending Update List, is collected. Then, upon explicit request by the user (via specific expressions), the Pending Update List is applied atomically, leading to a new snapshot. It is also possible for an engine to persist (to the local disk, to a database management system, to a data lake...) the resulting Pending Update List after a query has been completed.

In the middle of a program, several PULs can be produced against the same snapshot. They are then merged with upd:mergeUpdates (part of the XQuery Update Facility standard), which is extended as follows.

• Several deletes on the same object are replaced with a unique delete on that object, with a list of all selectors (names) to be deleted, where duplicates have been eliminated.

• Several deletes on the same array and selector (position) are replaced with a unique delete on that array and with that selector.

• Several inserts on the same array and selector (position) are equivalent to a unique insert on that array and selector with the content of those original inserts appended in an implementation-dependent order (like XQUF).

JSONiq 3.1 is an initiative of the RumbleDB team that aligns JSONiq more closely with XQuery 3.1, which has now become a W3C recommendation, but keeping what makes it JSONiq: the flagship feature being the ability to copy-paste JSON into a JSONiq query and with a navigation syntax that appeals to the JSON community.

JSONiq 3.1 does not require a distinct data model (JDM) since XQuery 3.1 support maps and arrays. As a result, JSONiq 3.1 objects are the same as XQuery 3.1 maps and JSONiq 3.1 arrays are the same as XQuery 3.1 arrays.

JSONiq 3.1 does not require a separate serialization mechanism, since XQuery 3.1 supports the JSON output method.

JSONiq 3.1 benefits from all the map and object builtin functions defined in XQuery 3.1.

JSONiq 3.1 is fully interoperable with XQuery 3.1 and can execute on the same virtual machine (similar to Scala and Java).

This also paves the way for JSONiq 4.0 which will also be aligned with XQuery 4.0 as much as is technically possible.

As a result, the specification for JSONiq 3.1 is even more minimal than that of JSONiq 1.0. This makes it easy to support for any existing XQuery engine to step into the JSON community.

RumbleDB is slowly deploying the use of JSONiq 3.1 but it will take some time as we make sure to sweep in all corners.

How JSONiq 3.1 amends XQuery 3.1

Context item

In JSONiq 3.1, the context item is obtained through $$ and not through a dot.

Escaping in strings

String literals use JSON escaping instead of XML escaping (backslash, not ampersand).

Map constructors

In map (object) constructors, the "map" keyword in front is optional.

Constraints on XPath

A name test must be prefixed with $$/ and cannot stand on its own.

True, null, and false literals

true and false exist as literals and do not have to be obtained through function calls (true(), false()).

null exists as a literal and stands for the empty sequence.

Navigation

The dot . and double square brackets [[ ]] act as syntactic sugars for ? lookup.

How JSONiq 3.1 differs from JSONiq 1.0

The data model standardized by the W3C working group is more generic and allows for atomic object keys that are not necessarily strings (dates, etc). Also, an object value or an array value can be a sequence of items and does not need to be a single item. The particular case in which object keys are strings and values are single items (or empty) corresponds to the JSON use.

Null does not exist as its own type in JSONiq 3.1, instead it is mapped to the empty sequence.

There are other minor changes in semantics that correspond to the alignment with XQuery 3.1 such as Effective Boolean Values, comparison, etc.

The JSON update syntax was not integrated yet into the core language. This is planned, and the syntax will be simplified (no json keyword, dot lookup allowed here as well).

The semantics for the JSON serialization method is the same as in the JSONiq Extension to XQuery. It is still under discussion how to escape special characters with the Text output method.

This section assumes that you have installed RumbleDB with one of the proposed ways, and guides you through your first queries.

Create some data set

Create, in the same directory as RumbleDB to keep it simple, a file data.json and put the following content inside. This is a small list of JSON objects in the JSON Lines format.

If you want to later try a bigger version of this data, you can also download a larger version with 100,000 objects from here. Wait, no, in fact you do not even need to download it: you can simply replace the file path in the queries below with "https://rumbledb.org/samples/products-small.json" and it will just work! RumbleDB feels just at home on the Web.

RumbleDB also scales without any problems to datasets that have millions or (on a cluster) billions of objects, although of course, for billions of objects HDFS or S3 are a better idea than the Web to store your data, for obvious reasons.

In the JSON Lines format that this simple dataset uses, you just need to make sure you have one object on each line (this is different from a plain JSON file, which has a single JSON value and can be indented). Of course, RumbleDB can read plain JSON files, too (with json-doc()), but below we will show you how to read JSON Line files, which is how JSON data scales.

Running simple queries locally

Depending on your installation method, the JSONiq queries will go to:

• A cell in a jupyter notebook and with the %%jsoniq magic: a simple click is sufficient to execute.

• The shell: type the query, and finish by pressing Enter twice.

• In a Python program, inside a rumble.jsoniq() call of which you can exploit the output with more Python code.

• A JSONiq query file, which you can execute with the RumbleDB CLI interface.

Either way, the meaning of the queries is the same.

or

or

The above queries do not actually use Spark. Spark is used when the I/O workload can be parallelized. The following query should output the file created above.

json-lines() reads its input in parallel, and thus will also work on your machine with MB or GB files (for TB files, a cluster will be preferable). You should specify a minimum number of partitions, here 10 (note that this is a bit ridiculous for our tiny example, but it is very relevant for larger files), as locally no parallelization will happen if you do not specify this number.

The above creates a very simple Spark job and executes it. More complex queries will create several Spark jobs. But you will not see anything of it: this is all done behind the scenes. If you are curious, you can go to in your browser while your query is running (it will not be available once the job is complete) and look at what is going on behind the scenes.

Data can be filtered with the where clause. Again, below the hood, a Spark transformation will be used:

RumbleDB also supports grouping and aggregation, like so:

RumbleDB also supports ordering. Note that clauses (where, let, group by, order by) can appear in any order. The only constraint is that the first clause should be a for or a let clause.

Finally, RumbleDB can also parallelize data provided within the query, exactly like Sparks' parallelize() creation:

Mind the double parenthesis, as parallelize is a unary function to which we pass a sequence of objects.

The docker installation is kindly contributed by Dr. Ingo Müller (Google).

Known issue

On occasion, the docker version of RumbleDB used to throw a Kryo NoSuchMethodError on some systems. This should be fixed with version 2.0.0, let us know if this is not the case.

You can upgrade to the newest version with

Running simple queries with Docker

Docker is the easiest way to get a standard environment that just works.

You can download Docker from .

Then, in a shell, type, all on one line:

The first time, it might take some time to download everything, but this is all done automatically. Subsequent commands will run immediately.

When there are new RumbleDB versions, you can upgrade with:

The RumbleDB shell appears:

You can now start typing simple queries like the following few examples. Press three times the return key to execute a query.

or

or

The above queries do not actually use Spark. Spark is used when the I/O workload can be parallelized. The following query should output the file created above.

json-lines() reads its input in parallel, and thus will also work on your machine with MB or GB files (for TB files, a cluster will be preferable). You should specify a minimum number of partitions, here 10 (note that this is a bit ridiculous for our tiny example, but it is very relevant for larger files), as locally no parallelization will happen if you do not specify this number.

The above creates a very simple Spark job and executes it. More complex queries will create several Spark jobs. But you will not see anything of it: this is all done behind the scenes. If you are curious, you can go to in your browser while your query is running (it will not be available once the job is complete) and look at what is going on behind the scenes.

Data can be filtered with the where clause. Again, below the hood, a Spark transformation will be used:

RumbleDB also supports grouping and aggregation, like so:

RumbleDB also supports ordering. Note that clauses (where, let, group by, order by) can appear in any order. The only constraint is that the first clause should be a for or a let clause.

Finally, RumbleDB can also parallelize data provided within the query, exactly like Sparks' parallelize() creation:

Mind the double parenthesis, as parallelize is a unary function to which we pass a sequence of objects.

Running the RumbleDB docker as a server

You can also run the docker as a server like so:

You can change the port to something else than 8001 at all three places it appears. Do not forget -p 8001:8001 that forwards the port to the outside of the docker. Then, you can use a connected to the RumbleDB docker server to write queries in it. Point the notebook to http://localhost:8001/jsoniq in the appropriate cell (or any other port).

Querying local files with the docker version of RumbleDB

In order to query your local files, you need to mount a local directory to a directory within the docker. This is done with the --mount option, and the source path must be absolute. For the target, you can pick anything that makes sense to you.

For example, imagine you have a file products-small.json in the directory /path/to/my/directory. Then you need to run RumbleDB with:

Then you can go ahead and use absolute paths in the target directory in input functions, like so:

You can also mount a local directory in this way running it as a server rather than a shell.

Running RumbleDB on a cluster

After you have tried RumbleDB locally as explained in the getting started section, you can take RumbleDB to a real cluster simply by modifying the command line parameters as documented here for spark-submit.

Creating a cluster

Creating a cluster is the easiest part, as most cloud providers today offer that with just a few clicks: Amazon EMR, Azure HDInsight, etc. You can start with 4-5 machines with a few CPUs each and a bit of memory, and increase later when you want to get serious on larger scales.

Maybe sure to select a cluster that has Apache Spark. On Amazon EMR, this is not the default and you need to make sure that you check the box that has Spark below the cluster version dropdown. We recommend taking the latest EMR version 6.5.0 and then picking Spark 3.1 in the software configuration. You will also need to create a public/private key pair if you do not already have one.

Wait for 5 or 6 minutes, and the cluster is ready.

Do not forget to terminate the cluster when you are done!

How to tune the RumbleDB command

Next, you need to use ssh to connect to the master node of your cluster as the hadoop user and specifying your private key file. You will find the hostname of the machine on the EMR cluster page. The command looks like:

ssh -i ~/.ssh/yourkey.pem [email protected]

If ssh hangs, then you may need to authorize your IP for incoming connections in the security group of your cluster.

And once you have connected with ssh and are on the shell, you can start using RumbleDB in a way similar to what you do on your laptop.

First you need to download it with wget (which is usually available by default on cloud virtual machines):

This is all you need to do, since Apache Spark is already installed. If spark-submit does not work, you might want to wait for a few more minutes as it might be that the cluster is not fully prepared yet.

Often, the Spark cluster is running on yarn. The --master option can be changed from local[*] (which was for running on your laptop) to yarn compared to the getting started guide.

Most of the time, though (e.g., on Amazon EMR), it needs not be specified, as this is already set up in the environment. So the same command will do:

When you are on a cluster, you can also adapt the number of executors, how many cores you want per executor, etc. It is recommended to use sqrt(n) cores per executor if a node has n cores. For the executor memory, it is just primary school math: you need to divide the memory on a machine with the number of executors per machine (which is also roughly sqrt(n)).

For example, if we have 6 worker nodes with each 16 cores and 64 GB, we can use 5 executores on each machine, with 3 cores and 10 GB per executor. This leaves a core and a bit of memory free for other cluster tasks.

If necesasry, the size limit for materialization can be made higher with --materialization-cap or its shortcut -c (the default is 200). This affects the number of items displayed on the shells as an answer to a query. It also affects the maximum number of items that can be materialized from a large sequence into, say, an array. Warnings are issued if the cap is reached.

Creation functions

json-lines() then takes an HDFS path and the host and port are optional if Spark is configured properly. A second parameter controls the minimum number of splits. By default, each HDFS block is a split if executed on a clustter. In a local execution, there is only one split by default.

The same goes for parallelize(). It is also possible to read text with text-file(), parquet files with parquet-file(), and it is also possible to read data on S3 rather than HDFS for all three functions json-lines(), text-file() and parquet-file().

Bigger data sets

If you need a bigger data set out of the box, we recommend the , which has 16 million objects. On Amazon EMR, we could even read several billion of objects on less than ten machines.

We tested this with each new release, and suggest the following queries to start with (we assume HDFS is the default file system, and that you copied over this dataset to HDFS with hadoop fs copyFromLocal):

Note that by default only the first 200 items in the output will be displayed on the shell, but you can change it with the --materialization-cap parameter on the CLI.

Execution of single queries and output to HDFS

RumbleDB also supports executing a single query from the command line, reading from HDFS and outputting the results to HDFS, with the query file being either local or on HDFS. For this, use the --query-path (optional as any text without parameter is recognized as a path in any case), --output-path (shortcut -o) and --log-path parameters.

The query path, output path and log path can be any of the supported schemes (HDFS, file, S3, WASB...) and can be relative or absolute.

As in most language, one can distinguish between physical equality and logical equality.

Atomics can only be compared logically. Their physically identity is totally opaque to you.

Logical comparison of two atomics

Result (run with Zorba):true

Logical comparison of two atomics

Result (run with Zorba):false

Logical comparison of two atomics

Result (run with Zorba):false

Logical comparison of two atomics

Result (run with Zorba):true

Two objects or arrays can be tested for logical equality as well, using deep-equal(), which performs a recursive comparison.

Logical comparison of two JSON items

Result (run with Zorba):true

Logical comparison of two JSON items

Result (run with Zorba):false

The physical identity of objects and arrays is not exposed to the user in the core JSONiq language itself. Some library modules might be able to reveal it, though.

Module

You can group functions and variables in separate library modules.

MainModule

Up to now, everything we encountered were main modules, i.e., a prolog followed by a main query.

LibraryModule

A library module does not contain any query - just functions and variables that can be imported by other modules.

A library module must be assigned to a namespace. For convenience, this namespace is bound to an alias in the module declaration. All variables and functions in a library module must be prefixed with this alias.

A library module

ModuleImport

Here is a main module which imports the former library module. An alias is given to the module namespace (my). Variables and functions from that module can be accessed by prefixing their names with this alias. The alias may be different than the internal alias defined in the imported module.

An importing main module

Result (run with Zorba):1764

JSONiq is a query language that was specifically designed for querying JSON, although its data model is powerful enough to handle more similar formats.

As stated on json.org, JSON is a "lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate."

A JSON document is made of the following building blocks: objects, arrays, strings, numbers, booleans and nulls.

JSONiq manipulates sequences of these building blocks, which are called items. Hence, a JSONiq value is a sequence of items.

Any JSONiq expression takes and returns sequences of items.

Comma-separated JSON-like building blocks is all you need to begin building your own sequences. You can mix and match, as JSONiq supports heterogeneous sequences seamlessly.

Out of memory error

By default, the memory allocated is limited. This depends on whether you run RumbleDB with the standalone jar or as the thin jar in a Spark environment.

If you run RumbleDB with a standalone jar, then your laptop will allocate by default one quarter of your total working memory. You can check this with:

In order to increase the memory, you can use -Xmx10g (for 10 GB, but you can use any other value):

If you run RumbleDB on your laptop (or a single machine) with the thin jar, then by default this is limited to around 2 GB, and you can change this with --driver-memory

This section shares more techniques for advanced users who want to make the most of RumbleDB in Python.

RumbleDB Item API

JSONiq has a rich type system, and the can lose type information.

An alternative consists of retrieving the sequence as a tuple of native items, which can be accessed with the .

Even though you can build your own JSON values with JSONiq by copying-and-pasting JSON documents, most of the time, your JSON data will come from an external input dataset.

How this dataset is access depends on the JSONiq implementation and of the context. Some engines can read the data from a file located on a file system, local or distributed (HDFS, S3); some others get data from the Web; some others are full-fledged datastores and have collections that can be created, queried, modified and persisted.

It is up to each engine to document which functions should be used, and how, in order to read datasets into a JSONiq Data Model instance. These functions will take implementation-defined parameters and typically return sequences of objects, or sequences of strings, or sequences of items, etc.

For the purpose of examples given in this specification, we assume that a hypothetical engine has collections that are sequences of objects, identified by a name which is a string. We assume that there is a collection() function that returns all objects associated with the provided collection name.

We assume in particular that there are three example collections, shown below.

A Pending Update List is an unordered list of update primitives. Update primitives are internal and do not appear in the syntax. Each kind of update primitive models one individual update to an object or an array.

A Pending Update List can by analogy be seen as the diff between two git revisions, and a single update primitive can be seen, with this same analogy, as the difference between two single lines of code. Thus, the JSONiq Update Facility is to trees what git is to lines of text: a "tree diff" language.

JSONiq adds the following new update primitives, specific to JSON. They are similar to those defined by the XQuery Update Facility for XML.

Update primitives within a PUL are applied with strict snapshot semantics. For examples, the positions are resolved against the array before the updates. Names are resolved on the object before the updates.

Update primitive for objects and arrays (in collections or in memory)

Update primitives at the collection level

Credits: Dwij Dixit/Ghislain Fourny (student project at ETH)

Update expressions are the visible part of JSONiq Updates in the language. Each primary updating expression contributes an update primitive to the Pending Update List being built.

Nested updates (memory or persistent)

These expressions may appear in a copy-modify-return (transform) expression (for in-memory updates on cloned values), or outside (for persistent updates to an underlying storage).

This section introduces prologs, which allows declaring functions and global variables that can then be used in the main query. A prolog also allows setting some default behaviour.

MainModule

Prolog

The prolog appears before the main query and is optional. It can contain setters and module imports, followed by function and variable declarations.

Module imports are explained in the next chapter.

• [FOAR0001] - Division by zero.

• [FOAR0002] - Numeric operation overflow/underflow

• [FOCA0002] - A value that is not lexically valid for a particular type has been encountered.

• [FOCH0001] - Raised by fn:codepoints-to-string if the input contains an integer that is not the codepoint of a valid XML character.

• [FOCH0003] - Raised by fn:normalize-unicode if the requested normalization form is not supported by the implementation.

• [FODC0002] - Error retrieving resource.

• [FODT0001] - Overflow/underflow in date/time operation.

• [FODT0002] - Overflow/underflow in duration operation.

• [FOFD1340] -This error is raised if the picture string or calendar supplied to fn:format-date, fn:format-time, or fn:format-dateTime has invalid syntax.

• [FOFD1350] - This error is raised if the picture string supplied to fn:format-date selects a component that is not present in a date, or if the picture string supplied to fn:format-time selects a component that is not present in a time.

• [FOTY0012] - The argument has no typed value (objects, arrays, functions cannot be atomized).

• [JNTY0004] - Unexpected non-atomic element. Raised when objects or arrays are supplied where an atomic element is expected.

• [JNTY0024] - Error getting the string value for array and object items

• [JNTY0018] - Invalid selector error code. It is a type error if there is not exactly one supplied parameter for an object or array selector.

• [RBDY0005] - Materialization Error: the sequence is too big to be materialized. Use --materialization-cap to increase the maximum materialization size, or add an output path to write to.

• [RBML0001] - Unrecognized RumbleDB ML Class Reference An unrecognized classname is used in query while accessing the RumbleDB ML API.

• [RBML0002] - Unrecognized RumbleDB ML Param Reference An unrecognized parameter is used in query while operating with a RumbleDB ML class.

• [RBML0003] - Invalid RumbleDB ML Param Provided parameter does not match the expected type or value for the referenced RumbleDB ML class.

• [RBML0004] - Input is not a DataFrame Provided input of items does not form a DataFrame as expected by RumbleDB ML.

• [RBML0005] - Invalid schema for DataFrame in annotate() The provided schema can not be applied to the item data while converting the data to a DataFrame

• [RBST0001] - CLI error. Raised when invalid parameters are supplied at launch.

• [RBST0002] - Unimplemented feature error. Raised when a JSONiq feature that is not yet implemented in RumbleDB is used.

• [RBST0003] - Invalid for clause expression error. Raised when an expression produces a different, big sequence of items for each binding within a big tuple, which would lead to a data flow explosion and to a nesting of jobs on the Spark cluster.

• [RBST0004] - Implementation Error.

• [SENR0001] - Serialization error. Function items can not be serialized.

• [XPDY0002] - It is a dynamic error if evaluation of an expression relies on some part of the dynamic context that is absent.

• [XPDY0050] - Dynamic type treat error. It is a dynamic error if the dynamic type of the operand of a treat expression does not match the sequence type specified by the treat expression. This error might also be raised by a path expression beginning with "/" or "//" if the context node is not in a tree that is rooted at a document node. This is because a leading "/" or "//" in a path expression is an abbreviation for an initial step that includes the clause treat as document-node().

• [XPDY0130] - Generic runtime exception [check error message].

• [XPST0003] - Parsing error. Invalid syntax or unsupported feature in query.

• [XPST0008] - Undefined element reference. It is a static error if an expression refers to an element name, attribute name, schema type name, namespace prefix, or variable name that is not defined in the static context, except for an ElementName in an ElementTest or an AttributeName in an AttributeTest.

• [XPST0017] - Invalid function call error. It is a static error if the expanded QName and number of arguments in a static function call do not match the name and arity of a function signature in the static context.

• [XPST0080] - Invalid cast error - It is a static error if the target type of a cast or castable expression is NOTATION anySimpleType, or anyAtomicType.

• [XPST0081] - Unknown namespace prefix - It is a static error if a QName used in a query contains a namespace prefix that cannot be expanded into a namespace URI by using the statically known namespaces.

• [XPTY0004] - Unexpected Type Error. It is a type error if, during the static analysis phase, an expression is found to have a static type that is not appropriate for the context in which the expression occurs, or during the dynamic evaluation phase, the dynamic type of a value does not match a required type. Example: using subtraction on strings.

• [XQDY0054] - It is a dynamic error if a cycle is encountered in the definition of a module's dynamic context components, for example because of a cycle in variable declarations.

• [XQTY0024] - Attribute After Non Attribute Error - It is a type error if the content sequence in an element constructor contains an attribute node following a node that is not an attribute node.

• [XQDY0025] - Duplicate Attribute Error - It is a dynamic error if any attribute of a constructed element does not have a name that is distinct from the names of all other attributes of the constructed element.

• [XQDY0074] - Invalid Element Name Error - It is a dynamic error if the value of the name expression in a computed element or attribute constructor cannot be converted to an expanded QName (for example, because it contains a namespace prefix not found in statically known namespaces.)

• [XQDY0096] - Invalid Node Name Error - It is a dynamic error if the node-name of a node constructed by a computed element constructor has any of the following properties: 1. Its namespace prefix is xmlns. 2. Its namespace URI is http://www.w3.org/2000/xmlns/. 3. Its namespace prefix is xml and its namespace URI is not http://www.w3.org/XML/1998/namespace. 4. Its namespace prefix is other than xml and its namespace URI is http://www.w3.org/XML/1998/namespace.

• [XQDY0137] - Duplicate pair name. It is a dynamic error if two pairs in an object constructor or in a simple object union have the same name.

• [XQST0016] - Module declaration error. Current implementation does not support the Module Feature raises a static error if it encounters a module declaration or a module import.

• [XQST0031] - Invalid JSONiq version. It is a static error if the version number specified in a version declaration is not supported by the implementation. For now, only version 1.0 is supported.

• [XQST0033] - Namespace prefix bound twice. It is a static error if a module contains multiple bindings for the same namespace prefix.

• [XQST0034] - Function already exists. It is a static error if multiple functions declared or imported by a module have the same number of arguments and their expanded QNames are equal (as defined by the eq operator).

• [XQST0038] - It is a static error if a Prolog contains more than one default collation declaration, or the value specified by a default collation declaration is not present in statically known collations.

• [XQST0039] - Duplicate parameter name. It is a static error for a function declaration or an inline function expression to have more than one parameter with the same name.

• [XQST0047] - It is a static error if multiple module imports in the same Prolog specify the same target namespace.

• [XQST0048] - It is a static error if a function or variable declared in a library module is not in the target namespace of the library module.

• [XQST0049] - It is a static error if two or more variables declared or imported by a module have the same name.

• [XQST0052] - Simple type error. The type must be the name of a type defined in the in-scope schema types, and the {variety} of the type must be simple.

• [XQST0059] - It is a static error if an implementation is unable to process a schema or module import by finding a schema or module with the specified target namespace.

• [XQST0069] - A static error is raised if a Prolog contains more than one empty order declaration.

• [XQST0088] - It is a static error if the literal that specifies the target namespace in a module import or a module declaration is of zero length.

• [XQST0089] - It is a static error if a variable bound in a for or window clause of a FLWOR expression, and its associated positional variable, do not have distinct names (expanded QNames).

• [XQST0094] - Invalid variable in group-by clause. The name of each grouping variable must be equal (by the eq operator on expanded QNames) to the name of a variable in the input tuple stream.

• [XQST0118] - In a direct element constructor, the name used in the end tag must exactly match the name used in the corresponding start tag, including its prefix or absence of a prefix.

This section describes JSONiq types as well as the sequence type syntax.

JSONiq manipulates semi-structured data: in general, JSONiq allows you, but does not require you to specify types. So you have as much or as little type verification as you wish.

JSONiq is still strongly typed, so that you will be told if there is a type inconsistency or mismatch in your programs.

Whenever you do not specify the type of a variable or the type signature of a function, the most general type for any sequence of items, item*, is assumed.

Section Expressions dealing with types introduces expressions which work with values of these types, as well as type operations (variable types, casts, ...).

Sequence types

JSONiq follows the regarding sequence occurrence indicators. The following explanations, provided as an informal summary for convenience, are non-normative.

A sequence is an ordered list of items.

All sequences match the sequence type js:item*.

A sequence type is made of an item type followed by an occurrence indicator:

• The symbol * (star) stands for a sequence of any length (zero or more)

• The symbol + (plus) stands for a non-empty sequence (one or more)

• The symbol ? (question mark) stands for an empty or a singleton sequence (zero or one)

• The absence of indicator stands for a singleton sequence (one).

Examples:

• string matches any singleton sequence containing a string.

• item+ matches any non-empty sequence.

• object? matches the empty sequence and any sequence containing one object.

JSONiq defines the syntax () for the empty sequence, rather than empty-sequence().

SequenceType

Item types

Item types are the first component of a sequence type, together with the cardinality indicator. Thus, an item type matches (or not) a single item. For example, "foo" matches the item type xs:string.

There are three categories of item types:

• Atomic types (W3C-conformant, additional js:null and js:atomic)

• Structured types (JSONiq-specific)

• Function types (W3C-conformant)

JSONiq uses a JSONiq-specific, implementation-defined default type namespace that acts as a proxy namespace to all types (xs: or js:). As a consequence, buitin atomic types do not need to be prefixed in the JSONiq syntax (integer instead of xs:integer, null instead of js:null).

All items match the item type js:item, which is a JSONiq-specific synonym for the W3C-confirmant item().

ItemType

Atomic types

JSONiq follows the for atomic types except for modifications in the list of available atomic types and a simplified syntax for xs:anyAtomicType. The following explanations, provided as an informal summary for convenience, are non-normative.

Atomic types are organized in a tree hierarchy.

JSONiq defines the following build-in types that have a direct relation with JSON:

• xs:string: the value space is all strings made of Unicode characters.

  All string literals build an atomic which matches string.

• xs:integer (W3C-conformant): the value space is that of all mathematical integral numbers (N), with an infinite range. This is a subtype of decimal, so that all integers also match the item type decimal.

  All integer literals build an atomic which matches integer.

• xs:decimal (W3C-conformant): the value space is that of all mathematical decimal numbers (D), with an infinite range.

JSONiq also supports further atomic types, which are conformant with .

These datatypes are already used as a set of atomic datatypes by the other two semi-structured data formats of the Web: XML and RDF, as well as by the corresponding query languages: XQuery and SPARQL, so it is natural for a complete JSON data model to reuse them.

• Further number types: xs:float, xs:long, xs:int, xs:short, xs:byte, xs:float, xs:positiveInteger, xs:negativeInteger, xs:nonPositiveInteger, xs:nonNegativeInteger, xs:unsignedLong, xs:unsignedInt, xs:unsignedShort, xs:unsignedByte.

• Date or time types: xs:date, xs:dateTime, xs:dateTimeStamp, xs:gDay, xs:gMonth, xs:gMonthDay, xs:gYear, xs:xs:gYearMonth, xs:time.

• Duration types: xs:duration, xs:dayTimeDuration, xs:yearMonthDuration.

• Binary types: xs:base64Binary, xs:hexBinary.

The support of xs:ID, xs:IDREF, xs:IDREFS, xs:NOTATION, xs:Name, xs:NCName, xs:NMTOKEN, xs:NMTOKENS, xs:ENTITY, xs:ENTITIES is not required by JSONiq, although engines that also support XML can support them.

AtomicType

Structured types

JSONiq introduces four more types for matching objects and arrays. Like atomic types, they do not need the js: prefix in the syntax (object instead of js:object, etc.).

All objects match the item type js:object.

All arrays match the item type js:array.

All objects and arrays match the item type js:json-item.

For engines that also support optionally XML, js:structured-item matches both XML nodes and JSON objects and arrays.

StructuredType

Function types

JSONiq follows the regarding function types. The following explanations are non-normative.

FunctionType

AnyFunctionType

TypedFunctionType

RumbleDB now supports user-defined array and object types both with the JSound compact syntax and the JSound verbose syntax.

JSound Schema Compact syntax

RumbleDB user-defined types can be defined with the JSound syntax. A tutorial for the JSound syntax can be found .

For now, RumbleDB only allows the definition of user-defined types for objects and arrays. User-defined atomic types and union types will follow soon. The @ (primary key) and ? (nullable) shortcuts are supported as of version 2.0.5. The behavior of nulls with absent vs. nullable fields can be tweaked in the configuration (e.g., if a null is present in an optional, non-nullable field, RumbleBD can be lenient and simply remove it instead of throwing an error).

The implementation is still experimental and bugs are still expected, which we will appreciate to be informed of.

Sequences vs. Arrays

Even though JSON supports arrays, JSONiq uses a different construct as its first class citizens: sequences. Any value returned by or passed to an expression is a sequence.

The main difference between sequences and arrays is that sequences are completely flat, meaning they cannot contain other sequences.

Since sequences are flat, expressions of the JSONiq language just concatenate them to form bigger sequences.

This is crucial to allow streaming results, for example through an HTTP session.

RumbleDB relies on the JSONiq language.

JSONiq reference

The complete specification can be found here and on the JSONiq.org website. The implementation is now in a very advanced stage and there remain only few unsupported core JSONiq features.

JSONiq tutorial

A tutorial can be found . All queries in this tutorial will work with RumbleDB.

JSONiq tutorial for Python users

A tutorial aimed at Python users can be found . Please keep in mind, though, that examples using not supported features may not work (see below).

Nested FLWOR expressions

FLWOR expressions now support nestedness, for example like so:

However, keep in mind that parallelization cannot be nested in Spark (there cannot be a job within a job), that is, the following will not work:

Expressions pushed down to Spark

Many expressions are pushed down to Spark out of the box. For example, this will work on a large file leveraging the parallelism of Spark:

What is pushed down so far is:

• FLWOR expressions (as soon as a for clause is encountered, binding a variable to a sequence generated with json-lines() or parallelize())

• aggregation functions such as count

• JSON navigation expressions: object lookup (as well as keys() call), array lookup, array unboxing, filtering predicates

• predicates on positions, include use of context-dependent functions position() and last(), e.g.,

More expressions working on sequences will be pushed down in the future, prioritized on the feedback we receive.

We also started to push down some expressions to DataFrames and Spark SQL (obtained via structured-json-lines, csv-file and parquet-file calls). In particular, keys() pushes down the schema lookup if used on parquet-file() and structured-json-lines(). Likewise, count() as well as object lookup, array unboxing and array lookup is also pushed down on DataFrames.

When an expression does not support pushdown, it will materialize automaticaly. To avoid issues, the materializion is capped by default at 200 items, but this can be changed on the command line with --materialization-cap. A warning is issued if a materialization happened and the sequence was truncated on screen. An error is thrown if this happens within a query.

External global variables.

Prologs with user-defined functions and global variables are supported. Global external variables are supported (use "--variable:foo bar" on the command line to assign values to them). If the declared type is not string, then the literal supplied on the command line is cast. If the declared type is anyURI, the path supplied on the command line is also resolved against the working directory to an absolute URI. Thus, anyURI should be used to supply paths dynamically through an external variable.

Context item declarations are supported and a global context item value can be passed with the "--context-item" or "-I" parameter on the command line.

Library modules

Library modules are now supported (experimental, please report bugs), and their namespace URI is used for resolution. If it is relative, it is resolved against the importing module location.

The same schemes are supported as for reading queries and data: file, hdfs, and so on. HTTP is also supported: you can import modules from the Web!

Example of library module (the file name is library-module.jq):

Example of importing module (assuming it is in the same directory):

Try/catch

Try/catch expressions are supported. Error codes are in the default, RumbleDB namespace and do not need prefixes.

Supported types

The JSONiq type system is fully supported. Below is a complete list of JSONiq types and their support status. All builtin types are in the default type namespace, so that no prefix is needed. These types are defined in the XML Schema standard. Note that some types specific to XML (e.g., NOTATION, NMTOKENS, NMTOKEN, ID, IDREF, ENTITY, etc) are not part of the JSONiq standard and not supported by RumbleDB.

Unsupported/Unimplemented features (beta release)

Most core features of JSONiq are now in place, and we are working on getting the last (less used) ones into RumbleDB as well. We prioritize their implementation on user requests.

Prolog

Some prolog settings (base URI, ordering mode, decimal format, namespace declarations) are not supported yet.

Location hints for the resolution of modules are not supported yet.

FLWOR features

Window clauses are not supported, because they are not compatible with the Spark execution model.

Function types

Function type syntax is supported.

Function annotations are not supported (%public, %private...), but this is planned.

Builtin functions

Most JSONiq and XQuery builtin functions are now supported (see function documentation), except XML-specific functions. A few are still missing, do not hesitate to reach out if you need them.

Constructors for atomic types are fully supported.

Buitin functions cannot yet be used with named function reference expressions (example: concat#2).

Error variables

Error variables ($err:code, ...) for inside catch blocks are not supported.

Updates and scripting

There are future plans to support JSONiq updates and scripting.

The parameters that can be used on the command line as well as on the planned HTTP server are shown below. They are also accessible via the Java API and via Python through the RumbleRuntimeConfiguration class.

RumbleDB runs in three modes. You can select the mode passing a verb as the first parameter. For example:

Previous parameters (--shell, --query-path, --server) work in a backward compatible fashion, however we do recommend to start using the new verb-based format.

RumbleDB is able to read a variety of formats from a variety of file systems and database management systems.

We support functions to read JSON, JSON Lines, XML, Parquet, CSV, Text, ROOT, Delta files from various storage layers such as S3 and HDFS, Azure blob storage. We run most of our tests on Amazon EMR with S3 or HDFS, as well as locally on the local file system, but we welcome feedback on other setups.

We also support some ETL-based systems such as PostgreSQL, MongoDB and the Hive metastore.

Supported formats