Course / The model / Lesson 10

Lesson 10 · The model

subQueryColumn and bagItemColumn

Declare collection and scalar sub-query columns, and read single items of an XML document stored in a column.

about 4 minutes · screenplay estimate8 scenesPrerequisites: lessons 06 and 08

Development course · Reading edition · API compatibility

Scene 01

subQueryColumn

subQueryColumn declares a virtual column computed by a sub-query. query is a dictionary with the same keys as the select of a formula: table, columns, where, and optionally order_by and limit. hash THIS correlates it with the outer row. mode chooses the shape of the result. With mode json, the rows become a JSON array. With mode xml, they become XML text. Without mode, the sub-query is a plain scalar sub-select, and it must return one column. A third argument, aggr, is shown in scene 4.

Example & screen notes

Slide title "subQueryColumn(name, query=, mode=, aggr=)". Code, additions copied from examples/test_invoice_pg/invc.py:

invoice.subQueryColumn('rows_json', query=dict(
    table='invc.invoice_row', columns='$product_id,$quantity,$unit_price',
    where='$invoice_id=#THIS.id'), mode='json')
invoice.subQueryColumn('notes_xml', query=dict(
    table='invc.invoice_note', columns='$note_type,$note_text',
    where='$invoice_id=#THIS.id'), mode='xml')
invoice.subQueryColumn('max_row_price', query=dict(
    table='invc.invoice_row', columns='MAX($unit_price)',
    where='$invoice_id=#THIS.id'), dtype='N')

Scene 02

mode='json' and the scalar form

With mode json, PostgreSQL builds the array with jsonb_agg. The driver returns it as a Python list of dictionaries. The keys are the column names of the sub-query. Invoice A-003 has no rows, and its value is None, not an empty list. This is the opposite of to_json inside aggr, which returns an empty array. max_row_price has no mode. It is a scalar sub-select with dtype N, like a select formula.

Example & screen notes

Slide title "rows_json, max_row_price". Code:

db.table('invc.invoice').query(columns='$inv_number, $rows_json, $max_row_price',
                               order_by='$inv_number')

SQL of rows_json:

( (SELECT jsonb_agg(row_to_json("invc_invoice_rows_json_json")) FROM (SELECT "t0_t0"."product_id" AS "product_id",
"t0_t0"."quantity" AS "quantity",
"t0_t0"."unit_price" AS "unit_price"
 FROM "invc"."invc_invoice_row" AS "t0_t0"
WHERE ("t0_t0"."invoice_id"="t0"."id")) AS "invc_invoice_rows_json_json") ) AS "rows_json",

Rows:

inv_number rows_json max_row_price
A-001 [{'quantity': 2, 'product_id': 'p1 ...', 'unit_price': 300.0}, {...}] 300.00
A-002 [{'quantity': 1, 'product_id': 'p3 ...', 'unit_price': 20.0}] 20.00
A-003 None None

Scene 03

mode='xml'

With mode xml, each row becomes one XML element. Its children are the columns of the sub-query. The element name is built from the table and column names, escaped by PostgreSQL. There is no root element. An invoice without notes gets None. This format is kept from earlier releases. For XML in the Bag format, with a GenRoBag root and rows named r underscore position, use to_xml inside aggr, as in lesson 06, or the aggr argument of the next scene.

Example & screen notes

Slide title "notes_xml". Value for A-001:

<invc_invoice_notes_x005F_xml_x005F_xml><note_type>WARN</note_type><note_text>Fragile</note_text></invc_invoice_notes_x005F_xml_x005F_xml>
<invc_invoice_notes_x005F_xml_x005F_xml><note_type>INFO</note_type><note_text>Deliver at 9</note_text></invc_invoice_notes_x005F_xml_x005F_xml>

Value for an invoice without notes: None

Scene 04

aggr=

aggr takes exactly one call of the aggr grammar of lesson 06: count, sum, avg, min, max, join, to_json or to_xml. The call replaces the columns of the query dictionary. The order_by of the dictionary orders join, to_json and to_xml. The results on an empty set are those of lesson 06. row_count gives zero and product_list an empty string. aggr and mode together raise ValueError. Two calls in aggr raise ValueError. A mode other than json or xml raises ValueError and names aggr.

Example & screen notes

Slide title "subQueryColumn(..., aggr='...')". Code:

lines = dict(table='invc.invoice_row', where='$invoice_id=#THIS.id')
invoice.subQueryColumn('row_count', query=lines, aggr='count()', dtype='L')
invoice.subQueryColumn('product_list', query=dict(lines, order_by='$unit_price DESC'),
                       aggr="join($product_name, ', ')", dtype='T')

Rows:

inv_number row_count product_list
A-001 2 Desk, Chair
A-002 1 Book
A-003 0 (empty string)

Scene 05

An XML document in a column

A column with dtype X holds an XML document. In PostgreSQL it is a text column. The value on the slide is a Bag document. specs contains weight, with a type attribute and a unit attribute, and color. A query with columns star leaves X columns out. The query option bagFields True includes them. Lesson 14 covers query options.

Example & screen notes

Slide title "dtype='X'". Code, an addition copied from examples/test_invoice_pg/invc.py:

product.column('details', dtype='X')

Value stored for product DSK:

<GenRoBag><specs><weight _T="N" unit="kg">12.5</weight><color>oak</color></specs></GenRoBag>

Bottom line: "columns='*' leaves X columns out; bagFields=True includes them."

Scene 06

bagItemColumn

bagItemColumn reads one item of the XML document. bagcolumn is the column that holds the document. itempath is a dotted path inside it. A question mark followed by a name reads an attribute of the item. The column is computed with the PostgreSQL xpath function. The path is a bound parameter of the column. The text is cast to the declared dtype, here numeric. A missing document or a missing item gives None. A bag item column works in where and order_by too. For example, where detail_weight greater than ten returns only the desk.

Example & screen notes

Slide title "bagItemColumn(name, bagcolumn=, itempath=)". Code:

product.bagItemColumn('detail_weight', bagcolumn='$details', itempath='specs.weight', dtype='N')
product.bagItemColumn('detail_color', bagcolumn='$details', itempath='specs.color', dtype='T')
product.bagItemColumn('weight_unit', bagcolumn='$details', itempath='specs.weight?unit', dtype='T')

SQL of detail_weight:

( CAST((CAST((xpath(%(f1_calculated_path)s, CAST("t0"."details" AS XML)))[1] AS text)) AS numeric) ) AS "detail_weight"

Rows:

code detail_weight detail_color weight_unit
BK None None None
CHR None None None
DSK 12.5 oak kg

Scene 07

PostgreSQL only

The scalar form of subQueryColumn works on both dialects. The json and xml modes work only on PostgreSQL. On SQLite they raise UnsupportedFeatureError when the query is compiled. The aggr form follows lesson 06: everything works on SQLite except to_xml. bagItemColumn needs xpath, so it works only on PostgreSQL.

Example & screen notes

Slide title "Dialect support". Table:

Declaration PostgreSQL SQLite
subQueryColumn without mode yes yes
subQueryColumn(mode='json'), mode='xml' yes UnsupportedFeatureError
subQueryColumn(aggr=...) as .aggr(...) in lesson 06 as .aggr(...); to_xml refused
bagItemColumn yes UnsupportedFeatureError

Scene 08

Recap

A sub-query column is scalar, a JSON or XML collection, or one aggr call. An X column holds an XML document, and a bag item column reads one value from it on PostgreSQL. The next lesson covers layered recipes.

Example & screen notes

Three bullets:

  • subQueryColumn: scalar, mode='json'/'xml' (empty set: None), or aggr= with one .aggr call
  • dtype='X' holds an XML document; * leaves it out unless bagFields=True
  • bagItemColumn reads one item or attribute with xpath, on PostgreSQL