OData TC meeting #219 Thursday June 28, 2018

Acting chair: Ralf

Chat transcript from room: odatatc
2018-06-28 0800-1000 PDT

1. Roll call

1.1 Members present

    George Ericson (Dell)
    Hubert Heijkers (IBM)
    Mark Biamonte (Progress Software)
    Matthew Borges (SAP SE) a.k.a. Matt
    Michael Pizzo (Microsoft) a.k.a. Mike
    Ralf Handl (SAP SE)
    Ramesh Reddy (Red Hat)
    Stefan Hagen (Individual)
    Ted Jones (Red Hat)

Quorum achieved. Details cf. normative attendance sheet for this meeting (event_id=46262).

Notes taken by all and subsequently edited for readability by Stefan.

2. Approve agenda

Agenda is approved unchanged as published

3. Approve minutes from previous meeting(s)

3.1 Minutes from June 21, 2018 TC meeting #218

URL = https://www.oasis-open.org/committees/download.php/63317/odata-meeting-218_on-20180621-minutes.html

Minutes approved unchanged as published

4. Review action items [Link to Action item list]

4.1 Action items due

None

5. Issues

5.1 Data Aggregation: NEW or OPEN

5.1.1 ODATA-839 - A recursive hierarchy annotation may also contain a navigation property to the children nodes

Ralf: Proposal:

<ComplexType Name="RecursiveHierarchyType">
  <Property Type="Edm.PropertyPath" Name="NodeProperty" Nullable="false">
    <Annotation Term="Core.Description" String="Property holding the hierarchy node value"/>
  </Property>
  <Property Type="Edm.NavigationPropertyPath" Name="ParentNavigationProperty" Nullable="false">
    <Annotation Term="Core.Description" String="Property for navigating to the parent node"/>
  </Property>
  <Property Type="Edm.NavigationPropertyPath" Name="ChildrenNavigationProperty" Nullable="true">
    <Annotation Term="Core.Description" String="Property for navigating to the children nodes"/>
  </Property>
  <Property Type="Edm.PropertyPath" Name="DistanceFromRootProperty" Nullable="true">
    <Annotation Term="Core.Description" String="Property holding the number of edges between the node and the root node"/>
  </Property>
  <Property Type="Edm.PropertyPath" Name="IsLeafProperty" Nullable="true">
    <Annotation Term="Core.RequiresType" String="Edm.Boolean"/>
    <Annotation Term="Core.Description" String="Property indicating whether the node is a leaf of the hierarchy"/>
  </Property>
</ComplexType>

Ralf: Description

Consider a client processing entities according to their hierarchical structure imposed by the given hierarchy annotation.

If the client wants to retrieve the descendants for the next three levels, this could be accomplished with a single request 
that makes use of a (cyclic) navigation property pointing to the children of a node. 
Assuming this navigation property has name childrenNodes, and the corresponding navigation property annotated 
as ParentNavigationProperty has name parentNode, the request would be:

GET set(nodeId)?$expand=childrenNodes($levels=3;$select=parentNode/nodeId,nodeId)


In order to let clients detect the availability of such navigations, the existing RecursiveHierarchy term needs 
to be Extended as proposed.

Ralf: ODATA-839 is OPEN

Ralf:

Naming: ParentNavigationProperty
Proposals: ChildrenNavigationProperty or ChildNavigationProperty
Result is an array of children

George: children is more appropriate

Mike agrees: navigates to a collection of children

Hubert: I move to resolve ODATA-839 as proposed. George seconds.

Ralf: ODATA-839 is RESOLVED as proposed

5.1.2 ODATA-904 - Example 55: clarify groupby in combination with relationships of instance cardinality zero

Ralf: Description

Example 55:

GET ~/Customers?$apply=groupby((Country,Sales/Product/Name))

The example result does not contain an entry for country France. 
The example data contains a customer in France that does not have any sales.
The prose text before the example states expansion "in a left-outer-join fashion".
This would suggest that a result row is missing:

{ "@odata.id": null, "Country": "France", "Sales": [ { "Product":
{ "Name": null }
} ] }

This would be consistent with both the left-outer-join statement and the URL conventions 
for path expressions where properties of related entities are treated as null if no entity is related:
the left-outer join would produce a single Sales row containing only null values, including the related product and its name
this would create a ("France",null) group during aggregation
folding back into the original shape would create the additional entry

Ralf: Alternative:

{ "@odata.id": null, "Country": "France", "Sales": [ ] }

Hubert: prefers the alternative of an empty Sales array

George: agrees

Ralf: ODATA-904 is OPEN

[17:24] Hubert Heijkers (IBM): I move to resolve ODATA-904 with the updated proposal to use the alternative representation using the empty array. [17:24] George Ericson (Dell): Second

Ralf: Updated proposal:

Explicitly state what "left-outer-join fashion" means, i.e. not inventing dummy records with null values, 
instead use empty arrays that match the $expand structure:

{ "@odata.id": null, "Country": "France", "Sales": [ ] }

Fix the unbalanced curly braces in all result rows while we are at it.

Ralf: ODATA-904 is RESOLVED with the amended proposal

5.1.3 ODATA-905 - Example 67: result row for USA, Sugar missing, Example 68: last total should be 7

Ralf:

Example 67: transformation sequences are also useful inside groupby: 
To get the aggregated amount by only considering the top two sales amounts per product and county:

GET ~/Sales?$apply=groupby((Customer/Country,Product/Name,Currency/Code),
                      topcount(2,Amount)/aggregate(Amount with sum as Total))

results in

{
  "@odata.context":
     "$metadata#Sales(Customer(Country),Product(Name),Total,Currency(Code))",
  "value": [
    { "@odata.id": null, "Customer": { "Country": "Netherlands" },
      "Product": { "Name": "Paper" },
      "Total":  3, "Currency": { "Code": "EUR" }
    },
    { "@odata.id": null, "Customer": { "Country": "Netherlands" },
      "Product": { "Name": "Sugar" },
      "Total":  2, "Currency": { "Code": "EUR" }
    },
    { "@odata.id": null, "Customer": { "Country": "USA" },
      "Product": { "Name": "Coffee" },
      "Total": 12, "Currency": { "Code": "USD" }
    },
    { "@odata.id": null, "Customer": { "Country": "USA" },
      "Product": { "Name": "Paper" },
      "Total":  5, "Currency": { "Code": "USD" } 
    }
  ]
}

Ralf: Example data: http://docs.oasis-open.org/odata/odata-data-aggregation-ext/v4.0/cs02/odata-data-aggregation-ext-v4.0-cs02.html#_Toc435016565

Ralf: Sugar is indeed missing

Ralf: Example 68:

The function takes the name of a numeric property as a parameter, retains those entities that topcount also would retain, 
and replaces the remaining entities by a single aggregated entity, where only the numeric property has a defined value 
being the aggregated value over those remaining entities:
GET ~/Sales?$apply=groupby((Customer/Country,Product/Name),
                         aggregate(Amount with sum as Total))
                  /groupby((Customer/Country),
                           Self.TopCountAndBalance(Count=1,Property='Total'))

results in

{
  "@odata.context": "$metadata#Sales(Customer(Country),Product(Name),Total)",
  "value": [
    { "@odata.id": null, "Customer": { "Country": "Netherlands" },
      "Product": { "Name": "Paper" }, "Total":  3 },
    { "@odata.id": null, "Customer": { "Country": "Netherlands" },
      "Product": { "Name": "**Other**" }, "Total":  2 },
    { "@odata.id": null, "Customer": { "Country": "USA" },
      "Product": { "Name": "Coffee" }, "Total": 12 },
    { "@odata.id": null, "Customer": { "Country": "USA" },
      "Product": { "Name": "**Other**" }, "Total":  5 }
  ]
}

Ralf: Last total indeed is 7

Ralf: ODATA-905 is OPEN

Hubert: I move to resolve ODATA-905 as proposed. George seconds.

Ralf: ODATA-905 is RESOLVED as proposed

5.1.4 ODATA-909 - ABNF for pathPrefix should also allow qualifiedComplexTypeName

Ralf:

https://tools.oasis-open.org/version-control/browse/wsvn/odata/trunk/spec/ABNF/odata-aggregation-abnf.txt?op=diff&rev=789

Ralf: ODATA-909 is OPEN

Hubert: I move to resolve ODATA-909 as proposed. George seconds.

Ralf: ODATA-909 is RESOLVED as proposed

Hubert: I move to close ODATA-909 as applied. George seconds.

Ralf: ODATA-909 is CLOSED as applied

5.1.5 ODATA-943 - Correct broken link for Groupable Property

Ralf: Section 2.1 includes a definition for the term groupable property. The link behind it does not point to section 6.2.1 as expected.

Ralf: ODATA-943 is OPEN

Hubert: I move to resolve ODATA-943 as proposed. George seconds.

Ralf: ODATA-943 is RESOLVED as proposed

Hubert: I move to close ODATA-943 as applied. George seconds.

Ralf: ODATA-943 is CLOSED as applied

5.1.6 ODATA-944 - Clarify scope of property paths in transformations

Ralf: Description:

Property paths in transformations always relate to the structure of the immediate input set, 
which is either the collection identified by the request resource path or the output of the preceding transformation. 

For some transformations, this has been specified explicitly, e.g. in section 3.1 for the aggregate transformation: 
an expression valid in a $filter system query option on the input set. But not for all, e.g. section 3.12 does 
not make a statement for the expand transformation. 
In order to avoid any possible misinterpretation or confusion, the document should spell this out clearly.

Ralf: Proposal:

Add another sentence to section 3, at the end of the third para (enclosed with *): 

"So the actual (or relevant) structure of each intermediary result will resemble a projection of the original 
data model that could also have been formed using the standard system query options $expand and $select 
defined in [OData-Protocol], with dynamic properties representing the aggregate values. 
The parameters of set transformations allow specifying how the result instances are constructed from the input instances. 
This especially means that all property paths relate to the structure of the immediate input set."

Ralf: ODATA-944 is OPEN

Hubert: I move to resolve ODATA-944 as proposed. George seconds.

Ralf: ODATA-944 is RESOLVED as proposed

5.1.7 ODATA-945 - Correct examples 53 and 54

Ralf: Description

Example 53 requests aggregation of property path Sales/Amount, but the response shows the dynamic 
property Total nested inside Sales. Rightly, it should have been added to the type containing the 
original expression (section 3.1.1).

In example 54, the response payload shows the dynamic property Total nested within Sales. 
According to the request, which uses the path expression Sales/Amount for aggregation, 
the Total property should be at the top level.

Ralf: Proposal:

Example 53: Replace response payload with

{
"@odata.context": "$metadata#Products(Name,Total)",
"value": [
{ "@odata.id": null, "Name": "Coffee", "Total": 12 }
,
{ "@odata.id": null, "Name": "Paper", "Total": 8 }
,
{ "@odata.id": null, "Name": "Pencil", "Total": null }
,
{ "@odata.id": null, "Name": "Sugar", "Total": 4 }
]
}

Example 54: Modify request to properly reflect the nesting shown in the response: 

GET ~/Products?$apply=groupby((Name,Sales/Currency/Code),
aggregate(Sales(Amount with sum as Total)))

Ralf: Current text:

Example 53: 

GET ~/Products?$apply=groupby((Name),

                              aggregate(Sales/Amount with sum as Total))

results in

{
  "@odata.context": "$metadata#Products(Name,Sales(Total))",
  "value": [
    { "@odata.id": null, "Name": "Coffee", "Sales": [ { "Total":   12 } ] },
    { "@odata.id": null, "Name": "Paper",  "Sales": [ { "Total":    8 } ] },
    { "@odata.id": null, "Name": "Pencil", "Sales": [ { "Total": null } ] },
    { "@odata.id": null, "Name": "Sugar",  "Sales": [ { "Total":    4 } ] }
  ]
}

Note that the base set of the request is Products, so there is a result item for 
product Pencil even though there are no sales item. As aggregate returns exactly 
one result item even if there are no items to be aggregated, the Sales navigation 
propertys value is an array with one element representing the sum over no input values, which is null.

Example 54: careful observers will notice that the above amounts have been aggregated 
across currencies, which is semantically wrong. 
Yet it is the correct response to the question asked, so be careful what you ask for. 
The semantically meaningful question

GET ~/Products?$apply=groupby((Name,Sales/Currency/Code),
                              aggregate(Sales/Amount with sum as Total))

results in

{
  "@odata.context": "$metadata#Products(Name,Sales(Total,Currency(Code)))",
  "value": [
    { "@odata.id": null, "Name": "Coffee",
      "Sales": [ { "Total": 12, "Currency": { "Code": "USD" } } ] },
    { "@odata.id": null, "Name: "Paper",
      "Sales": [ { "Total":  3, "Currency": { "Code": "EUR" } },
                 { "Total":  5, "Currency": { "Code": "USD" } } ] },
    { "@odata.id": null, "Name: "Pencil",
      "Sales": [] },
    { "@odata.id": null, "Name: "Sugar",  
      "Sales": [ { "Total":  2, "Currency": { "Code": "EUR" } },
                 { "Total":  2, "Currency": { "Code": "USD" } } ] }
  ]
}

Note that navigation properties are "expanded" in a left-outer-join fashion, 
starting from the target of the aggregation request, before grouping the entities for aggregation. 
Afterwards the results are folded back to match the cardinality of the navigation properties.

Ralf: Correct request URL for 54 is

GET ~/Products?$apply=groupby((Name,Sales/Currency/Code),
aggregate(Sales(Amount with sum as Total)))

Ralf: ODATA-945 is OPEN

Hubert: we have to explain more about the differences

Ralf:

Plain paths with / work like compute: pull arguments from related entities, place result "at the top"
Parentheses can be used to place the result within the related/nested structure

Ralf: Slight difference:

GET ~/Products?$apply=groupby((Name,Sales/Currency/Code),aggregate(Sales(Amount with sum as Total)))
GET ~/Products?$apply=groupby((Name,Sales/Currency/Code),aggregate(Sales/Amount with sum as Total))

Ralf: 3.1.1 Keyword as

Aggregate expressions can define an alias using the as keyword, followed by a SimpleIdentifier (see [OData-CSDL, section 17.2]).

The alias will introduce a dynamic property in the aggregated result set. The introduced dynamic property is added to the 
type containing the original expression or custom aggregate. 
The alias MUST NOT collide with names of declared properties, custom aggregates, or other aliases in that type.

Ralf:

Sales/Amount with sum as Total
Sales(Amount with sum as Total)

George: be more explicit in the definition and not define by example

Ralf:

First example: in top context we are interested in the Amount
Second example: we are interested in the Sales, and within the Sales in the Amount
Need to revise the specification text: section 3.1.1

5.1.8 ODATA-967 - Chapter 3: remove restriction for input type of bound functions

Ralf: Description:

When generalizing $apply from "entities" to "instances" of any type we forgot to reformulate the paragraph on 
service defined "custom functiions".

Ralf: Proposal:

Prose spec chapter 3, second-to-last paragraph new:

Service-defined bound functions that take a collection as their binding parameter MAY be used as set transformations 
within $apply if the type of the binding parameter matches the type of the result set of the preceding transformation. 
If it returns a collection, further transformations can follow the bound function. 
The parameter syntax for bound function segments is identical to the parameter syntax for bound functions in 
resource path segments or $filter expressions. See section 7.6 for an example.

old:

Service-defined bound functions that take an entity set as their binding parameter MAY be used as set transformations 
within $apply if the type of the binding parameter matches the type of the result set of the preceding transformation. 
If it returns an entity set, further transformations can follow the bound function. 
The parameter syntax for bound function segments is identical to the parameter syntax for bound functions in 
resource path segments or $filter expressions. See section 7.6 for an example.

ABNF new:

customFunction = namespace "." ( entityColFunction / complexColFunction / primitiveColFunction ) functionExprParameters

old:

customFunction = namespace "." entityColFunction functionExprParameters

George: use "structured values" instead of "instances"

Ralf: ODATA-967 is OPEN

Hubert: I move to resolve ODATA-967 as proposed. George seconds.

Ralf: ODATA-967 is RESOLVED as proposed

5.1.9 ODATA-968 - 3.1.4: add example for from in aggregate() with multiple arguments

[18:23] Ralf Handl (SAP SE): https://issues.oasis-open.org/browse/ODATA-968

Ralf: Description

Explain how in aggregate() with multiple arguments the "from" keyword can be resolved into a groupby/aggregate 
sequence by initially grouping by all "from" properties appearing in any argument.

Will probably work only if all "from" chains are compatible, i.e. there exists a "super-chain" so that 
any from chain is a sub-chain of this super-chain.

1) from A from B from D,from A from C from D works with super-chain from A from B from C from D (B and C can be swapped to get another possible super-chain.

2) from A from B, from B from A won't work

Ralf:

GET ~/Sales?$apply=aggregate(Amount with sum as DailyAverage from Time with average)
GET ~/Sales?$apply=groupby((Time),aggregate(Amount with sum as Total)) 
                  /aggregate(Total with average as DailyAverage)
GET ~/Sales?$apply=groupby((Time/Date),aggregate(Amount with sum as Total)) 
                  /aggregate(Total with average as DailyAverage)

Ralf: should we keep this in, seeing how much trouble we had reconstructing the non-intuitive meaning of the "from" syntax?

Ralf: ODATA-968 is OPEN

Hubert: park this and reconstruct together with Gerald why we invented the "from" syntax

5.1.10 ODATA-971 - 7.1: Add example for grouping by single-valued navigation property

Ralf: Description:

Section 7.1 defines the result of grouping by a single-valued navigation property without giving an example.

Any example in this section that groups by the key property of a related entity could be rephrased to group by the 
navigation property instead. Here's how example 52 would look like:

GET ~Sales?$apply=groupby((Customer,Product))
and results in
{
"@odata.context": "$metadata#Sales(Customer(Name,ID),Product(Name))",
"value": [
{ "@odata.id": null, "Customer@odata.navigationLink": "Customers('C1')", "Product@odata.navigationLink": "Products('P2')" }
,
{ "@odata.id": null, "Customer@odata.navigationLink": "Customers('C1')", "Product@odata.navigationLink": "Products('P3')" }
,
{ "@odata.id": null, "Customer@odata.navigationLink": "Customers('C1')", "Product@odata.navigationLink": "Products('P1')"}
,
{ "@odata.id": null, "Customer@odata.navigationLink": "Customers('C2')", "Product@odata.navigationLink": "Products('P2')" }
,
{ "@odata.id": null, "Customer@odata.navigationLink": "Customers('C2')", "Product@odata.navigationLink": "Products('P3')" }
,
{ "@odata.id": null, "Customer@odata.navigationLink": "Customers('C3')", "Product@odata.navigationLink": "Products('P3')"}
},

{ "@odata.id": null, "Customer@odata.navigationLink": "Customers('C3')", "Product@odata.navigationLink": "Products('P1')" }
]
}

Ralf:

GET ~Sales?$apply=groupby((Customer/ID,Product/ID))
and results in
{
"@odata.context": "$metadata#Sales(Customer(ID),Product(ID))",
"value": [
{ "@odata.id": null, "Customer":{"ID": "C1"}, "Product":{"ID": "P2"} },

Ralf: ODATA-971 is OPEN

Ralf: Response seems somewhat non-intuitive but can be derived from JSON Format rules

6. Next meetings

Thursday July 5, 2018 during 8-10 am PDT (17:00-19:00 CEST): 
    no meeting due to public holiday in the US on July 4

Agreed next meetings:

Thursday July 12, 2018 during 8-10 am PDT (17:00-19:00 CEST)
Thursday July 19, 2018 during 8-10 am PDT (17:00-19:00 CEST)

Hubert: cannot attend July 12

7. AOB and wrap up

None.

Meeting adjourned by chair.