OData TC meeting #233 Thursday October 11, 2018

Acting chair: Ralf

Chat transcript from room: odatatc
2018-10-11 0800-1000 PDT

1. Roll call

1.1 Members present

    George Ericson (Dell)
    Hubert Heijkers (IBM)
    Mark Biamonte (Progress Software)
    Matthew Borges (SAP SE) a.k.a. Matt
    Michael Pizzo (Microsoft) a.k.a. Mike
    Ralf Handl (SAP SE)
    Ramesh Reddy (Red Hat)
    Stefan Hagen (Individual)
    Ted Jones (Red Hat)

Quorum achieved. Details cf. normative attendance sheet for this meeting (event_id=46277).

Notes taken by all and subsequently edited for readability by Stefan.

2. Approve agenda

Ralf:

New issue: ODATA-1241 - Extend in operator to allow comparison with multiple properties
Hubert's leave of absence

Agenda is approved with the above changes.

3. Approve minutes from previous meeting(s)

3.1 Minutes from October 04 2018 Meeting #232

URL = https://www.oasis-open.org/committees/download.php/64018/odata-meeting-232_on-20181004-minutes.html

Minutes are approved unchanged as published.

4. Review action items [Link to Action item list]

4.1 Action items due

None.

4.2 Action items upcoming

4.2.1 #0037 Concept for Google Protocol Buffers as a data format Hubert Heijkers 2018-09-27

Nothing noted

4.3 Action items pending

4.3.1 #0036 Register the OData- headers and preferences with IANA Mark Biamonte 2018-07-26

Nothing noted

5. Leave of absence for Hubert

Ralf: No objections, Hubert will be absent for 3 weeks starting October 15th

6. Issues

6.1 sssss

6.1.1 ODATA-1212 - pull request https://github.com/oasis-tcs/odata-vocabularies/pull/18

Ralf:

<Term Name="AllowedTerms" Type="Collection(Core.QualifiedTermName)" AppliesTo="Term Property">
  <Annotation Term="Core.Description"
    String="Annotate a property or term of type Edm.AnnotationPath to restrict the terms that can be targeted by the path." />
  <Annotation Term="Core.LongDescription"
    String="Annotation path expressions assigned to the annotated term or property are intended to resolve to annotations with one of the listed terms. For forward compatibility, clients should be prepared for the annotation to reference terms besides those listed." />
  <Annotation Term="Core.RequiresType" String="Edm.AnnotationPath" />
</Term>

Ralf:

Change description to "term or property of a structured term"
Annotate a term of type Edm.AnnotationPath, or a property of type Edm.AnnotationPath 
that is used within a structured term, to restrict the terms that can be targeted by the path.

Ralf: Example:

"RepresentAs":"@UI.LineItem#compact"
Annotation path expressions are intended to resolve to annotations with one of the listed terms..
Annotation path expressions are intended to end in a path segment with one of the listed terms...

Ralf: More verbose example:

"@UI.Visualization":{
  "RepresentAs":"@UI.LineItem#compact",
  ...
}
"@UI.LineItem#compact": {...},
"@UI.LineItem#informative": {...},
"@UI.Chart#bar": {...},

Ralf: New LongDescription:

The annotation path expression is intended to end in a path segment with one of the listed terms.
For forward compatibility, clients should be prepared for the annotation to reference terms besides those listed.

Ralf: Pull request adapted accordingly

Ralf: New definition:

"AllowedTerms": {
    "$Kind": "Term",
    "$Collection": true,
    "$Type": "Core.QualifiedTermName",
    "$AppliesTo": [
        "Term",
        "Property"
    ],
    "@Core.Description": "Annotate a term of type Edm.AnnotationPath, or a property of type Edm.AnnotationPath that is used within a structured term, to restrict the terms that can be targeted by the path.",
    "@Core.LongDescription": "The annotation path expression is intended to end in a path segment with one of the listed terms. For forward compatibility, clients should be prepared for the annotation to reference terms besides those listed.",
    "@Core.RequiresType": "Edm.AnnotationPath"
}

Ralf: I move to merge pull request https://github.com/oasis-tcs/odata-vocabularies/pull/18 Hubert seconds.

Ralf: No objection, motion passes

6.2 NEW or OPEN

6.2.1 ODATA-1241 - Support common expressions as values in URL-JSON

Ralf: Ralf Handl created ODATA-1241:

---------------------------------

             Summary: Extend in operator to allow comparison with multiple properties
                 Key: ODATA-1241
                 URL: https://issues.oasis-open.org/browse/ODATA-1241
             Project: OASIS Open Data Protocol (OData) TC
          Issue Type: New Feature
          Components: URL Conventions
    Affects Versions: V4.01_CS01
         Environment: Proposed
            Reporter: Ralf Handl
             Fix For: V4.01_CS02

Ralf: Description

The in operator allows comparing a property to a list of values, e.g.

$filter=Name in ('Milk', 'Cheese')
For multi-part keys it would be helpful to allow comparison of a tuple of properties to a list of value tuples:

$filter=(Firstname,Lastname) in (('John', 'Doe'),('Jane','Smith'))
 Note that this is natively supported by some databases:

https://docs.oracle.com/cd/B19306_01/server.102/b14200/conditions013.htm
https://dev.mysql.com/doc/refman/8.0/en/comparison-operators.html#function_in

Ralf: Proposal:

Allow using "property tuples" as the left operand of the in operator, and allow "tuples of tuples" as its right operand:

$filter=(Firstname,Lastname) in (('John', 'Doe'),('Jane','Smith'))
 Alternative syntax proposal with array notation:

$filter=[Firstname,Lastname] in [['John', 'Doe'],['Jane','Smith']]

Ralf: Name in (["Milk", "Cheese"])

Mike:

$filter=[Firstname,Lastname] in (['John', 'Doe'],['Jane','Smith'])
$filter=['Jane','Smith'] in (['John', 'Doe'],['Jane','Smith'])

Ralf:

["Jane","Smith"] in (["John", "Doe"],["Jane","Smith"])

Mike:

[1+2]
==[3]
Do we just allow expressions in array constructors?

Ramesh: In SQL (Firstname,Latname) is expression that represents a array, I am not sure how that can be used in comparisions

Ralf:

> let n = [1+2]
undefined
> n
[ 3 ]
>
> [n]
[ [ 3 ] ]
>

Mike: Taking one step further, do we allow expressions in JSON constructs in the URL?

{"foo":1+2}
=={"foo":3}
{"foo":lastName}
Customers?$filter=BusinessAddress eq {"Street":"123","City":"abc","Region":"xyz","ZipCode":HomeAddress/ZipCode}
better to extend URL-JSON generically than add a one-off construct for left side of "in"

Hubert: use commonExpression and not just allow memberExpressions (paths to properties)

Ralf:

{Street:"Highway"}
Shortcut for {"Street":"Highway"}?
Or means: take current value of property Street and use it as the name in the name-value pair?

Ralf: Adapted proposal:

llow using common expressions as "values" in URL-JSON. This will allow to construct an array from properties 
as the left operand of the in operator:

$filter=[Firstname,Lastname] in (["John","Doe"],["Jane","Smith"])
Note: a list of arrays is already allowed as the right operand.

Ralf: ODATA-1241 is OPEN

Ralf: Rename issue

Mike: Renamed to:"Support common expressions as values in URL-JSON"

Ralf: Already allowed:

$filter=["Joe","Miller"] in (["John","Doe"],["Jane","Smith"])

Mike: I move to resolve ODATA-1241 as proposed. George.

Ralf: ODATA-1241 is RESOLVED as proposed

6.2.2 ODATA-1240 - Chapter 2: wrong description of how to split URLs into syntax components

Ralf: Description

The text currently says

Split undecoded URL into components scheme, hier-part, query, and fragment at first ":", then first "?", and then first "#"
The query part is optional, and the fragment may contain the question mark character, so the descriptio is incorrect. 

Don't try to capture the RFC3986 ABNF into concise and correct text.
[18:16] Ralf Handl (SAP SE): Proposal:
Shorten text to

Split undecoded URL into components scheme, hier-part, query, and fragment

Ralf: ODATA-1240 is OPEN

Mike: a little more context:

2     URL Components
A URL used by an OData service has at most three significant parts: the service root URL, resource path and query options. Additional URL constructs (such as a fragment) can be present in a URL used by an OData service; however, this specification applies no further meaning to such additional constructs.
Example 2: OData URL broken down into its component parts:
http://host:port/path/SampleService.svc/Categories(1)/Products?$top=2&$orderby=Name
\______________________________________/\____________________/ \__________________/
                  |                               |                       |
          service root URL                  resource path           query options
Mandated and suggested content of these three significant URL components used by an OData service are covered in sequence in the three following chapters.
OData follows the URI syntax rules defined in [RFC3986] and in addition assigns special meaning to several of the sub-delimiters defined by [RFC3986], so special care has to be taken regarding parsing and percent-decoding.
[RFC3986] defines three steps for URL processing that MUST be performed before percent-decoding: 
       Split undecoded URL into components scheme, hier-part, query, and fragment at first ":", then first "?", and then first "#" 
       Split undecoded hier-part into authority and path
       Split undecoded path into path segments at "/"
After applying these steps defined by RFC3986 the following steps MUST be performed:
       Split undecoded query at "&" into query options, and each query option at the first "=" into query option name and query option value
       Percent-decode path segments, query option names, and query option values exactly once
       Interpret path segments, query option names, and query option values according to OData rules

Mike: Probably better to talk about hier-part in the parsing section, to avoid confusion with resource path in first part. Here is the full text of section 2:

2     URL Components
A URL used by an OData service has at most three significant parts: the service root URL, resource path and query options. Additional URL constructs (such as a fragment) can be present in a URL used by an OData service; however, this specification applies no further meaning to such additional constructs.
Example 2: OData URL broken down into its component parts:
http://host:port/path/SampleService.svc/Categories(1)/Products?$top=2&$orderby=Name
\______________________________________/\____________________/ \__________________/
                  |                               |                       |
          service root URL                  resource path           query options
Mandated and suggested content of these three significant URL components used by an OData service are covered in sequence in the three following chapters.
OData follows the URI syntax rules defined in [RFC3986] and in addition assigns special meaning to several of the sub-delimiters defined by [RFC3986], so special care has to be taken regarding parsing and percent-decoding.
[RFC3986] defines three steps for URL processing that MUST be performed before percent-decoding: 
       Split undecoded URL into components scheme, hier-part, query, and fragment at first ":", then first "?", and then first "#" 
       Split undecoded hier-part into authority and path
       Split undecoded path into path segments at "/"
After applying these steps defined by RFC3986 the following steps MUST be performed:
       Split undecoded query at "&" into query options, and each query option at the first "=" into query option name and query option value
       Percent-decode path segments, query option names, and query option values exactly once
       Interpret path segments, query option names, and query option values according to OData rules
The OData rules are defined in this document and the [OData-ABNF]. Note that the ABNF is not expressive enough to define what a correct OData URL is in every imaginable use case. This specification document defines additional rules that a correct OData URL MUST fulfill. In case of doubt on what makes an OData URL correct the rules defined in this specification document take precedence. Note also that the rules in [OData-ABNF] assume that URLs and URL parts have been percent-encoding normalized as described in section 6.2.2.2 of [RFC3986] before applying the grammar to them, i.e. all characters in the unreserved set (see rule unreserved in  [OData-ABNF]) are plain literals and not percent-encoded. For characters outside of the unreserved set that are significant to OData the ABNF rules explicitly state whether the percent-encoded representation is treated identical to the plain literal representation. This is done to make the input strings in the ABNF test cases more readable.
For example, one of these rules is that single quotes within string literals are represented as two consecutive single quotes.
Example 3: valid OData URLs:
http://host/service/People('O''Neil')
http://host/service/People(%27O%27%27Neil%27) 
http://host/service/People%28%27O%27%27Neil%27%29
http://host/service/Categories('Smartphone%2FTablet')
Example 4: invalid OData URLs:
http://host/service/People('O'Neil') 
http://host/service/People('O%27Neil') 
http://host/service/Categories('Smartphone/Tablet') 
The first and second examples are invalid because a single quote in a string literal must be represented as two consecutive single quotes. The third example is invalid because forward slashes are interpreted as path segment separators and Categories('Smartphone is not a valid OData path segment, nor is Tablet').

Mike: I would do a 2.1 as Parsing rules and a 2.2 as OData ABNF, keep the three parts of an OData url as top-level and use hier-part in section 2.1

Mike:

1) Put parsing rules in a 2.1 subsection, and OData ANBF in a 2.2 subsection.
2) In Parsing rules, shorten text of first step to:
 * Split undecoded URL into components scheme, hier-part, query, and fragment

Hubert: I move to resolve ODATA-1240 as per the amended proposal. Mike.

Ralf: ODATA-1240 is RESOLVED with the amended proposal

6.2.3 ODATA-1239 - Define a mechanism to distinguish between inserted and updated entities in a Delta Response

Ralf: ODATA-1239 is OPEN

Mike: concern that delta until now is idempotent, so you can apply the same delta twice, giving the same result

Ralf: Indicating that an entity is inserted versus updated might tempt implementations to break this

Hubert: make sure that we describe that entities in delta payloads are UPSERT-ed and that UPSERT rules are applied

George: Use of & is defined in https://tools.ietf.org/html/rfc6570

6.2.4 ODATA-1238 - Clarifications for select-list in ContextUrl

Mike: Currently we say:

Context URL templates:
  {context-url}#{entity-set}{/type-name}{select-list}/$entity
  {context-url}#{singleton}{select-list}
  {context-url}#{type-name}{select-list}
If a single entity contains a subset of properties, the parenthesized comma-separated list of the selected defined or dynamic properties, 
navigation properties, functions, and actions is appended to the entity-set after an optional type-cast segment and prior to appending /$entity. 
If the response is not bound to a single entity set, the select-list is instead appended to the 
Unknown macro: {type-name} 
of the returned entity. 
The shortcut * represents the list of all structural properties. Properties defined on types derived from the type of the entity set 
(or type specified in the type-cast segment if specified) are prefixed with the qualified name of the derived type as defined in [OData-ABNF]. 
Note that expanded properties are implicitly selected.
OData 4.01 responses MAY use the shortcut pattern namespace.* to represent the list of all bound actions or functions available for 
entities in the collection, see system query option $select.

Mike: note the statement:

"Note that expanded properties are implicitly selected."

Mike: Proposal 1:

Remove the statement "Note that expanded properties are implicitly selected."  We do not expect that the navigation link is 
implicitly included just because you expanded the nav prop.

Ralf: Agreed

Mike:

Customers?$select=name,Orders =>contextUrl= Customers(name,Orders) => result contains nav link for Orders
Customers?$select=name&$expand=Orders => contextUrl Customers(name,Orders()) => results contains expanded orders

Ralf: Do we need the Orders() part?

Mike:

Customers?$select=name&$expand=Orders($select=Amount) => ContextUrl = Customers(name,Orders(Amount)) => results contains amount of expanded orders
Customers?$select=name,Orders&$expand=Orders => ContextUrl = Customers(name,Orders,Orders()) => includes both nav link and expanded orders

Mike: Next issue:

If my contextUrl contains only expanded properties, what is selected? i.e., ContextUrl => Customers(Orders())
http://host/service/Employees(1)/Sales.Manager?
$expand=DirectReports($select=FirstName,LastName;$levels=4)
http://host/service/$metadata
#Employees/Sales.Manager(DirectReports+(FirstName,LastName))/$entity

Mike: This implies that if I only have expanded properties in the contextUrl, I am implicitly selecting all properties of the outer resource

Ralf: Agreed

Mike:

If we do Customers?$select=Address/City, Address/State => ContextUrl = Customers(Address/City,Address/State)
Alternately, you can say Customers?$select=Address(City,State) => ?
Assertion: this is just syntactic sugar over the former; the contexturl should be the same as the first example.

Mike: Next issue: How are functions/actions represented in ContextUrl?

Ralf: Agreed, with slight regret

Mike:

1) Always qualified with (namespace?alias?)

Mike: in the abnf:

select         = ( "$select" / "select" ) EQ selectItem *( COMMA selectItem )
selectItem     = STAR
               / allOperationsInSchema 
               / [ ( qualifiedEntityTypeName / qualifiedComplexTypeName ) "/" ] 
                 ( selectProperty
                 / qualifiedActionName  
                 / qualifiedFunctionName  
                 )
selectProperty = primitiveProperty  
               / primitiveColProperty [ OPEN selectOptionPC *( SEMI selectOptionPC ) CLOSE ]
               / navigationProperty
               / selectPath [ OPEN selectOption *( SEMI selectOption ) CLOSE
                            / "/" selectProperty 
                            ]
qualifiedFunctionName = namespace "." function [ OPEN parameterNames CLOSE ]

Ralf:

; The parameterNames uniquely identify the bound function overload
; Necessary only if it has overloads
optionallyQualifiedActionName   = [ namespace "." ] action
optionallyQualifiedFunctionName = [ namespace "." ] function [ OPEN parameterNames CLOSE ]

Mike: In contextUrl functions w/o parens mean all overloads, functions w/parens mean specific overloads. actions never have parens.

Ralf: Note: bug in ABNF - case of no non-binding-parameters not covered

7. Next meetings

Agreed next meetings:

Thursday October 18, 2018 during 8-10 am PDT (17:00-19:00 CEST)
Thursday October 25, 2018 during 8-10 am PDT (17:00-19:00 CEST)

8. AOB and wrap up

Meeting adjourned by chair.