muon, mu object notation

Mu object notation, Muon, is a permissive textual format for structured data, which is also a substrate for describing Mu source code via additional higher-level parsing. Muon is a superset of Mu source code.

Muon structure is designed to reflect human visual perception of source code and to be loosely compatible with other structured formats (e.g. json).

Its top-level entity is a "sequence", <seq>. A standalone Muon file is a <seq> (maybe empty one).

whitespace and newline handling

Muon is indentation-insensitive, but it is sensitive to newlines, \n. Any ebnf rules defined below:

• if whitespace is forbidden explicitly, no whitespace between productions
• otherwise: if removing whitespace would change the meaning (e.g. foo bar -> foobar, <sym> instead of <sym> <sym>), require at least one whitespace character.
• otherwise: allow zero or more whitespace characters between productions (e.g. foo(bar) is <sym> <tuple>, foo//comment == foo //comment, etc). A <sym> and a <punct> are disjoint character classes and cannot merge, so a:b == a : b — both are <sym> a, <punct> :, <sym> b. The one place whitespace does change the reading is . (see <prefixed> / <punct>): whether it is a sigil is decided by whether an item is glued to its right, so x.0 is <sym> <prefixed> (a .-prefixed 0) while x .0 is the same two items and x . 0 is <sym> <punct> <sym> (a bare .). Likewise ... is one nested item (.(.(.))) but .. . splits at the space into two items.

Allowed whitespace characters are (\x20), \r, \t.

\r\n in muon text is treated as a whitespace character \r, followed by <sep>. \r\n <-> \n conversions do not change the parsed muon structure.

To convert multi-line representation into one-line representation (e.g. for minification):

• split multiline comments into sequences of comments like /* first line*//* second line*/, preserving order of the comment lines.
• convert oneline comments like //comment into /*comment*/. The trailing newline is not part of the comment and is converted to a comma as usual.
• replace all other newlines with commas

grammar

<seq>

A <seq> consists of zero or more "chains", <chain>, separated by separator <sep> (can be a newline or a comma interchangeably, to format long chains as lines and short chains as a comma-separated lists in one line).

Zero chains is a valid empty seq.

Preceding/trailing/duplicate separators are valid, to allow for blank lines visually, hanging commas in lists, newline before EOF, etc. The content of a sequence is defined by chains inside of it, whereas separators only delimit chains. Chains by their definition are non-empty, but empty space between <sep> is allowed and does not produce a chain.

(,,,)      // no chains inside
(a b, c)   // two chains: `a b`, `c`
(, ,,a b, , c,,)  // same

In source code, a sequence corresponds to the inner content of a module, a tuple, a block, a list, etc.

<sep> ::= "\n" | ","
<seq> ::= <chain>? (<sep> <chain>?)*

<chain>

A chain, <chain> is a juxtaposition of one or more <item>s. If literal concatenation changes meaning, items should be separated by one or more whitespace, otherwise zero or more whitespace is allowed between items.

• e.g. chains like foo bar/foo bar contain two different symbols, but foobar is one symbol.
• e.g. f(x) == f (x ) == f ( x )

Visually, a chain is a "sentence" of visually atomic "items", juxtaposition of <item>s.

A chain cannot be empty (but a place between two <sep> in a <seq> can be, this does not produce a chain).

In source code, a chain is roughly an "expression"/"statement". Source code can treat multiple adjacent chains as one expression (e.g. to allow visual separation of declaration and definition of a function, to spread long expressions) according to simple unambiguous rules.

<chain> ::= <item>+

<item>

An item is either a visually atomic entity like a number/string literal, a symbolic name, or something clearly delineated by an opening and a closing element, e.g. it may be:

• a tuple is a <seq> wrapped in parenthesis
• a list is a <seq> wrapped in brackets
• a block is a <seq> wrapped in braces
• a prefixed item is another item marked with a leading sigil

A prefixed item is a leading sigil (& or .) immediately followed (no whitespace) by any <item>. Muon assigns it no meaning — it is structure only, just like a : (<punct>) or -1 (<sym>) — leaving the interpretation to higher levels (e.g. Elly uses &<item> to introduce a binding and .<item> for a symbol / projection; see elly-spec.md). The . sigil applies only when the dot is glued to an item; a dot not followed by an item is a <punct> (see <prefixed> and <punct> below).

<item> ::= 
  | <comm>     (* comments are valid items and are preserved *)
  | <str>      (* string literals *)
  | <sym>      (* numbers, names, +/-/. operators, etc *)
  | <punct>    (* punctuation standing between items, e.g. ":" *)
  | <tuple>    (* parenthesis-wrapped <seq> *)
  | <list>     (* bracket-wrapped <seq> *)
  | <block>    (* brace-wrapped <seq> *)
  | <prefixed> (* an item marked with a leading sigil *)

<prefixed>

A prefixed item is a sigil directly attached to an item. The sigils are & and .. There is no whitespace between the sigil and the item.

The two sigils differ in how they handle a dangling sigil (one not directly followed by an item):

• & must be followed by an item; a dangling & is a syntax error.
• . is a sigil only when directly followed by an item (an item-opening character: a letter, digit, ", ( / [ / {, &, : / =, or another .). A . followed by a closing character — whitespace, a <sep> (, / \n), a closer ) / ] / }, or end of input — is not a sigil but a standalone <punct> (see <punct>). The decision is purely right-gluing: .g is a prefixed item, . / ., / .) are punctuation. Because another . counts as an item, dots nest: a run of N dots is (N−1) .-prefixed items around a final <punct> ., so ... is .(.(.)) — one item, distinct from .. . (which is .(.) then a separate <punct> .).

(* no whitespace between sigil and item *)
<prefixed> ::= <sigil> <item>
<sigil>    ::= "&" | "."   (* "." only when right-glued to an item; else <punct> *)

<comm>

Comments are valid atomic items which preserve their inner unicode textual content literally. They are preserved at the Muon level and usually are ignored by higher level semantics. This still allows to reuse them in tools like formatters.

Visually, comments are C-style // one-line and /*-*/ multiline comments.

Comments are not nested:

• // any // after is just text
• /* a slash-star, /* here is just text, terminated by the first star-slash, */

<comm> ::=
  | "//" (any character except "\n")*
  | "/*" (any character sequence without "*/") "*/"

Newlines inside multiline comments are not <sep> and are preserved.

<str>

String literals are wrapped in ". They contain any unescaped Unicode characters (except a literal newline or unescaped ") or escape sequences starting with \.

Escape sequences are: \", \n, \t, \r, \\. Any other character after \ is currently a syntax error, open to changes in the future.

<str> ::= "\"" (any utf-8 character except unescaped '"' and literal newline)* "\""

TODO: check json strings compatibility TODO: \u{NNNN}, \xNN encoding?

<sym>

Symbols are any (non-whitespace) sequences of one or more of:

• latin alphanumeric characters
• _
• - | +

A symchar is exactly a character that can occur inside a single atomic literal — a number or an identifier. That is why -, + are symchars (-1, +123) but : and . are not: neither occurs inside a literal, so : is a <punct> (below) and . is a sigil-or-punct (see <prefixed> and <punct>). A consequence is that a . breaks a symbol: foo.bar is not one symbol but the chain foo .bar (<sym> then a .-prefixed item), and 3.14 is the chain 3 .14.

They can encode:

• identifiers / plain words (e.g. x, foo, foo_bar)
• binary/decimal/hexadecimal numbers (e.g. 0, -1, +123, 0xf123)
• operators like +, ++, --
• mixed identifiers like --1, 1a, 1+1, +a: the meaning is decided by higher level parsers.

(* no whitespace *)
<sym> ::= <symchar>+
<symchar> ::= "0".."9" | "A".."Z" | "a".."z"
            | `_` | `-` | `+`

TODO: +, - genuinely belong to both classes (they are operators too), but they stay symchars so numeric literals stay whole. A consequence is that a token cannot straddle the two classes by maximal munch: :+ is <punct> : then <sym> +, and a future -> would not munch as one token. Cross-class multi-char tokens (like ->, or / once it joins a class) would be handled as explicit lexical exceptions, the way // and /* already are. (. used to be a symchar too, for 3.14 / foo.bar; it has been split out as a sigil/punct — float-literal notation is deferred until Elly grows a Num type.)

<punct>

Punctuation, <punct>, is a maximal run of one or more punctchars — non- whitespace characters that stand between items rather than occurring inside a literal. The punctchars are : and =. Like <sym>, Muon assigns <punct> no meaning: they are just structure, read by higher levels (e.g. Elly reads key : value as a map entry, x = 1 as a binding, or a future => as an arrow; see elly-spec.md).

Punctuation is maximal munch like <sym>, so :: and == are each a single <punct> (write : : / = = for two), and — since : and = share the class — := munches as one punct too. Future punctchars (>, …) would let => fall out the same way, with no per-operator rule.

A . is also a <punct>, but a special one: it is a standalone single-character punct and is not a punctchar, so it never joins the :/= run (:. is <punct> : then <punct> ., not one token). A . is this punct only when it is not glued to an item; a . directly followed by an item is instead the sigil of a <prefixed> (see <prefixed>). Muon gives the . punct no meaning either — a higher level (Elly) reads a spaced . as an application / composition combinator.

(* no whitespace *)
<punct> ::= <punctchar>+ | "."
<punctchar> ::= ":" | "="    (* reserved to grow: > | ~ ! ? ... *)

<tuple>

A tuple is a <seq> wrapped in parenthesis. It's used for grouping chains together. Its exact semantics is defined by higher levels (e.g. in Elly).

Empty tuple, ()/( )/(,,)/etc, is valid and wraps an empty sequence.

Since the content of a tuple is a <seq>, they can spread over multiple lines:

(a, (b, c))  
// is equivalent to
(a
(b,c))
// is equivalent to
(
  a
  (b, c)
)
// is equivalent to
(
  a
  (
    b
    c
  )
)

Definition:

<tuple> ::= "(" <seq> ")"

<list>

A list is a <seq> wrapped in brackets. Like <tuple>, it only groups chains; Muon assigns it no meaning, leaving it to higher levels (e.g. Elly reads a list as a map's computed key or, juxtaposed after a map, as bracket access; see docs/elly-spec.md).

Empty list, []/[ ]/[,,]/etc, is valid and wraps an empty sequence. Its content being a <seq>, a list can spread over multiple lines exactly like a tuple. Since juxtaposition without whitespace is allowed after an item, m[k] is a chain of <sym> <list> (mirroring f(x) as <sym> <tuple>).

<list> ::= "[" <seq> "]"

<block>

A block is a <seq> wrapped in braces. Like <tuple> and <list>, it only groups chains; Muon assigns it no meaning (e.g. Elly reads a block as a map literal; see docs/elly-spec.md).

Empty block, {}/{ }/{,,}/etc, is valid and wraps an empty sequence, and it follows the same multi-line and juxtaposition rules as a tuple and a list.

<block> ::= "{" <seq> "}"

json compatibility

• {"foo":42} is parsed as a block wrapping a chain of "foo" (<str>), : (<punct>), 42 (<sym>) — the same three items with or without spaces around the :, since : is not a symchar and never glues onto the value. A higher-level parser reads the key-value pair; Muon does not.
• [] is a <list> and {} is a <block>, so JSON arrays and objects are structurally covered (their :/, reading is left to a higher level).
• A JSON number with a fractional part is not a single atom: 3.14 is the chain 3 .14 (<sym> then a .-prefixed 14), because . is no longer a symchar. Integer JSON numbers are still whole <sym>s. A higher level can reassemble the fraction; a dedicated float literal is deferred (see the <sym> TODO).

Requires TODOs:

• check that strings are compatible
• fractional / exponent number literals (3.14, 1e9) once Elly grows Num