The Test Outcome Format

The test adapters and the test file structure utilize a custom format to encode the expected or actual parsing results of a configuration file. This chapter provides an in-depth description of this format.

Design Rationale

The test outcome format is intentionally line-based, avoiding dependencies on external libraries, making it robust and easy to produce, even in restricted environments.

Although standard formats like JSON or XML might seem suitable, creating such formats from scratch can introduce additional complexity, including escape sequences and unique syntax constraints. Relying on external libraries for rendering introduces a risk that errors may result from the library rather than the tested code itself.

General Format

The test outcome must be UTF-8 encoded text, with each value separated by a line break. A line break can be either a single newline or a carriage return followed by a newline. The document must always end with a line break.

Each line follows the structure <name> = <value>, where name is either a lowercase name path (as described below) or the special uppercase keyword FAIL to indicate an error. The separator `` = `` must consist of a space, an equal sign, and another space. The format for values is outlined below.

While there is no specific line length limit, test systems may impose an upper limit, such as 200 kB, to manage memory and processing requirements effectively.

The test outcome format is intended to be as compact as possible, containing only essential data. It does not support syntax for comments, empty lines, indentation, or trailing spaces.

Failure Format

In the event of a failure, the document should appear as follows:

FAIL = NameConflict

The name FAIL must be uppercase, followed by the error name as specified in Error Categories and Codes. Ideally, the error name should use the case detailed in that chapter (e.g., “NameConflict”), but it is matched in a case-insensitive manner, so NAMECONFLICT and nameconflict are also acceptable.

Optionally, the FAIL keyword can be followed by parentheses enclosing a detailed error message:

FAIL = NameConflict("The name 'example' is already used.")
FAIL = NameConflict(line: 5, column: 6, message: "The name 'example' is already used.")

This optional part, beginning with the opening parenthesis, is ignored by the test system. However, you may include it for debugging purposes or to provide additional context during testing.

Passing Format

When the parser successfully parses a document, it must return all values from the parsed value tree. This includes not only parsed values but also all value maps generated by sections, even intermediate or empty ones.

Here’s an example of what the outcome file might look like:

@version = String("1.0")
main = SectionWithNames()
main.connection = SectionList()
main.connection[0] = SectionWithNames()
main.connection[0].filter = String("test\u{12fe}")
main.connection[0].flag = Boolean(true)
main.connection[1] = SectionWithNames()
main.connection[1].filter = String("test\u{12fe}")
main.connection[1].flag = Boolean(false)
main.server = Integer(-473945)
translation = SectionWithTexts()
translation."text" = Float(12.9)

Format of the Name Path

The name path format resembles that described in Name Paths, with an important modification for lists.

Regular Paths

Consider a simple example:

[Main . Example Section]
Value : 123

This should produce an outcome file as follows:

main = IntermediateSection()
main.example_section = SectionWithNames()
main.example_section.value = Integer(123)

Each name in the path is written in its normalized form (see Name Normalization), with segments separated by a period and no spaces allowed in the path.

Section Lists

Consider the following example:

*[main.server]
value: 123
*[main.server]
value: 123

This should produce an outcome file as follows:

main = IntermediateSection()
main.server = SectionList()
main.server[0] = SectionWithNames()
main.server[0].value = Integer(123)
main.server[1] = SectionWithNames()
main.server[1].value = Integer(123)

A section list is represented in the outcome file by its name path. Each entry in the section list uses this name path, with the entry’s index enclosed in square brackets. Index numbering starts at zero for the first entry in the configuration, formatted as a decimal without leading zeros.

Each entry, if it is a SectionWithNames or SectionWithTexts, must also appear in the output. Paths for any subsections or values continue from this path after the closing square bracket.

Value Lists

Consider the following example:

[main]
value: 1, 2, 3

This should produce an outcome file as follows:

main = SectionWithNames()
main.value = ValueList()
main.value[0] = Integer(1)
main.value[1] = Integer(2)
main.value[2] = Integer(3)

The format for value lists is similar to that of section lists. The list path itself is part of the output and is identified as a ValueList. Each value in the list uses this path, with its index in square brackets. Index numbering starts at zero, with decimal formatting and no leading zeros.

Text Names

For text names that are part of a name path, the text must be enclosed in double quotes. This is the recommended internal format for storing text names. The text name must first be normalized, as outlined in Text Name Normalization.

Once normalized, the text must be escaped as described in Escaping of Text. The same escaping logic applies to both text names and text values, so you can reuse the logic for both.

For test output, the indexed short form, described in Parser-Specific Usage of Text Names, must not be used.

Meta Values

Meta values such as @version and @features can be included in the output, but they are ignored by the test system. These values are ignored because parsers are not required to expose these values through their API. Requiring them solely for testing would necessitate unnecessary parser modifications.

Format of the Values

Each value follows the format Type(Content), where the type identifier is immediately followed by the actual value enclosed in parentheses, with no spaces around the parentheses.

The type identifier should match one of the recommended type names from the parser section. While type names are matched case-insensitively, it is recommended to use the original casing for readability.

The content of each type is standardized as shown in the table below.

Type	Example	Description
Integer	`123`	The integer value in decimal format. Negative values use a minus sign; positive values are unprefixed. No leading zeros or digit separators.
Boolean	`true`	Either `true` or `false`, both in lowercase.
Float	`0.45e+20`	A floating-point number, or `inf` or `nan`, all lowercase. Represented in a compact format compatible with Python.
Text	`"text"`	Text enclosed in double quotes. Characters are escaped as described in Escaping of Text.
Date	`2024-06-12`	Date in ISO format, using four digits for the year, two digits for the month and day, separated with a hyphen (`-`).
Time	`12:23:00z`	Time in ISO format, with two digits for hour, minute and second, separated by a colon (`:`). If the time has an offset, the offset must be present with hour and minute. If the offset is zero or the time is marked as UTC (which is the same), a lowercase `z` must be used. If the second fractions are zero, the fractions must be omitted. If the second fractions are non-zero, it must be provided without trailing zeros.
DateTime	`2024-06-12 12:23:00z`	Date and time in ISO format, separated by a space. Same rules as for dates and times.
Bytes	`02ca2e`	Byte-data represented in lowercase hexadecimal, without separators.
TimeDelta	`5,second`	An integer, as specified for integer values, followed by a comma and the unit in lowercase singular form. For parsers that combine a list of time-delta values into a single value, this behavior must be avoided by the test adapter.
RegEx	`"^reg$"`	Formatted with the same rules as text.
ValueList SectionList IntermediateSection SectionWithNames SectionWithTexts	–	Containers of any kind have their content ignored by the test system. Content after the opening parenthesis is optional but should end with a closing parenthesis. For example, `size=12` can be included as content.

Order of the Values

The order of named values is not significant; the test system does not consider the sequence of these entries in its comparisons.

However, the order of list entries is important. The test system requires that list indices match the sequence specified in the test files.

Comparison of Values

Most values are compared as text, evaluating each character by its code point. If the value text matches the reference in the test file exactly, it is considered a pass. Therefore, it’s crucial to disable any Unicode normalization when using the test system, particularly for text values.

One value type has additional comparison logic:

Float: Floating-point values are compared using a default relative tolerance of 1e-9 and an absolute tolerance of 1e-10. For positive and negative values larger than 1e+307, inf or -inf is also accepted. These tolerances can be configured within the test system, allowing you to adjust the precision to meet specific requirements.

Escaping of Text

When rendering text in values or name paths for the test output, certain characters must be escaped to ensure clarity and consistency. The following characters must always be escaped:

Unicode control characters in the range U+0000–U+001F
Unicode code points in the extended range U+007F–U+1FFFFF
Backslash (\)
Double quote (")
Period (.)
Equal sign (=)

All escapes must use the Unicode syntax \u{X}, where X is the hexadecimal code point (without leading zeros). No other escape sequences are allowed.

This escaping strategy matches the one recommended for parsers in Parser-Specific Usage of Text Names.

Important

In test output rendering, the short-form index notation for text names (e.g. book.""[1]) must not be used. Test adapters should always emit the full, quoted text form to allow a name based comparison.