.. Copyright (c) 2024-2025 Tobias Erbsland - Erbsland DEV. https://erbsland.dev SPDX-License-Identifier: Apache-2.0 .. _test-outcome-format: .. index:: single: Test Outcome Format ======================= The Test Outcome Format ======================= The :ref:`test adapters` and the :ref:`test file structure` utilize a custom format to encode the expected or actual parsing results of a configuration file. This chapter provides an in-depth description of this format. .. design-rationale:: The test outcome format is intentionally line-based, avoiding dependencies on external libraries, making it robust and easy to produce, even in restricted environments. Although standard formats like JSON or XML might seem suitable, creating such formats from scratch can introduce additional complexity, including escape sequences and unique syntax constraints. Relying on external libraries for rendering introduces a risk that errors may result from the library rather than the tested code itself. General Format ============== The test outcome must be UTF-8 encoded text, with each value separated by a line break. A line break can be either a single newline or a carriage return followed by a newline. The document must always end with a line break. Each line follows the structure `` = ``, where ``name`` is either a lowercase name path (as described below) or the special uppercase keyword ``FAIL`` to indicate an error. The separator `` = `` must consist of a space, an equal sign, and another space. The format for values is outlined below. While there is no specific line length limit, test systems may impose an upper limit, such as 200 kB, to manage memory and processing requirements effectively. The test outcome format is intended to be as compact as possible, containing only essential data. It does not support syntax for comments, empty lines, indentation, or trailing spaces. Failure Format ============== In the event of a failure, the document should appear as follows: .. code-block:: text FAIL = NameConflict The name ``FAIL`` must be uppercase, followed by the error name as specified in :ref:`ref-error-code`. Ideally, the error name should use the case detailed in that chapter (e.g., "NameConflict"), but it is matched in a case-insensitive manner, so ``NAMECONFLICT`` and ``nameconflict`` are also acceptable. Optionally, the ``FAIL`` keyword can be followed by parentheses enclosing a detailed error message: .. code-block:: text FAIL = NameConflict("The name 'example' is already used.") FAIL = NameConflict(line: 5, column: 6, message: "The name 'example' is already used.") This optional part, beginning with the opening parenthesis, is ignored by the test system. However, you may include it for debugging purposes or to provide additional context during testing. Passing Format ============== When the parser successfully parses a document, it must return all values from the parsed value tree. This includes not only parsed values but also all value maps generated by sections, even intermediate or empty ones. Here’s an example of what the outcome file might look like: .. code-block:: text @version = String("1.0") main = SectionWithNames() main.connection = SectionList() main.connection[0] = SectionWithNames() main.connection[0].filter = String("test\u{12fe}") main.connection[0].flag = Boolean(true) main.connection[1] = SectionWithNames() main.connection[1].filter = String("test\u{12fe}") main.connection[1].flag = Boolean(false) main.server = Integer(-473945) translation = SectionWithTexts() translation."text" = Float(12.9) Format of the Name Path ----------------------- The name path format resembles that described in :ref:`ref-name-path`, with an important modification for lists. Regular Paths ~~~~~~~~~~~~~ Consider a simple example: .. code-block:: erbsland-conf [Main . Example Section] Value : 123 This should produce an outcome file as follows: .. code-block:: text main = IntermediateSection() main.example_section = SectionWithNames() main.example_section.value = Integer(123) Each name in the path is written in its normalized form (see :ref:`ref-name-normalization`), with segments separated by a period and no spaces allowed in the path. Section Lists ~~~~~~~~~~~~~ Consider the following example: .. code-block:: erbsland-conf *[main.server] value: 123 *[main.server] value: 123 This should produce an outcome file as follows: .. code-block:: text main = IntermediateSection() main.server = SectionList() main.server[0] = SectionWithNames() main.server[0].value = Integer(123) main.server[1] = SectionWithNames() main.server[1].value = Integer(123) A section list is represented in the outcome file by its name path. Each entry in the section list uses this name path, with the entry’s index enclosed in square brackets. Index numbering starts at zero for the first entry in the configuration, formatted as a decimal without leading zeros. Each entry, if it is a ``SectionWithNames`` or ``SectionWithTexts``, must also appear in the output. Paths for any subsections or values continue from this path after the closing square bracket. Value Lists ~~~~~~~~~~~ Consider the following example: .. code-block:: erbsland-conf [main] value: 1, 2, 3 This should produce an outcome file as follows: .. code-block:: text main = SectionWithNames() main.value = ValueList() main.value[0] = Integer(1) main.value[1] = Integer(2) main.value[2] = Integer(3) The format for value lists is similar to that of section lists. The list path itself is part of the output and is identified as a ``ValueList``. Each value in the list uses this path, with its index in square brackets. Index numbering starts at zero, with decimal formatting and no leading zeros. Text Names ~~~~~~~~~~ For text names that are part of a name path, the text must be enclosed in double quotes. This is the recommended internal format for storing text names. The text name must first be normalized, as outlined in :ref:`ref-text-name-normalization`. Once normalized, the text must be escaped as described in :ref:`test-outcome-escaping`. The same escaping logic applies to both text names and text values, so you can reuse the logic for both. For test output, the *indexed short form*, described in :doc:`../reference/name-path/text-names-for-parsers`, **must not** be used. Meta Values ~~~~~~~~~~~ Meta values such as ``@version`` and ``@features`` *can* be included in the output, but they are ignored by the test system. These values are ignored because parsers are not required to expose these values through their API. Requiring them solely for testing would necessitate unnecessary parser modifications. Format of the Values -------------------- Each value follows the format ``Type(Content)``, where the type identifier is immediately followed by the actual value enclosed in parentheses, with no spaces around the parentheses. The type identifier should match one of the recommended type names from the parser section. While type names are matched case-insensitively, it is recommended to use the original casing for readability. The content of each type is standardized as shown in the table below. .. list-table:: :header-rows: 1 :width: 100% :widths: 25, 25, 50 * - Type - Example - Description * - :text-code:`Integer` - ``123`` - The integer value in decimal format. Negative values use a minus sign; positive values are unprefixed. No leading zeros or digit separators. * - :text-code:`Boolean` - ``true`` - Either ``true`` or ``false``, both in lowercase. * - :text-code:`Float` - ``0.45e+20`` - A floating-point number, or ``inf`` or ``nan``, all lowercase. Represented in a compact format compatible with Python. * - :text-code:`Text` - ``"text"`` - Text enclosed in double quotes. Characters are escaped as described in :ref:`test-outcome-escaping`. * - :text-code:`Date` - ``2024-06-12`` - Date in ISO format, using four digits for the year, two digits for the month and day, separated with a hyphen (:cp:`-`). * - :text-code:`Time` - ``12:23:00z`` - Time in ISO format, with two digits for hour, minute and second, separated by a colon (:cp:`:`). If the time has an offset, the offset must be present with hour and minute. If the offset is zero or the time is marked as UTC (which is the same), a lowercase ``z`` must be used. If the second fractions are zero, the fractions must be omitted. If the second fractions are non-zero, it must be provided without trailing zeros. * - :text-code:`DateTime` - ``2024-06-12 12:23:00z`` - Date and time in ISO format, separated by a space. Same rules as for dates and times. * - :text-code:`Bytes` - ``02ca2e`` - Byte-data represented in lowercase hexadecimal, without separators. * - :text-code:`TimeDelta` - ``5,second`` - An integer, as specified for integer values, followed by a comma and the unit in lowercase singular form. For parsers that combine a list of time-delta values into a single value, this behavior **must be avoided** by the test adapter. * - :text-code:`RegEx` - ``"^reg$"`` - Formatted with the same rules as text. * - | :text-code:`ValueList` | :text-code:`SectionList` | :text-code:`IntermediateSection` | :text-code:`SectionWithNames` | :text-code:`SectionWithTexts` - – - Containers of any kind have their content ignored by the test system. Content after the opening parenthesis is optional but should end with a closing parenthesis. For example, ``size=12`` can be included as content. Order of the Values ------------------- The order of named values is not significant; the test system does not consider the sequence of these entries in its comparisons. However, the order of list entries *is important*. The test system requires that list indices match the sequence specified in the test files. Comparison of Values -------------------- Most values are compared as text, evaluating each character by its code point. If the value text matches the reference in the test file exactly, it is considered a pass. Therefore, it’s crucial to disable any Unicode normalization when using the test system, particularly for text values. One value type has additional comparison logic: - ``Float``: Floating-point values are compared using a default relative tolerance of ``1e-9`` and an absolute tolerance of ``1e-10``. For positive and negative values larger than ``1e+307``, ``inf`` or ``-inf`` is also accepted. These tolerances can be configured within the test system, allowing you to adjust the precision to meet specific requirements. .. _test-outcome-escaping: Escaping of Text ================ When rendering text in values or name paths for the test output, certain characters must be escaped to ensure clarity and consistency. The following characters must always be escaped: * Unicode control characters in the range :text-code:`U+0000`–:text-code:`U+001F` * Unicode code points in the extended range :text-code:`U+007F`–:text-code:`U+1FFFFF` * Backslash (:cp:`5c`) * Double quote (:cp:`"`) * Period (:cp:`.`) * Equal sign (:cp:`=`) All escapes must use the Unicode syntax ``\u{X}``, where ``X`` is the hexadecimal code point (without leading zeros). No other escape sequences are allowed. This escaping strategy matches the one recommended for parsers in :doc:`../reference/name-path/text-names-for-parsers`. .. important:: In test output rendering, the **short-form index notation** for text names (e.g. ``book.""[1]``) **must not be used**. Test adapters should always emit the full, quoted text form to allow a name based comparison.