onnx-mlir/docs/doc_check/README.md

127 lines
5.3 KiB
Markdown
Raw Normal View History

[RFC] Doc-check utility. (#12) * 1. Implement doc-check utility. * 1. Move ONNF installation script to a standalone script file. * 1. Modify build script to install llvm-project next to ONNF. The build script used to install llvm-project inside ONNF, which didn't make sense. * 1. Check out code to ONNF directory. * 1. Pass path parameter correctly. * 1. Debugging buildbot. * 1. Remove debug code. * 1. Update installation instructions in README.md. 2. Enforce consistency with scripts used in testing using doc-check. * 1. Fix error with respect to syntax to build multiple CMake targets. * 1. Move doc-check to doc_check. 2. Remove directive_config in top-level driver. * 1. Build onnf and check-mlir-lit separately because only CMake 3.15+ supports building multiple targets in one cmake --build run. * 1. Use new env variables to locate LLVM-Project. * 1. Documentation nits. * 1. Prettify buildbot scripts. * 1. Fix build script error. * 1. Support exclude_dirs in DocCheck. 2. Add README for DocCheck. * 1. Mark python3 interpreter as required. 2. Use imported interpreter target. * 1. Automatically deduce doc file extension in DocCheckCtx. 2. Rename ctx.open -> ctx.open_doc since it should only be used to open doc file. 3. Always read line in parser, instead of reading lines in driver and then passing it to parser.py. * 1. Rename parser -> doc_parser due to name conflict with python built-in module. 2. Explose doc_check module directory first before importing; otherwise if the doc_check utility is invoked by other script, importing will not work correctly. * 1. Keep renaming parser -> doc_parser. 2. Explicitly define a default configuration parser that parses the configuration into a python dictionary. * 1. Add test for doc-check. 2. Exclude doc-check tests from project dock-check because base directory is different. * 1. Raise ValueError if directive configuration fails to parse. 2. Format code. * Shorten test case documentation. Show example of using same-as-file directive, check with DocCheck. * 1. Shorten test case documentation. 2. More documentation, check documentation with DocCheck. * 1. Add copyright notice. * 1. Make documentation clearer. 2. Prettify build-scripts. * 1. Provide more documentation. 2. Fix some non-compliance with pep8 recommendations. Co-authored-by: Gheorghe-Teodor Bercea <gt.bercea@gmail.com>
2020-01-10 07:35:52 +08:00
# DocCheck
### Goal
It is always desirable to ensure that every piece of knowledge has a single, unambiguous, authoritative representation
in our codebase. However, sometimes violating such principle can result in improved overall quality of the software
project. For instance, when we write documentation containing example code snippets, it is desirable to write tests
for them - however, if we do so, the same code example will exist both in documentation and in tests! Such duplication
of knowledge has tangible adverse consequences - when documentation is updated to new examples, tests become obsolete.
Moreover, the discrepancy between multiple copies of the same knowledge (e.g., code example) can only be spotted with
manual inspection.
Under such circumstances, to establish a single source of trough in an enforceable manner, we can turn to the DocCheck
tool. Simply put, DocCheck enforces the consistency constraints as specified by the users between textual artifacts in
our codebase. Textual artifacts can be:
- Sections in documentation
- Content of a file
- Output of command execution
- ...
Specifically, DocCheck allows us to precisely specify how a textual artifact is derived from another. Such
specification is then parsed and verified by our software testing infrastructure to ensure the consistency between
derived textual artifact and the original one. This overall workflow provides an enforceable way to establish a single,
unambiguous and authoritative representation of knowledge in our codebase.
[RFC] Doc-check utility. (#12) * 1. Implement doc-check utility. * 1. Move ONNF installation script to a standalone script file. * 1. Modify build script to install llvm-project next to ONNF. The build script used to install llvm-project inside ONNF, which didn't make sense. * 1. Check out code to ONNF directory. * 1. Pass path parameter correctly. * 1. Debugging buildbot. * 1. Remove debug code. * 1. Update installation instructions in README.md. 2. Enforce consistency with scripts used in testing using doc-check. * 1. Fix error with respect to syntax to build multiple CMake targets. * 1. Move doc-check to doc_check. 2. Remove directive_config in top-level driver. * 1. Build onnf and check-mlir-lit separately because only CMake 3.15+ supports building multiple targets in one cmake --build run. * 1. Use new env variables to locate LLVM-Project. * 1. Documentation nits. * 1. Prettify buildbot scripts. * 1. Fix build script error. * 1. Support exclude_dirs in DocCheck. 2. Add README for DocCheck. * 1. Mark python3 interpreter as required. 2. Use imported interpreter target. * 1. Automatically deduce doc file extension in DocCheckCtx. 2. Rename ctx.open -> ctx.open_doc since it should only be used to open doc file. 3. Always read line in parser, instead of reading lines in driver and then passing it to parser.py. * 1. Rename parser -> doc_parser due to name conflict with python built-in module. 2. Explose doc_check module directory first before importing; otherwise if the doc_check utility is invoked by other script, importing will not work correctly. * 1. Keep renaming parser -> doc_parser. 2. Explicitly define a default configuration parser that parses the configuration into a python dictionary. * 1. Add test for doc-check. 2. Exclude doc-check tests from project dock-check because base directory is different. * 1. Raise ValueError if directive configuration fails to parse. 2. Format code. * Shorten test case documentation. Show example of using same-as-file directive, check with DocCheck. * 1. Shorten test case documentation. 2. More documentation, check documentation with DocCheck. * 1. Add copyright notice. * 1. Make documentation clearer. 2. Prettify build-scripts. * 1. Provide more documentation. 2. Fix some non-compliance with pep8 recommendations. Co-authored-by: Gheorghe-Teodor Bercea <gt.bercea@gmail.com>
2020-01-10 07:35:52 +08:00
### Directives
Directives can be used to communicate the relationship between derived and original textual artifacts to DocCheck.
DocCheck will perform consistency constraints checking according to the specification. In this section, supported
directives are explained in details.
Currently, directives can be specified either in a Markdown file or in a standalone DocCheck configuration file (a file
ending with `.dc` suffix). For markdown file, specify directive using the following syntax:
[RFC] Doc-check utility. (#12) * 1. Implement doc-check utility. * 1. Move ONNF installation script to a standalone script file. * 1. Modify build script to install llvm-project next to ONNF. The build script used to install llvm-project inside ONNF, which didn't make sense. * 1. Check out code to ONNF directory. * 1. Pass path parameter correctly. * 1. Debugging buildbot. * 1. Remove debug code. * 1. Update installation instructions in README.md. 2. Enforce consistency with scripts used in testing using doc-check. * 1. Fix error with respect to syntax to build multiple CMake targets. * 1. Move doc-check to doc_check. 2. Remove directive_config in top-level driver. * 1. Build onnf and check-mlir-lit separately because only CMake 3.15+ supports building multiple targets in one cmake --build run. * 1. Use new env variables to locate LLVM-Project. * 1. Documentation nits. * 1. Prettify buildbot scripts. * 1. Fix build script error. * 1. Support exclude_dirs in DocCheck. 2. Add README for DocCheck. * 1. Mark python3 interpreter as required. 2. Use imported interpreter target. * 1. Automatically deduce doc file extension in DocCheckCtx. 2. Rename ctx.open -> ctx.open_doc since it should only be used to open doc file. 3. Always read line in parser, instead of reading lines in driver and then passing it to parser.py. * 1. Rename parser -> doc_parser due to name conflict with python built-in module. 2. Explose doc_check module directory first before importing; otherwise if the doc_check utility is invoked by other script, importing will not work correctly. * 1. Keep renaming parser -> doc_parser. 2. Explicitly define a default configuration parser that parses the configuration into a python dictionary. * 1. Add test for doc-check. 2. Exclude doc-check tests from project dock-check because base directory is different. * 1. Raise ValueError if directive configuration fails to parse. 2. Format code. * Shorten test case documentation. Show example of using same-as-file directive, check with DocCheck. * 1. Shorten test case documentation. 2. More documentation, check documentation with DocCheck. * 1. Add copyright notice. * 1. Make documentation clearer. 2. Prettify build-scripts. * 1. Provide more documentation. 2. Fix some non-compliance with pep8 recommendations. Co-authored-by: Gheorghe-Teodor Bercea <gt.bercea@gmail.com>
2020-01-10 07:35:52 +08:00
```markdown
[{directive}]: <> ({configuration})
```
For standalone DocCheck configuration file, use the following syntax:
```
{directive}({configuration})
```
where `{directive}` is the name of the directive and `{configuration}` expresses the specific
[RFC] Doc-check utility. (#12) * 1. Implement doc-check utility. * 1. Move ONNF installation script to a standalone script file. * 1. Modify build script to install llvm-project next to ONNF. The build script used to install llvm-project inside ONNF, which didn't make sense. * 1. Check out code to ONNF directory. * 1. Pass path parameter correctly. * 1. Debugging buildbot. * 1. Remove debug code. * 1. Update installation instructions in README.md. 2. Enforce consistency with scripts used in testing using doc-check. * 1. Fix error with respect to syntax to build multiple CMake targets. * 1. Move doc-check to doc_check. 2. Remove directive_config in top-level driver. * 1. Build onnf and check-mlir-lit separately because only CMake 3.15+ supports building multiple targets in one cmake --build run. * 1. Use new env variables to locate LLVM-Project. * 1. Documentation nits. * 1. Prettify buildbot scripts. * 1. Fix build script error. * 1. Support exclude_dirs in DocCheck. 2. Add README for DocCheck. * 1. Mark python3 interpreter as required. 2. Use imported interpreter target. * 1. Automatically deduce doc file extension in DocCheckCtx. 2. Rename ctx.open -> ctx.open_doc since it should only be used to open doc file. 3. Always read line in parser, instead of reading lines in driver and then passing it to parser.py. * 1. Rename parser -> doc_parser due to name conflict with python built-in module. 2. Explose doc_check module directory first before importing; otherwise if the doc_check utility is invoked by other script, importing will not work correctly. * 1. Keep renaming parser -> doc_parser. 2. Explicitly define a default configuration parser that parses the configuration into a python dictionary. * 1. Add test for doc-check. 2. Exclude doc-check tests from project dock-check because base directory is different. * 1. Raise ValueError if directive configuration fails to parse. 2. Format code. * Shorten test case documentation. Show example of using same-as-file directive, check with DocCheck. * 1. Shorten test case documentation. 2. More documentation, check documentation with DocCheck. * 1. Add copyright notice. * 1. Make documentation clearer. 2. Prettify build-scripts. * 1. Provide more documentation. 2. Fix some non-compliance with pep8 recommendations. Co-authored-by: Gheorghe-Teodor Bercea <gt.bercea@gmail.com>
2020-01-10 07:35:52 +08:00
parameters of this directive. In general, a directive configuration is expressed using a python dictionary literal,
with supported configuration parameter name as keys and the desired state of configuration as values.
Special shorthands exist for each directive individually.
[RFC] Doc-check utility. (#12) * 1. Implement doc-check utility. * 1. Move ONNF installation script to a standalone script file. * 1. Modify build script to install llvm-project next to ONNF. The build script used to install llvm-project inside ONNF, which didn't make sense. * 1. Check out code to ONNF directory. * 1. Pass path parameter correctly. * 1. Debugging buildbot. * 1. Remove debug code. * 1. Update installation instructions in README.md. 2. Enforce consistency with scripts used in testing using doc-check. * 1. Fix error with respect to syntax to build multiple CMake targets. * 1. Move doc-check to doc_check. 2. Remove directive_config in top-level driver. * 1. Build onnf and check-mlir-lit separately because only CMake 3.15+ supports building multiple targets in one cmake --build run. * 1. Use new env variables to locate LLVM-Project. * 1. Documentation nits. * 1. Prettify buildbot scripts. * 1. Fix build script error. * 1. Support exclude_dirs in DocCheck. 2. Add README for DocCheck. * 1. Mark python3 interpreter as required. 2. Use imported interpreter target. * 1. Automatically deduce doc file extension in DocCheckCtx. 2. Rename ctx.open -> ctx.open_doc since it should only be used to open doc file. 3. Always read line in parser, instead of reading lines in driver and then passing it to parser.py. * 1. Rename parser -> doc_parser due to name conflict with python built-in module. 2. Explose doc_check module directory first before importing; otherwise if the doc_check utility is invoked by other script, importing will not work correctly. * 1. Keep renaming parser -> doc_parser. 2. Explicitly define a default configuration parser that parses the configuration into a python dictionary. * 1. Add test for doc-check. 2. Exclude doc-check tests from project dock-check because base directory is different. * 1. Raise ValueError if directive configuration fails to parse. 2. Format code. * Shorten test case documentation. Show example of using same-as-file directive, check with DocCheck. * 1. Shorten test case documentation. 2. More documentation, check documentation with DocCheck. * 1. Add copyright notice. * 1. Make documentation clearer. 2. Prettify build-scripts. * 1. Provide more documentation. 2. Fix some non-compliance with pep8 recommendations. Co-authored-by: Gheorghe-Teodor Bercea <gt.bercea@gmail.com>
2020-01-10 07:35:52 +08:00
##### `same-as-file`:
Use `same-as-file` directive to ensure that the code section following this directive is the same as a source file.
This is useful primarily because testing code snippet in documentation directly is often impossible. However,
unit tests can be written utilizing an exact copy of the code snippet content. We can use `same-as-file` directive
to ensure the code snippet is always the same as its copy used in some unit tests.
[RFC] Doc-check utility. (#12) * 1. Implement doc-check utility. * 1. Move ONNF installation script to a standalone script file. * 1. Modify build script to install llvm-project next to ONNF. The build script used to install llvm-project inside ONNF, which didn't make sense. * 1. Check out code to ONNF directory. * 1. Pass path parameter correctly. * 1. Debugging buildbot. * 1. Remove debug code. * 1. Update installation instructions in README.md. 2. Enforce consistency with scripts used in testing using doc-check. * 1. Fix error with respect to syntax to build multiple CMake targets. * 1. Move doc-check to doc_check. 2. Remove directive_config in top-level driver. * 1. Build onnf and check-mlir-lit separately because only CMake 3.15+ supports building multiple targets in one cmake --build run. * 1. Use new env variables to locate LLVM-Project. * 1. Documentation nits. * 1. Prettify buildbot scripts. * 1. Fix build script error. * 1. Support exclude_dirs in DocCheck. 2. Add README for DocCheck. * 1. Mark python3 interpreter as required. 2. Use imported interpreter target. * 1. Automatically deduce doc file extension in DocCheckCtx. 2. Rename ctx.open -> ctx.open_doc since it should only be used to open doc file. 3. Always read line in parser, instead of reading lines in driver and then passing it to parser.py. * 1. Rename parser -> doc_parser due to name conflict with python built-in module. 2. Explose doc_check module directory first before importing; otherwise if the doc_check utility is invoked by other script, importing will not work correctly. * 1. Keep renaming parser -> doc_parser. 2. Explicitly define a default configuration parser that parses the configuration into a python dictionary. * 1. Add test for doc-check. 2. Exclude doc-check tests from project dock-check because base directory is different. * 1. Raise ValueError if directive configuration fails to parse. 2. Format code. * Shorten test case documentation. Show example of using same-as-file directive, check with DocCheck. * 1. Shorten test case documentation. 2. More documentation, check documentation with DocCheck. * 1. Add copyright notice. * 1. Make documentation clearer. 2. Prettify build-scripts. * 1. Provide more documentation. 2. Fix some non-compliance with pep8 recommendations. Co-authored-by: Gheorghe-Teodor Bercea <gt.bercea@gmail.com>
2020-01-10 07:35:52 +08:00
`same-as-file` directive supports a convenient short-hand configuration format where the directive configuration can
be fully specified using the name of the reference file to check against. For example, to ensure a code snippet is the
same as a unit-tested file `reference.cpp`, use the following directive as shown in the documentation snippet:
[RFC] Doc-check utility. (#12) * 1. Implement doc-check utility. * 1. Move ONNF installation script to a standalone script file. * 1. Modify build script to install llvm-project next to ONNF. The build script used to install llvm-project inside ONNF, which didn't make sense. * 1. Check out code to ONNF directory. * 1. Pass path parameter correctly. * 1. Debugging buildbot. * 1. Remove debug code. * 1. Update installation instructions in README.md. 2. Enforce consistency with scripts used in testing using doc-check. * 1. Fix error with respect to syntax to build multiple CMake targets. * 1. Move doc-check to doc_check. 2. Remove directive_config in top-level driver. * 1. Build onnf and check-mlir-lit separately because only CMake 3.15+ supports building multiple targets in one cmake --build run. * 1. Use new env variables to locate LLVM-Project. * 1. Documentation nits. * 1. Prettify buildbot scripts. * 1. Fix build script error. * 1. Support exclude_dirs in DocCheck. 2. Add README for DocCheck. * 1. Mark python3 interpreter as required. 2. Use imported interpreter target. * 1. Automatically deduce doc file extension in DocCheckCtx. 2. Rename ctx.open -> ctx.open_doc since it should only be used to open doc file. 3. Always read line in parser, instead of reading lines in driver and then passing it to parser.py. * 1. Rename parser -> doc_parser due to name conflict with python built-in module. 2. Explose doc_check module directory first before importing; otherwise if the doc_check utility is invoked by other script, importing will not work correctly. * 1. Keep renaming parser -> doc_parser. 2. Explicitly define a default configuration parser that parses the configuration into a python dictionary. * 1. Add test for doc-check. 2. Exclude doc-check tests from project dock-check because base directory is different. * 1. Raise ValueError if directive configuration fails to parse. 2. Format code. * Shorten test case documentation. Show example of using same-as-file directive, check with DocCheck. * 1. Shorten test case documentation. 2. More documentation, check documentation with DocCheck. * 1. Add copyright notice. * 1. Make documentation clearer. 2. Prettify build-scripts. * 1. Provide more documentation. 2. Fix some non-compliance with pep8 recommendations. Co-authored-by: Gheorghe-Teodor Bercea <gt.bercea@gmail.com>
2020-01-10 07:35:52 +08:00
[same-as-file]: <> (docs/doc_check/test/same-as-file/simple/README.md)
[RFC] Doc-check utility. (#12) * 1. Implement doc-check utility. * 1. Move ONNF installation script to a standalone script file. * 1. Modify build script to install llvm-project next to ONNF. The build script used to install llvm-project inside ONNF, which didn't make sense. * 1. Check out code to ONNF directory. * 1. Pass path parameter correctly. * 1. Debugging buildbot. * 1. Remove debug code. * 1. Update installation instructions in README.md. 2. Enforce consistency with scripts used in testing using doc-check. * 1. Fix error with respect to syntax to build multiple CMake targets. * 1. Move doc-check to doc_check. 2. Remove directive_config in top-level driver. * 1. Build onnf and check-mlir-lit separately because only CMake 3.15+ supports building multiple targets in one cmake --build run. * 1. Use new env variables to locate LLVM-Project. * 1. Documentation nits. * 1. Prettify buildbot scripts. * 1. Fix build script error. * 1. Support exclude_dirs in DocCheck. 2. Add README for DocCheck. * 1. Mark python3 interpreter as required. 2. Use imported interpreter target. * 1. Automatically deduce doc file extension in DocCheckCtx. 2. Rename ctx.open -> ctx.open_doc since it should only be used to open doc file. 3. Always read line in parser, instead of reading lines in driver and then passing it to parser.py. * 1. Rename parser -> doc_parser due to name conflict with python built-in module. 2. Explose doc_check module directory first before importing; otherwise if the doc_check utility is invoked by other script, importing will not work correctly. * 1. Keep renaming parser -> doc_parser. 2. Explicitly define a default configuration parser that parses the configuration into a python dictionary. * 1. Add test for doc-check. 2. Exclude doc-check tests from project dock-check because base directory is different. * 1. Raise ValueError if directive configuration fails to parse. 2. Format code. * Shorten test case documentation. Show example of using same-as-file directive, check with DocCheck. * 1. Shorten test case documentation. 2. More documentation, check documentation with DocCheck. * 1. Add copyright notice. * 1. Make documentation clearer. 2. Prettify build-scripts. * 1. Provide more documentation. 2. Fix some non-compliance with pep8 recommendations. Co-authored-by: Gheorghe-Teodor Bercea <gt.bercea@gmail.com>
2020-01-10 07:35:52 +08:00
````markdown
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
[same-as-file]: <> (reference.cpp)
```cpp
#include<iostream>
using namespace std;
int main() {
cout<<"Hello World";
return 0;
}
```
````
In the canonical form of directive configuration (as a python dictionary literal), this directive supports these parameters in it:
`ref` (string): reference file to check against.
`skip-doc` (int): number of lines to skip when checking the documentation.
`skip-ref` (int): number of lines to skip when scanning the reference file.
For example, to ensure the following code snippet is the same as a unit-tested file `reference.cpp`, except for the first 2 lines of the code used in documentation, and the first 3 lines of code used in the reference file, the following directive configuration can be used:
[same-as-file]: <> (docs/doc_check/test/same-as-file/skip-doc-ref/README.md)
[RFC] Doc-check utility. (#12) * 1. Implement doc-check utility. * 1. Move ONNF installation script to a standalone script file. * 1. Modify build script to install llvm-project next to ONNF. The build script used to install llvm-project inside ONNF, which didn't make sense. * 1. Check out code to ONNF directory. * 1. Pass path parameter correctly. * 1. Debugging buildbot. * 1. Remove debug code. * 1. Update installation instructions in README.md. 2. Enforce consistency with scripts used in testing using doc-check. * 1. Fix error with respect to syntax to build multiple CMake targets. * 1. Move doc-check to doc_check. 2. Remove directive_config in top-level driver. * 1. Build onnf and check-mlir-lit separately because only CMake 3.15+ supports building multiple targets in one cmake --build run. * 1. Use new env variables to locate LLVM-Project. * 1. Documentation nits. * 1. Prettify buildbot scripts. * 1. Fix build script error. * 1. Support exclude_dirs in DocCheck. 2. Add README for DocCheck. * 1. Mark python3 interpreter as required. 2. Use imported interpreter target. * 1. Automatically deduce doc file extension in DocCheckCtx. 2. Rename ctx.open -> ctx.open_doc since it should only be used to open doc file. 3. Always read line in parser, instead of reading lines in driver and then passing it to parser.py. * 1. Rename parser -> doc_parser due to name conflict with python built-in module. 2. Explose doc_check module directory first before importing; otherwise if the doc_check utility is invoked by other script, importing will not work correctly. * 1. Keep renaming parser -> doc_parser. 2. Explicitly define a default configuration parser that parses the configuration into a python dictionary. * 1. Add test for doc-check. 2. Exclude doc-check tests from project dock-check because base directory is different. * 1. Raise ValueError if directive configuration fails to parse. 2. Format code. * Shorten test case documentation. Show example of using same-as-file directive, check with DocCheck. * 1. Shorten test case documentation. 2. More documentation, check documentation with DocCheck. * 1. Add copyright notice. * 1. Make documentation clearer. 2. Prettify build-scripts. * 1. Provide more documentation. 2. Fix some non-compliance with pep8 recommendations. Co-authored-by: Gheorghe-Teodor Bercea <gt.bercea@gmail.com>
2020-01-10 07:35:52 +08:00
````markdown
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
[same-as-file]: <> ({"ref": "reference.cpp", "skip-doc": 2, "skip-ref": 3})
```cpp
// First line unique to documentation
// Second line unique to documentation
#include<iostream>
using namespace std;
int main() {
cout<<"Hello World";
return 0;
}
```
````
#### `file-same-as-stdout`
Use `file-same-as-stdout` to ensure that file content is the same as the output of executing a command.
This directive supports these parameters in it:
`file` (string): file to compare with.
`cmd` (List[str]): the command (expressed as a list of command components), e.g. `["ls", "-l"]`.
For example, to ensure that the content of a file `test.in`:
[same-as-file]: <> (docs/doc_check/test/file-same-as-stdout/success/test.in)
```
dog
```
is exactly the same as the output of command execution `echo dog`, one can use the following directive:
[same-as-file]: <> (docs/doc_check/test/file-same-as-stdout/success/test.in.dc)
```
file-same-as-stdout({"file": "test.in", "cmd": ["echo", "dog"]})
```