onnx-mlir/docs/doc_check/doc_parser.py

50 lines
1.6 KiB
Python
Raw Permalink Normal View History

[RFC] Doc-check utility. (#12) * 1. Implement doc-check utility. * 1. Move ONNF installation script to a standalone script file. * 1. Modify build script to install llvm-project next to ONNF. The build script used to install llvm-project inside ONNF, which didn't make sense. * 1. Check out code to ONNF directory. * 1. Pass path parameter correctly. * 1. Debugging buildbot. * 1. Remove debug code. * 1. Update installation instructions in README.md. 2. Enforce consistency with scripts used in testing using doc-check. * 1. Fix error with respect to syntax to build multiple CMake targets. * 1. Move doc-check to doc_check. 2. Remove directive_config in top-level driver. * 1. Build onnf and check-mlir-lit separately because only CMake 3.15+ supports building multiple targets in one cmake --build run. * 1. Use new env variables to locate LLVM-Project. * 1. Documentation nits. * 1. Prettify buildbot scripts. * 1. Fix build script error. * 1. Support exclude_dirs in DocCheck. 2. Add README for DocCheck. * 1. Mark python3 interpreter as required. 2. Use imported interpreter target. * 1. Automatically deduce doc file extension in DocCheckCtx. 2. Rename ctx.open -> ctx.open_doc since it should only be used to open doc file. 3. Always read line in parser, instead of reading lines in driver and then passing it to parser.py. * 1. Rename parser -> doc_parser due to name conflict with python built-in module. 2. Explose doc_check module directory first before importing; otherwise if the doc_check utility is invoked by other script, importing will not work correctly. * 1. Keep renaming parser -> doc_parser. 2. Explicitly define a default configuration parser that parses the configuration into a python dictionary. * 1. Add test for doc-check. 2. Exclude doc-check tests from project dock-check because base directory is different. * 1. Raise ValueError if directive configuration fails to parse. 2. Format code. * Shorten test case documentation. Show example of using same-as-file directive, check with DocCheck. * 1. Shorten test case documentation. 2. More documentation, check documentation with DocCheck. * 1. Add copyright notice. * 1. Make documentation clearer. 2. Prettify build-scripts. * 1. Provide more documentation. 2. Fix some non-compliance with pep8 recommendations. Co-authored-by: Gheorghe-Teodor Bercea <gt.bercea@gmail.com>
2020-01-10 07:35:52 +08:00
# ===------------ doc_parser.py - Documentation Parsing Utility ------------===//
#
# Copyright 2019-2020 The IBM Research Authors.
#
# =============================================================================
#
# ===----------------------------------------------------------------------===//
import logging
[RFC] Doc-check utility. (#12) * 1. Implement doc-check utility. * 1. Move ONNF installation script to a standalone script file. * 1. Modify build script to install llvm-project next to ONNF. The build script used to install llvm-project inside ONNF, which didn't make sense. * 1. Check out code to ONNF directory. * 1. Pass path parameter correctly. * 1. Debugging buildbot. * 1. Remove debug code. * 1. Update installation instructions in README.md. 2. Enforce consistency with scripts used in testing using doc-check. * 1. Fix error with respect to syntax to build multiple CMake targets. * 1. Move doc-check to doc_check. 2. Remove directive_config in top-level driver. * 1. Build onnf and check-mlir-lit separately because only CMake 3.15+ supports building multiple targets in one cmake --build run. * 1. Use new env variables to locate LLVM-Project. * 1. Documentation nits. * 1. Prettify buildbot scripts. * 1. Fix build script error. * 1. Support exclude_dirs in DocCheck. 2. Add README for DocCheck. * 1. Mark python3 interpreter as required. 2. Use imported interpreter target. * 1. Automatically deduce doc file extension in DocCheckCtx. 2. Rename ctx.open -> ctx.open_doc since it should only be used to open doc file. 3. Always read line in parser, instead of reading lines in driver and then passing it to parser.py. * 1. Rename parser -> doc_parser due to name conflict with python built-in module. 2. Explose doc_check module directory first before importing; otherwise if the doc_check utility is invoked by other script, importing will not work correctly. * 1. Keep renaming parser -> doc_parser. 2. Explicitly define a default configuration parser that parses the configuration into a python dictionary. * 1. Add test for doc-check. 2. Exclude doc-check tests from project dock-check because base directory is different. * 1. Raise ValueError if directive configuration fails to parse. 2. Format code. * Shorten test case documentation. Show example of using same-as-file directive, check with DocCheck. * 1. Shorten test case documentation. 2. More documentation, check documentation with DocCheck. * 1. Add copyright notice. * 1. Make documentation clearer. 2. Prettify build-scripts. * 1. Provide more documentation. 2. Fix some non-compliance with pep8 recommendations. Co-authored-by: Gheorghe-Teodor Bercea <gt.bercea@gmail.com>
2020-01-10 07:35:52 +08:00
from typing import List
from utils import *
from directive import Directive, generic_config_parser
logger = logging.getLogger('doc-check')
[RFC] Doc-check utility. (#12) * 1. Implement doc-check utility. * 1. Move ONNF installation script to a standalone script file. * 1. Modify build script to install llvm-project next to ONNF. The build script used to install llvm-project inside ONNF, which didn't make sense. * 1. Check out code to ONNF directory. * 1. Pass path parameter correctly. * 1. Debugging buildbot. * 1. Remove debug code. * 1. Update installation instructions in README.md. 2. Enforce consistency with scripts used in testing using doc-check. * 1. Fix error with respect to syntax to build multiple CMake targets. * 1. Move doc-check to doc_check. 2. Remove directive_config in top-level driver. * 1. Build onnf and check-mlir-lit separately because only CMake 3.15+ supports building multiple targets in one cmake --build run. * 1. Use new env variables to locate LLVM-Project. * 1. Documentation nits. * 1. Prettify buildbot scripts. * 1. Fix build script error. * 1. Support exclude_dirs in DocCheck. 2. Add README for DocCheck. * 1. Mark python3 interpreter as required. 2. Use imported interpreter target. * 1. Automatically deduce doc file extension in DocCheckCtx. 2. Rename ctx.open -> ctx.open_doc since it should only be used to open doc file. 3. Always read line in parser, instead of reading lines in driver and then passing it to parser.py. * 1. Rename parser -> doc_parser due to name conflict with python built-in module. 2. Explose doc_check module directory first before importing; otherwise if the doc_check utility is invoked by other script, importing will not work correctly. * 1. Keep renaming parser -> doc_parser. 2. Explicitly define a default configuration parser that parses the configuration into a python dictionary. * 1. Add test for doc-check. 2. Exclude doc-check tests from project dock-check because base directory is different. * 1. Raise ValueError if directive configuration fails to parse. 2. Format code. * Shorten test case documentation. Show example of using same-as-file directive, check with DocCheck. * 1. Shorten test case documentation. 2. More documentation, check documentation with DocCheck. * 1. Add copyright notice. * 1. Make documentation clearer. 2. Prettify build-scripts. * 1. Provide more documentation. 2. Fix some non-compliance with pep8 recommendations. Co-authored-by: Gheorghe-Teodor Bercea <gt.bercea@gmail.com>
2020-01-10 07:35:52 +08:00
def parse_code_section_delimiter(ctx):
assert ctx.doc_file_ext() == ".md"
if not ctx.doc_file.next_non_empty_line().strip().startswith("```"):
raise ValueError("Did not parse a code section delimiter")
def try_parse_and_handle_directive(ctx):
from directive_impl import same_as_file
from directive_impl import file_same_as_stdout
try:
line = ctx.doc_file.next_non_empty_line()
except RuntimeError as e:
# Do not raise exception when next non-empty line
# does not exist. Instead, return failure.
if str(e) != "Enf of file.":
raise
return failure()
[RFC] Doc-check utility. (#12) * 1. Implement doc-check utility. * 1. Move ONNF installation script to a standalone script file. * 1. Modify build script to install llvm-project next to ONNF. The build script used to install llvm-project inside ONNF, which didn't make sense. * 1. Check out code to ONNF directory. * 1. Pass path parameter correctly. * 1. Debugging buildbot. * 1. Remove debug code. * 1. Update installation instructions in README.md. 2. Enforce consistency with scripts used in testing using doc-check. * 1. Fix error with respect to syntax to build multiple CMake targets. * 1. Move doc-check to doc_check. 2. Remove directive_config in top-level driver. * 1. Build onnf and check-mlir-lit separately because only CMake 3.15+ supports building multiple targets in one cmake --build run. * 1. Use new env variables to locate LLVM-Project. * 1. Documentation nits. * 1. Prettify buildbot scripts. * 1. Fix build script error. * 1. Support exclude_dirs in DocCheck. 2. Add README for DocCheck. * 1. Mark python3 interpreter as required. 2. Use imported interpreter target. * 1. Automatically deduce doc file extension in DocCheckCtx. 2. Rename ctx.open -> ctx.open_doc since it should only be used to open doc file. 3. Always read line in parser, instead of reading lines in driver and then passing it to parser.py. * 1. Rename parser -> doc_parser due to name conflict with python built-in module. 2. Explose doc_check module directory first before importing; otherwise if the doc_check utility is invoked by other script, importing will not work correctly. * 1. Keep renaming parser -> doc_parser. 2. Explicitly define a default configuration parser that parses the configuration into a python dictionary. * 1. Add test for doc-check. 2. Exclude doc-check tests from project dock-check because base directory is different. * 1. Raise ValueError if directive configuration fails to parse. 2. Format code. * Shorten test case documentation. Show example of using same-as-file directive, check with DocCheck. * 1. Shorten test case documentation. 2. More documentation, check documentation with DocCheck. * 1. Add copyright notice. * 1. Make documentation clearer. 2. Prettify build-scripts. * 1. Provide more documentation. 2. Fix some non-compliance with pep8 recommendations. Co-authored-by: Gheorghe-Teodor Bercea <gt.bercea@gmail.com>
2020-01-10 07:35:52 +08:00
# Register all directives.
all_directives: List[Directive] = [
Directive(same_as_file.ext_to_patterns, [generic_config_parser, same_as_file.parse], same_as_file.handle),
Directive(file_same_as_stdout.ext_to_patterns, [generic_config_parser], file_same_as_stdout.handle),
[RFC] Doc-check utility. (#12) * 1. Implement doc-check utility. * 1. Move ONNF installation script to a standalone script file. * 1. Modify build script to install llvm-project next to ONNF. The build script used to install llvm-project inside ONNF, which didn't make sense. * 1. Check out code to ONNF directory. * 1. Pass path parameter correctly. * 1. Debugging buildbot. * 1. Remove debug code. * 1. Update installation instructions in README.md. 2. Enforce consistency with scripts used in testing using doc-check. * 1. Fix error with respect to syntax to build multiple CMake targets. * 1. Move doc-check to doc_check. 2. Remove directive_config in top-level driver. * 1. Build onnf and check-mlir-lit separately because only CMake 3.15+ supports building multiple targets in one cmake --build run. * 1. Use new env variables to locate LLVM-Project. * 1. Documentation nits. * 1. Prettify buildbot scripts. * 1. Fix build script error. * 1. Support exclude_dirs in DocCheck. 2. Add README for DocCheck. * 1. Mark python3 interpreter as required. 2. Use imported interpreter target. * 1. Automatically deduce doc file extension in DocCheckCtx. 2. Rename ctx.open -> ctx.open_doc since it should only be used to open doc file. 3. Always read line in parser, instead of reading lines in driver and then passing it to parser.py. * 1. Rename parser -> doc_parser due to name conflict with python built-in module. 2. Explose doc_check module directory first before importing; otherwise if the doc_check utility is invoked by other script, importing will not work correctly. * 1. Keep renaming parser -> doc_parser. 2. Explicitly define a default configuration parser that parses the configuration into a python dictionary. * 1. Add test for doc-check. 2. Exclude doc-check tests from project dock-check because base directory is different. * 1. Raise ValueError if directive configuration fails to parse. 2. Format code. * Shorten test case documentation. Show example of using same-as-file directive, check with DocCheck. * 1. Shorten test case documentation. 2. More documentation, check documentation with DocCheck. * 1. Add copyright notice. * 1. Make documentation clearer. 2. Prettify build-scripts. * 1. Provide more documentation. 2. Fix some non-compliance with pep8 recommendations. Co-authored-by: Gheorghe-Teodor Bercea <gt.bercea@gmail.com>
2020-01-10 07:35:52 +08:00
]
for directive in all_directives:
directive_config = []
if succeeded(directive.try_parse_directive(line, ctx.doc_file_ext(), directive_config)):
[RFC] Doc-check utility. (#12) * 1. Implement doc-check utility. * 1. Move ONNF installation script to a standalone script file. * 1. Modify build script to install llvm-project next to ONNF. The build script used to install llvm-project inside ONNF, which didn't make sense. * 1. Check out code to ONNF directory. * 1. Pass path parameter correctly. * 1. Debugging buildbot. * 1. Remove debug code. * 1. Update installation instructions in README.md. 2. Enforce consistency with scripts used in testing using doc-check. * 1. Fix error with respect to syntax to build multiple CMake targets. * 1. Move doc-check to doc_check. 2. Remove directive_config in top-level driver. * 1. Build onnf and check-mlir-lit separately because only CMake 3.15+ supports building multiple targets in one cmake --build run. * 1. Use new env variables to locate LLVM-Project. * 1. Documentation nits. * 1. Prettify buildbot scripts. * 1. Fix build script error. * 1. Support exclude_dirs in DocCheck. 2. Add README for DocCheck. * 1. Mark python3 interpreter as required. 2. Use imported interpreter target. * 1. Automatically deduce doc file extension in DocCheckCtx. 2. Rename ctx.open -> ctx.open_doc since it should only be used to open doc file. 3. Always read line in parser, instead of reading lines in driver and then passing it to parser.py. * 1. Rename parser -> doc_parser due to name conflict with python built-in module. 2. Explose doc_check module directory first before importing; otherwise if the doc_check utility is invoked by other script, importing will not work correctly. * 1. Keep renaming parser -> doc_parser. 2. Explicitly define a default configuration parser that parses the configuration into a python dictionary. * 1. Add test for doc-check. 2. Exclude doc-check tests from project dock-check because base directory is different. * 1. Raise ValueError if directive configuration fails to parse. 2. Format code. * Shorten test case documentation. Show example of using same-as-file directive, check with DocCheck. * 1. Shorten test case documentation. 2. More documentation, check documentation with DocCheck. * 1. Add copyright notice. * 1. Make documentation clearer. 2. Prettify build-scripts. * 1. Provide more documentation. 2. Fix some non-compliance with pep8 recommendations. Co-authored-by: Gheorghe-Teodor Bercea <gt.bercea@gmail.com>
2020-01-10 07:35:52 +08:00
directive.handle(directive_config.pop(), ctx)
return success(directive_config)
return failure()