Skip to content

Developing for the trestle documentation website¤

This page describes the developing for the trestle (website) which is deployed at https://ibm.github.io/compliance-trestle.

Documentation for use within the github project.¤

Github uses certain files within a project such as /README.md, /CONTRIBUTING.md, LICENSE which are specifically indexed by github. The current documentation website build reuses some of these files, specifically:

  • Contents of README.md
  • Entirety of LICENSE
  • Entirety of CONTRIBUTING.md
  • Entirety of CODE_OF_CONDUCT.md
  • Entirety of CHANGELOG.md
  • Entirely of MAINTAINERS.md
  • Entirely of DCO1.1.txt

For this to work correctly no relative links within the github repository should exist. All links should be absolute to the documentation website.

Build system and local testing of the website.¤

Trestle has adopted the mkdocs system to generate this website using a small number of extensions to mkdocs. The website can be viewed locally from a clone of the compliance-trestle repo by running make docs-serve in the root directory bringing the website up at https://localhost:8000. If you experience issues run make develop to ensure the appropriate markdown extensions are in your python environment.

make docs-serve performs two actions:

  • Runs the custom automation script scripts/website_automation.py
  • Serves the website on localhost.

All documentation specific assets are stored within the ./docs folder. The exception being mkdocs.yml which configures the documentation tree. Before opening a PR users should ensure:

  • No warnings are generated by mkdocs
  • All markdown documents within ./docs are included in the website navigation defined in mkdocs.yml

trestle custom automation.¤

In order to streamline development, and ensure the website remains up to date, a small automation script has been built. This automation script principally ensures that:

  • License is consistent with github.com
  • All modules are in the reference documentation

running make docs-automation will ensure that the website is ready to deploy.

Building the models from the OSCAL schemas.¤

The creation of the OSCAL models in trestle/oscal is a multi-step process:

  • The oscal schemas are downloaded as modules from NIST into the nist-source/json/schema directory.
  • The script scripts/gen-oscal.py loads each schema file and converts it to pydantic/python with datamodel-codegen.
  • The generated python files may need some fixup, so a separate script scripts/fix_any.py is run on each file.
  • Note that there is one schema specific to IBM needs and it is loaded from 3rd-party-schema-documents/IBM_target_schema_v1.0.0.json.

The whole process is handled in the Makefile by make code-gen. A normal user would never need to run this but developers may need to, particularly if there are changes to the OSCAL schemas.

Also note that the depenedent tools, pydantic and datamodel-codegen, may get updated by doing a fresh make install or make develop, which may then result in a change to the model files.

Items handled by fix_any.py.¤

The original motivation for this script was to replace numerous situations where the type assigned to a given variable was simply Any, which meant no type enforcement would apply for that variable, defeating the purpose of the strict type enforcement provided by Pydantic. As of this writing the number of such cases has been reduced to just one - which is handled by the script.

Other issues handled by the script are:

  • The current OSCAL schemas have situations where objects are defined within different classes in a schema using the same name, but the contents of those classes may or may not be different. datamodel-codegen handles this by creating separate classes as needed and appending 1, 2 etc. to the names, keeping them distinct. The resulting high level classes that reference them behave as expected, but if components of those classes are added in a granular way by a user or developer, the correct index must be used.
  • To reduce side-effects of the duplicate classes, classes are checked to see if they are identical or not. If they are identical the separate 1, 2 classes are culled and references to them are pointed to the non-indexed class.
  • Currently there are 5 classes that require a separate '1' version: Status, Type, Entry, LocalDefinitions, and Action.
  • In order to guarantee there are no induced forward references in the files, the classes are reordered to minimize the need for forwards, and any that can't be avoided are explicitly provided at the bottom of the file.
  • The generated files have many classes that simply have a __root__ element defined, along with a description. Such classes don't have particular value in such a simple form and could instead simply be defined in the parent class.

Seeing the changes induced by fix_any.py on the classes.¤

As a convenience for developers, a separate script, scripts/order_classes.py is available, which orders the classes in a given file alphabetically. This way, if you use the script on files before and after applying fix_any.py you can use a normal diff tool to see the changes made. This is strictly as a development tool for doing the comparison and the resulting files will not work since they will have forward references.