Requirements

TODO: Before I really refine this, I need to refine and implement the parts of yc that this needs.

  • Use the filename rigbuild.* where the file extension can be any programming language file extension that has access to librig.

  • Be able to split the configure script into files, one in each directory that the users wants. In C, the user could just #include "dir/rigbuild.c", define a procedure, and have that procedure called.

  • Handle both configure and build.

    • Use multi-call binary.

    • For configure, use rigc.

      • For configure with curses, use crigc.

    • For build, use rig.

      • For build with curses, use crig.

        • If crig detects that configure is not necessary (see below), there is no need to open the curses GUI.

  • There needs to be a curses GUI.

    • The curses GUI should be able to generate completely every time “c” is pressed so that the user can stop whenever they like the options they have instead of needing to press it until all options appear and then pressing “g”.

  • Cross-compilation should be supported from day 1.

    • This is actually necessary, since rig will need a host compiler, so it must know the difference between target and host.

    • DarkUranium thinks that what my bc does (host compiler, host cflags) is sufficient, but I may also want to add host link flags and libraries.

    • To cross-compile, I could have an option to specify the cross-compile target.

  • Use all file attributes from https://apenwarr.ca/log/20181113 to detect changes to a file.

    • Only one attribute needs to be different, even mtime.

  • Provide a C library for build scripts.

  • Defaults should be required for every option.

    • This is so a rig invocation will do the right thing.

    • Should encourage the default to be whatever users will want to build. (Release mode with sane defaults for project-specific stuff.)

  • Allow user to set defaults with the environment (from DarkUranium on IRC). This could be by an environment file in the (build?) directory, or by environment variables. DarkUranium suggests some sort of config files, not environment variables, since env vars can only be strings.

  • Use relative paths so that generated files can be moved. See https://ofekshilon.com/2016/08/30/cmake-rants/

  • Do NOT generate targets that call into rig, so that if generated source files are put into source control (to make it easier on users), rig is not a dependency. This would require being able to generate Makefiles, ninja files, etc however.

  • Have output like ninja:

    • Configurable.

    • Use the same line (unless compiler warnings/errors happen).

    • Gather compiler warnings/errors from a step and output all at once, not a bit at a time. (Serialize the logs.)

  • Make it possible (and fairly easy) for users to change automatically added CFLAGS and other such things. This is to prevent things like https://stackoverflow.com/questions/34575066/how-to-prevent-cmake-from-issuing-implib.

  • There are three types of dependencies:

    • Configuration: These are the items that, if any are changed, require a reconfigure. I may want to split these into two:

      • Items that force a reconfigure, but do not require recompiling the configure script materials.

      • Items that force recompiling the configure script materials and force a recompile.

    • Build: These are the items that, if changed, require the equivalent of a make clean and rebuild.

    • Code dependencies: These are what we usually think of as dependencies, libraries and headers that the project needs, either at build time or runtime.

    • Target dependencies: These are dependencies between targets and/or other things defined in the build system.

  • Data on which versions of the project work with which versions of its dependencies could be a JSON file that would be imported by rig (which would automatically make that JSON file a config dependency).

  • When linking static libraries to dynamic libraries, the static libraries need to be compiled with -fPIC.

  • Support PHONY targets.

  • Have an option for a “fully correct” mode (as shown in https://chadaustin.me/2010/06/scalable-build-systems-an-analysis-of-tup/):

    • Hash files, to ensure that any changes redo the build/configure.

    • Use of a C preprocessor to ensure that header dependencies are complete and accurate.

    • Ensure that all implicit dependencies are found.

      • Compiler invocations. (Should be checked by default.)

      • Binaries of all tools used (compiler, linker, ar, user commands, etc).

      • Environment variables. (Should be checked by default.)

  • In tup, there are two types of nodes: commands and files. Files can depend on commands, and commands can depend on files. This makes commands a first-class citizen. Will that be necessary in rig?

  • In tup, arrows are backwards, but that’s because it can get notifications from the filesystem. I don’t want to depend on that, but architecting the code to make it easy to switch in the future would be nice.

    • Also, it might be good to start checking files at the leaves. When one is found, I could skip checking everything that depends on it.

  • A build that has incorrect results with correct use IS AN INSTANT BUG.

  • Allow source files to list their dependencies, like rcb2 (https://github.com/rofl0r/rcb2)? This shouldn’t be required, though.

    • How it actually works is for headers to list the files that their dependents also depend on.

    • I could make it so that headers can list rig targets that their dependents depend on.

rigc/crigc

  • Create a list of files used in previous configuration.

  • Create and fill the configuration database, which should contain the following things:

    • Filenames, timestamps, inode, etc (from redo).

    • Compiler/command invocations for every target.

    • Compiler/command versions.

  • Automatically create header dependencies by searching files for #include.

    • Default to searching for #(\s)*include (whitespace between # and include), but have a setting for not searching for that (for speed).

    • Label all as dependencies, even if that’s too conservative. I want a fast and easily built detection rather than a more involved one that will skip unnecessary building. Most #include‘s will be used anyway.

    • Note that rigc/crigc should only care about headers that change in user-defined include directories, not system directories, except when doing reproducible builds.

    • Include a full C preprocessor, which can be turned on for full correctness. In fact, the default should be on and turned off for performance.

      • DarkUranium told me about the possibility of someone doing something like #include SOMEMACRO.

      • Should probably adapt mcpp.

  • Before running the configure, delete any files in the old files list that are not in the new one.

  • Required command-line arguments:

    • Profile?

    • Directory where the source is, though if the configure files exist, assume it’s in the same place? Also, should I default to “..” (parent directory)?

rig/crig

  • Create a list of files created/used in the previous build.

  • Automatically detect when configuration is needed and if so, do what rigc/crigc would do before actually doing the build.

    • Check against the build database. If any of the environment variables, etc. are different from the build configuration, do the configure.

    • Also check the modification of all files that make up the configure script. If any are different, redo the configure step.

  • Create and fill the build database, which should contain the following things:

    • Filenames, timestamps, inode, etc (from redo).

    • Compiler/command invocations for every target.

    • Compiler/command versions.

  • Before running the build, delete any files in the old files list that are not in the new one.

  • Sort the most recently failed task first (subject to dependencies)? This would allow for faster iterating.

  • Schedule linking tasks for as soon as possible.

  • Required command-line arguments:

    • Target to build, though there should be a default target for if there is no target given.

  • Default targets (should be overridable):

    • clean: Clean build files.

    • distclean: Clean build and config files.

    • install: Obvious.

    • uninstall: Obvious.

    • list: List the available targets.

    • help: List the available targets with descriptions (if any).

librig

  • Procedures from parent directories should not be called directly. Instead, they should call something like rig_add_dir("<directory_name>", <directory_proc>), and rig_add_dir should add the <directory_name> to a stack and call <directory_proc>.

  • Procedures should receive a directory? The parent?

  • rig_add_dir() should be callable more than once on the same directory. This would be to add two different versions of the same git submodule, for instance.

  • rig_add_dir() needs to have an argument for adding a prefix to all targets in a directory (default to directory name + /?). This would be for when adding a directory twice, as above.

    • This should be used judiciously.

    • All targets in the directory need to be statically linked to the targets that depend on them to prevent symbol collision.

  • rig_add_dir() (or whatever does that job) should add configuration files to be checked for modification to the build or configuration database.

  • Source files should be able to be added by globs.

  • Add a way to turn a file into byte arrays. The function should also take a function pointer to do changes on the file like bc does on its library files.

  • librig probably needs some way to let a project say, “I have a subproject with an option named <x>, and I want it to be set to <value>. This would allow for projects to properly set things without needing recursive calls to rig.

  • Like the above, librig should provide some way for users to say that it is okay if an option is missing, or if it is an error if an option is missing.

  • Add an easy way to generate a config header.

  • Waf and Bazel let a target say what include directories that clients (items that depend on the target) need to include.

  • “The frozen and documented SCons API is fairly high-level while the (interesting) internals are treated as private APIs. It should be the opposite: a dependency graph is a narrow, stable, and general API. By simplifying and documenting the DAG API, SCons could enable broader uses, such as unit test execution.” From https://chadaustin.me/2010/06/scalable-build-systems-an-analysis-of-tup/

Debugging Builds

  • Profiling of build time?

  • Add a rigwhy command (like waf’s why) for people to be able to see why things needed to be built or reconfigured.

  • Other debug information to output can include:

    • Why something needs to be rebuilt.

    • The state of a task. (Whether it has run, has missing outputs, failed, skipped, or succeeded.)

    • The status of a task. (Not ready, skip, or ready.)

  • Be able to generate graphs of dependencies. (Will need an external tool for this.)

Documentation

  • MUST INCLUDE EXAMPLES!

  • Include comparisons to let people give to their managers to convince them to switch.

    • Benchmarks.

    • Features.

    • Non-features.

  • What NOT to do:

    • “And with that, they document the things Gradle does, not how to accomplish basic tasks using Gradle.” - From https://news.ycombinator.com/item?id=25807471

    • Should have API docs (what rig does) as well as functional docs (how to accomplish various certain tasks).

Issues

  • How do I handle the project name and rig minimum version (like CMake has)? Do I provide a riginit command that fills in a basic template?

    • If I do this, I could make it read a Makefile or whatever, or try to detect source files and generate commented-out targets for everything it detects WITH EXPLANATION COMMENTS that will help a noob understand what the commented-out stuff does.

    • If the directory is empty, I could have riginit fill out the directory with default directories (src, docs, lib, tests, etc.) and default rigbuild.* files.

    • Most importantly, riginit should create a main project script that has the boilerplate needed to allow the project to be easily added as a subproject.

  • How do I make it so users can pass args to the configure step and build step when using rig/crig when configuration is needed?

  • How do I make it so users can pass args to the configure script while still passing args to rigc/crigc themselves? Do I need to?

  • How do I pass arguments to the configure script while also passing args to the configure step and the build step?

  • How do I make it so that C scripts, which should be runnable as git submodules, can be called? Obviously, using main() makes that impossible. In fact, if I don’t use main(), I can ensure that the user’s command-line args are parsed in the right way without their help.

  • Handling incompatible library versions will be hard. I think developers should be able to say what versions of their library are compatible with each other.

  • If I implement something like CMake’s find_package() with find files, that can be installed when installing a library, along with something like the above, then the version compatibilty information should be in the find file.

  • Find scripts should be automatically generated, if I use them.

  • Profiles (sets of defaults) could be used to make it easy to build different versions.

  • DarkUranium on IRC doesn’t want profiles to be mixed with build types (Release, Debug, RelWithDebInfo, MinSizeRel). He thinks this will lead to a combinatoric explosion of profiles. I agree.

    • One thing we talked about was that a profile doesn’t have to have settings for all options. For the ones that it doesn’t have a setting for, it just uses the default.

    • Instead of splitting the build type profiles from the regular profiles, I could instead have a system where a user can specify more than one profile (split by colons on the command-line?). The first profile would set settings first, the next one would set them, overriding the first where they don’t agree, etc.

    • This will allow the build type profiles to be separate but still use the same code while allowing arbitrary nesting of profiles, if necessary.

  • How much boilerplate is okay?

  • Separate computed variable values from parameters set by the user? See https://news.ycombinator.com/item?id=24203503

  • Use compilation of small files, like autotools, to detect dependencies? This would mean that most missing dependencies are caught at configure time.

  • How do I support building for multiple architectures at once? See https://news.ycombinator.com/item?id=17899118

  • How would I do user-definable dependencies? By allowing the user to write a function that returns a bool and then use it as a function pointer? This would require librig to provide a way for users to store and retrieve data from the config and build databases.

    • The above might be how I implement dependencies on the compiler version, linker version, CFLAGS, etc. See the Shake build system.

  • The config script files might have dependencies on other files, such as JSON files that hold necessary data for the build. How do I make it easy to add those files as configuration dependencies?

  • Can I figure out a way to make Debug the default for developers and Release the default for users?

    • Maybe I can do that by allowing users to provide a “release” script, a script to run on release.

      • This would be convenient; I already have something like that for bc.

      • Also, librig could have a function to switch the default to Release or something.

Future

  • Parallel processing by default, and a -j/--jobs flag. Maybe even --load?

  • Be able to parse configs from ldconfig, pkg-config, gtk-config, llvm’s config, etc.

  • Be able to produce pkg-config .pc files.

  • Implement make and ninja. This is to more easily integrate projects that use them.

  • When implementing make, only make it work with sane Makefiles? (From pixelherodev on IRC.)

  • When implementing make, only import targets that matter? (From pixelherodev on IRC.)

  • Make it possible for rig to run executables? This isn’t easy (see https://anteru.net/blog/2017/build-systems-bazel/), since it requires making sure generated shared libraries are in the search path, but it might be good.

    • It would also require some rethinking of command-line options.

    • Or add a new command rigrun.

  • Include API’s for the following:

    • C++

    • Python

    • Lua

    • Zig?

    • And more.

  • Make it possible for the user to add their own API?

  • Inplement a way to have canonical headers? (See https://anteru.net/blog/2012/canonical-include-files-for-frameworks-libraries/)

  • Implement a server for package management. This is to implement rigkits.io and for people to create their own package managers. The above features need to be in place before this because in order to have packages from many projects, I need to be able to make the packages myself, using the the projects already do.

    • If the server can detect when an upgrade to a minor version causes trouble for clients, that would be cool. It would make it so that clients could have word of troublesome stuff.

    • Users should be able to specify pre-downloaded dependencies.

  • All of the above are needed for reproducible (deterministic) builds.

  • Once I implement make and ninja, generate a Makefile and ninja file to be checked into source control for yc, which should allow projects to use yc and rig as a submodule with almost no effort.

  • Integrate with VCS (that I still need to build) to make it easy to check that the distribute build artifacts are, in fact, deterministically generated from the correct source.

  • Implement a ccache-like cache? This should only be activated if deterministic builds are turned on.

    • ccache checks the following things:

      • The compiler binary (rig should check all redo attributes)

      • Preprocessed headers for changes (should be easily implementable without preprocessing headers by simply checking system headers for changes as well; changes to project headers will already be checked).

      • Compiler options (will already be checked).

    • ccache does not cache link steps!

  • Estimated completion time, like Shake?

  • Build order constraints (like waf). This can allow targets that don’t depend on each other to be able to be run before/after each other.

    • Could also implement a priority system so that users can tell rig which tasks take the longest.

      • Could also implement the above by reading profile times from the database, if profile times are written to the database.

  • Build groups, like waf.

    • Could also allow the user to throttle certain groups, like waf.

  • Handle a clang compilation database? (See https://github.com/rizsotto/Bear/blob/master/README.md.) This would need very little on top of what I need in the initial version.

  • A rig daemon that watches for changes?

Dependencies

  • Allow dependencies to be used by dependencies in a flat format.

  • However, if dependency versions conflict, error.

    • But allow the user to override and specify a specific version instead.