Process

This process is for me (Gavin D. Howard) as I develop programs in C (and Yao once it exists). It is meant to create high-quality, robust, secure systems software.

Purpose

To guide the development of large systems software, written in C or Yao, through a series of subprojects, ensuring that the software is high-quality, robust, and secure at the end of each subproject.

General Instructions

This script is to be used for every subproject, so the entry and exit criteria apply to each subproject.

This is also how this process avoids the problems that Waterfall causes.

This script is meant to be comprehensive. This means that some activities, and maybe even some phases, may be skipped entirely. That is okay.

After each subproject, all known technical debt, reported in the form of defects, must be eliminated.

When an item begins with “For each”, treat each subitem as its own separate checklist item and go through the entire software for that subitem.

Entry Criteria

These criteria are for the entry of every subproject.

  • Problem description.

  • psp.py tools.

Exit Criteria

These criteria are for the exit of every subproject.

  • A thoroughly tested program.

  • Accurate data in personal PSP repo from psp.py use through the process.

  • Elimination of known technical debt.

Phases

Planning

Purpose

To guide the planning process for C and Yao software.

General Instructions

Resource estimates will be off until data is collected.

They will also be wrong until the suite of tools used in every program is available.

Until then, Resource Estimate can be skipped.

Entry Criteria

  • Problem description.

  • psp.py tools.

Exit Criteria

  • Documented requirements list.

  • Resource estimates (if done).

Activities

Start Subproject
Purpose

To start the subproject in psp.py tools.

General Instructions

The psp.py tools will ask for estimates. This is okay.

The reason for it is for the planner to get data about their estimate instincts.

The actual estimates (from the Program Requirements Activity and Resource Estimate item) should be stored in the software repository, not the planner’s PSP repo.

Checklist
  • Choose a name for the subproject.

  • Create the subproject using psp.py tools, using best instinct estimates.

Software Requirements
Purpose

To ensure that the requirements list is clear, is unambiguous, has no conflicts, and has no outstanding questions.

General Instructions

This is one of the most manual, error-prone, skill-intensive, and creativity-oriented parts of the process.

Ensuring that requirements are clear and unambiguous requires carefully spelling out everything. Unfortunately, spelling out everything requires having an eye for ambiguity.

Also, resolving conflicts between requirements requires looking at N! or more relationships, where N is the number of requirements.

The planner needs to specify functional requirments (what the software should be be able to do).

The planner needs to specify algorithmic requirements (the big-O limits in space and time).

The planner needs to specify performance requirements (how fast the software needs to run according to the big-O limits, limits on its memory usage according to the big-O limits, limits on disk usage according to the big-O limits, etc.).

The planner needs to specify security requirements (making sure secrets are wiped for cryptography, making sure encryption and decryption are constant-time, whether DoS attacks need to be mitigated, access control, etc.).

The planner needs to specify safety requirements (what the software should absolutely not do).”,

All of the above means that the skill of the planner is the most important factor in the success of this activity.

Also, if there are any requirements that don’t fit any of these categories neatly, put them into Functional Requirements.

Examples would be requirements for a DSL that is part of the software.

Checklist
  • Functional Requirements:

    • List everything the software should do.

    • Clear up any ambiguity.

    • Identify and resolve conflicts.

  • Algorithmic Requirments:

    • Identify every possible portion of the software that may run an algorithm.

    • Decide on time algorithmic requirements. The requirements may change with the size of the problem. If so, list both or all requirements and the size(s) where they should be split.

    • Decide on space algorithmic requirements. The requirements may change with the size of the problem. If so, list both or all requirements and the size(s) where they should be split.

    • Identify and resolve conflicts between time and space algorithmic requirements.

    • Identify and resolve conflicts between algorithmic requirements and functional requirements.

  • Performance Requirements:

    • Identify the expected scale (possible size of the problem) that the software must reach.

    • Identify the time resources that the software will be allocated.

    • Using the expected scale and the time resources, calculate the needed performance requirements for each type of time resource.

    • Identify the space resources that the software will be allocated.

    • Using the expected scale and the space resources, calculate the needed performance requirements for each type of space resource.

    • Identify and resolve conflicts between time and space performance requirements.

    • Identify and resolve conflicts between performance requirements, algorithmic requirements, and functional requirements.

  • Security Requirements:

    • Create the threat model.

    • Clear up any ambiguity in the model.

    • Identify all types of attacks in the threat model.

    • Clear up any ambiguity in the types of attacks.

    • For each type of attack in the threat model, come up with a security requirement to mitigate it, or if necessary, eliminate it.

    • Clear up any ambiguity in the security requirements.

    • Identify and resolve conflicts between security requirements.

    • Identify and resolve conflicts between security requirements and all previous types of requirements.

  • Safety Requirements:

    • List everything the software must not do. (Cannot crash, cannot drop messages, etc.)

    • Clear up any ambiguity.

    • Identify and resolve conflicts between safety requirements.

    • Identify and resolve conflicts between safety requirements and all other types of requirements.

  • Final Checks:

    • Clear up any remaining ambiguity.

    • Identify and resolve any remaining conflicts between requirements.

    • Identify and resolve any remaining questions.

    • Check all requirements into the software repository.

Resource Estimate
Purpose

To estimate the amount of time and resources it will take to accomplish the subproject.

General Instructions

If an estimate is not necessary, this step can be skipped.

However, once there is good data for making estimates, it might be worth creating them anyway.

Checklist
  • Identify all programmers who will be working on the project.

  • Gather time estimates from all programmers.

  • Use the estimates to make your best estimate of the time required for the subproject.

Design

Purpose

To design the software (or changes to the software), create or obtain the specifications, and design the test tools and methods.

General Instructions

Like specifying the requirements, design takes a lot of creativity and skill.

Very little of this part of the process can be automated.

In fact, this phase of the process should take as long as the rest of the phases combined.

The reason for that is that if this phase is done right, the rest of the phases should be close to busywork; the hard part will be over.

Entry Criteria

  • Requirements list.

Exit Criteria

  • Complete design for the software.

  • Complete formal specification for the software (if applicable).

  • Complete design for the software’s testing.

Activities

Design
Purpose

To design the software (or changes to the software).

General Instructions

The design process should be an iterative process, switching between designing from the top all the way down, starting with high level design and working down by filling in more and more details, including data structures, until the bottom is reached.

Then design from the bottom all the way up, starting with the bottom procedures, building them in pseudocode using Correctness by Construction, and work all the way up.

Repeat as necessary.

The iteration will be necessary as the bottom-up design results (API’s) will reveal weaknesses with the top-down design, and the results of the top-down designs (architecture) will show weaknesses with the bottom-up design.

Eventually, there should be a stable design with no changes, and the design process will be done.

Data structures should be considered while going top-down (contents, etc.) and bottom-up (order, indirection, etc.).

Checklist
  • Review the requirements.

  • Produce a design to meet the requirements, iterating down and up until the design is stable.

  • Document all state machines.

  • Document all logic in pseudocode.

Design Review
Purpose

To ensure the completeness and correctness and correctness of the design.

General Instructions

Review the entire software for each checklist item; do not attempt to review for more than one item at a time.

Checklist
  • Preparation:

    • Examine the software and checklist and decide on a review strategy.

    • Identify all state machines in the design.

    • Identify all internal loops in the design.

    • Identify all variable limits in the design.

    • Identify all system limits in the design.

    • Use a trace table or another analytical method to verify the correctness of the high level design.

  • Completeness:

    • Verify that the design covers all applicable requirements.

    • Verify that the design allows all specified outputs to be produced.

    • Verify that the design requires all needed inputs to be furnished.

    • Verify that all required includes/imports are stated.

  • External Limits (Where Applicable):

    • Document all external limits.

    • Determine if behavior is correct at nominal values.

    • Determine if behavior is correct at limits.

    • Determine if behavior is correct beyond limits.

  • Logic (from Correctness by Construction):

    • Use a trace table, mathematical proof, or similar method to verify the logic.

    • Verify that program sequencing is proper.

    • Verify that data structures are in the proper order.

    • Verify that recursion has a base case.

    • Verify that recursion unwinds properly.

    • Check for and document any accidental indirect recursion.

    • Verify that all loops are properly initiated, incremented, and terminated (invariants, variants, postconditions, and preconditions are correct).

    • Examine each conditional statement and verify all that cases are complete and correct.

    • Examine each conditional statement and verify all that cases are complete and correct.

    • Add asserts, preconditions, and postconditions for all assumptions in functions.

    • Add asserts, preconditions, and postconditions for all uses of libraries.

    • Add invariants for all assumptions on data structures.

  • State Machine Analysis:

    • For each state machine, verify that the state transitions are complete and orthogonal.

    • Document each state machine.

  • Internal Limits (Where Applicable):

    • Document all internal limits.

    • Determine if behavior is correct at nominal values.

    • Determine if behavior is correct at limits.

    • Determine if behavior is correct beyond limits.

  • Special Cases:

    • Check all special cases.

    • Ensure proper operation with empty values for all variables.

    • Ensure proper operation with full values for all variables.

    • Ensure proper operation with min values for all variables.

    • Ensure proper operation with max values for all variables.

    • Ensure proper operation with negative values for all variables.

    • Ensure proper operation with zero values for all variables.

    • Protect against out-of-limits conditions.

    • Protect against overflow/underflow conditions.

    • Ensure that all “impossible” conditions are documented and absolutely impossible.

    • Handle all possible incorrect or error conditions and document them.

  • Data Structures:

    • Verify that all data structures are fully understood.

    • Verify that all data structures are properly documented.

    • Verify that all data structures are properly used.

  • Functions/Procedures:

    • Verify that all functions/procedures are fully understood.

    • Verify that all functions/procedures are properly documented.

    • Verify that all functions/procedures are properly used.

  • Security Considerations:

    • Verify that all security-sensitive data are from trusted sources.

    • Verify that the threat model is complete and comprehensive.

    • Verify that the list of attacks is comprehensive for the threat model.

    • For each type of attack, verify that the software properly mitigates, or eliminates, threats of all attacks.

  • Safety Considerations:

    • Verify that the software does not cause system limits to be exceeded.

    • For each safety requirement, verify that the software does not violate that requirement.

  • Names:

    • Verify that all special names are clear, defined, authenticated, and documented.

    • Verify that the scope of all variables and parameters are defined and documented.

    • Verify that all named items are used within their declared scopes.

  • Standards:

    • For each applicable design standard, verify that the design conforms to the design standard.

  • Check:

    • For each design defect, fix the defect and check that the fix is correct.

    • If there were defects, go back to the beginning of the Design Review activity and start again.

    • Check all design documents into the software repository.

Specification
Purpose

To ensure that a specification of the software is documented.

General Instructions

This activity should not be skipped, except when a documented specification is entirely unnecessary and not useful at all.

However, most of the time, a documented specification is useful, especially for documentation used by shareholders.

While detailed specifications are especially useful for languages or programs, such specifications may also be useful for libraries.

Checklist
  • Decide what language the specification will be written in, as well as any languages it will be translated into.

  • Write the specification in the chosen primary language.

  • Translate the specification into the other languages.

  • Check the specification and translations into the software repository.

Specification Review
Purpose

To ensure that the specification is understood by all programmers and that it meets the expectations of shareholders.

General Instructions

If the Specification activity is done, this activity should be done.

The only exceptions should be when there are no shareholders available yet, and there is only one programmer working on the project.

Checklist
  • Shareholder Review:

    • Have every shareholder review the specification and submit questions.

    • Update the specification and translations to answer the questions.

    • Have every shareholder submit desired changes.

    • Negotiate changes with all shareholders.

    • Make the negotiated changes to the specification and all translations.

    • Repeat the above five steps until there are no more questions or disagreements.

  • Programmer Review:

    • Have every programmer review the specification and submit questions.

    • Update the specification and translations to answer the questions.

    • Have every programmer submit objections.

    • Take the objections to the shareholders for renegotiation.

    • Make the negotiated changes to the specification and all translations.

    • Repeat the above five steps until there are no more questions.

Formal Specification
Purpose

To ensure that a specification of the software is formalized.

General Instructions

This activity may be skipped.

If this activity is done, it can be done in a language like TLA+ or Alloy that would make it possible to do model checking in the Formal Specification Test activity.

This activity can even be partially done. For example, programmers might make a formal specification of protocols that will be implemented or relationships between some data, but not formally specify the whole system.

It is highly recommended that this activity not be skipped, even if it is only done partially.

Checklist
  • Decide on the tools that will be used to create the formal specification.

  • Create the model.

  • For each behavior in the model, create assertions about the behavior.

  • For each limit, create assertions about the limit.

  • Do all other work necessary to create the specification.

Formal Specification Review
Purpose

To ensure the correctness of the formal specification according to the decided-on design.

General Instructions

If the Formal Specification activity is not done, this activity does not need to be done either.

Checklist
  • Limits:

    • For each external limit, verify that the specification checks the limit.

    • For each internal limit, verify that the specification respects the limit.

    • For each system limit, verify that the specification respects the limit.

  • Logic:

    • For each loop, verify that the formal specification correctly describes the loop.

    • For each conditional statement, verify that the formal specification correctly describes the conditions and the cases.

    • For each instance of recursion, verify that the base case and non-base case are correctly described and separated.

    • Verify that all state is correctly described.

  • State Machines:

    • For each state machine, verify that the specification properly specs the state machine.

Formal Specification Test
Purpose

To ensure that the design does not have defects by testing the formal specification of the design.

General Instructions

If the Formal Specification activity is not done, this activity does not need to be done either.

This activity also does not need to be done even if a formal specification is created.

However, it is highly recommended to do so, even if full formal proofs are not employed, since even model checking can find defects in the design.

Checklist
  • Model Checking (Where Applicable):

    • If the formal specification is written in a language that supports model checking, run the model checker on the specification.

    • Fix all defects.

  • Formal Proof (Where Applicable):

    • If the formal specification is written in a language that supports formal proofs, and doing a proof is feasible, prove the correctness of the specification.

    • Fix all defects.

Test Design
Purpose

To design the testing regimen for the software.

General Instructions

This activity is necessary in all cases, even when a formal proof of the specification is created.

The reason for that is because the testing regimen will then be used to test the correctness of the implementation.

For most paths through the code, the chosen method will be property-based testing, with failing test cases saved.

For tests that don’t have an oracle (an easy way to get a right or wrong answer, use metamorphic testing.

For the most difficult paths, fuzzing may be the only viable option.

Some paths may require “setup,” in the form of setting up state with multiple uses of the software (which means that the paths that need setup are dependent on the paths that are run during the setup).

Those paths that technically do the same thing, but require different setup, should be considered separate paths.

Possible test methods include (but are not limited to): unit testing, integration testing, system testing, exhaustive testing (usually only unit tests for pure functions), property-based testing, metamorphic testing, fuzzing, dynamic analysis, error injection, performance tests, and algorithmic tests (to test the performance difference as problem sizes grow).

Test coverage should be tested with mutation-driven testing.

Checklist
  • Requirements:

    • For each non-functional requirement, identify and document whether the requirement is worth testing.

    • For each non functional requirement worth testing, identify and document at least one test method for testing the requirement.

  • Paths:

    • For each loop, identify and document paths to get zero, one, and many iterations.

    • For each recursion, identify and document paths to get zero, one, and many recursions.

    • For each condition, identify and document paths to reach all cases.

    • Identify and document interaction between paths to identify all possible paths, including which paths depend on other paths.

    • For all paths, identify and document whether the path is worth testing.

    • For all paths worth testing, identify and document at least one method to test that path.

  • Tools:

    • For all test methods, identify and document a tool needed to accomplish that method.

    • For all tools that exist, ensure that access is available.

    • For all tools that don’t exist, design the tool using the Design checklists.

  • Tests:

    • For each non-functional requirement worth testing, design and document at least one test for each method that was decided to test the requirement.

    • For each path worth testing, design and document at least one test for each method that was decided to test the path.

Test Design Review
Purpose

To ensure that the testing regimen design is complete, viable, and correct.

General Instructions

Testing is expensive; therefore, it is necessary to ensure that each test is viable before doing it.

This is the reason that fuzzing may be the only viable option for some paths.

Checklist
  • Tools:

    • For all tools that do not exist, review the design of the tool using the Design Review checklist.

  • Complete:

    • Verify that the list of paths is complete and comprehensive.

    • For each test, verify that the test completely tests the path it is supposed to test.

  • Viable:

    • For each path worth testing, verify that at least one test method is viable for testing that path.

    • For each test method, verify that the tool used to accomplish that method is viable and sufficient.

Code

Purpose

To actually create the software.

General Instructions

The programmer(s) should not attempt to compile or run the software during this entire phase.

This will be difficult, as it goes against instinct, but it will slow down the process and reveal nothing.

At least, it will reveal nothing if the Design phase was done correctly.

Entry Criteria

  • A fully fleshed-out design for the software (or changes).

  • A fully tested formal specification (if applicable).

Exit Criteria

  • Zero technical debt from the changes.

  • Completed and reviewed software.

  • Updated and completed documentation comments.

  • Updated and completed comments necessary for understanding the code.

Activities

Remove New Technical Debt
Purpose

To remove all new technical debt caused by the design of the project.

General Instructions

After designing a project, there will be required changes to the existing codebase to prepare it for the project.

These changes are considered new technical debt.

This technical debt should be removed before implementing the project, and that’s what this activity is about.

Checklist
  • Identify each item of technical debt.

  • For each item of technical debt:

    • Remove the item.

    • Make any changes required to fix the build.

    • Make any changes required to fix the tests.

  • For all changes:

    • Write necessary design comments.

    • Write necessary why comments.

    • Write necessary teacher comments.

    • Write necessary checklist comments.

    • Write necessary debt comments.

    • Split the code into code paragraphs and write necessary guide comments.

  • Formatting:

    • Format the code, usually with an automatic formatter.

Code
Purpose

To actually code the software (or changes to the software).

General Instructions

Once the previous phases are done, this phase should be more busywork, even mechanical, than anything else.

However, it will require deep attention to detail to implement everything exactly as specified and put the same assertions, preconditions, postconditions, invariants, and variants in the code.

All of the code should be commented to make it easier to understand the code beyond the required documentation comments.

There are several types of useful comments beyond documentation comments.

Function and documentation comments document external and internal API’s and data structures.

Design comments are high-level overviews of algorithms and techniques and are what/how in macro.

Why comments should document the reasons behind why the code is the way it is, which can include talking about previous attempts and why they failed, making it clear why certain decisions were made.

Teacher comments explain the domain the code is operating in.

Checklist comments tell the programmer what needs to be changed if a certain piece of code has to be changed.

Debt comments are markers of technical debt and include things like TODO’s and marking where subpar implementations or algorithms are used.

Guide comments are what/how comments in micro, like design comments are in macro.

In micro, code should be split up in “code paragraphs”, though splitting code up like this requires judgment.

Not every code paragraph needs to be commented, but a lot will be, and those are the guide comments.

When deciding what code paragraph comments to add, try to identify the context hidden in your head in words, and put that into the comments.

Comments are parity checks, and should be used as such.

Assumptions should always be documented in some form of assert and never in comments.

Checklist
  • Form:

    • For each function/procedure in the design, create a stub for the function/procedure and write the documentation comment for it.

    • For each data structure in the design, create the data structure and document it with documentation comments.

    • For each data structure in the design, write the data structure’s invariant(s) down as assertions in the functions/procedures that operate on the data structure.

    • For each function/procedure in the design, write the preconditions and postconditions down as assertions in the function/procedure.

  • For each function/procedure:

    • Fill in the function/procedure.

    • For each loop, write down the postconditions(s), precondition(s), invariant(s), and variant(s) as assertions.

    • For each conditional statement, verify that the condition(s) are mutually exclusive and complete, including “impossible” conditions, which should be written down as negative assertions.

    • Write down all remaining assumptions as assertions.

    • For each basic block, put a failing assert at the beginning of the block.

  • For all code:

    • Write necessary design comments.

    • Write necessary why comments.

    • Write necessary teacher comments.

    • Write necessary checklist comments.

    • Write necessary debt comments.

    • Split the code into code paragraphs and write necessary guide comments.

  • Formatting:

    • Format the code, usually with an automatic formatter.

Code Inspection
Purpose

To have every programmer inspect the code they each produced during the previous activity for defects.

General Instructions

This phase should be extremely thorough, taking about 10% of the total time for the subproject.

Considering that the Design phase should take 50% of the development time, Code Inspection will probably be the biggest user of time in the other 50%, besides maybe the Test phase.

If the Design Phase and Code Inspection phases are done correctly, however, the Test phase should be shorter than the Code Inspection phase.

When declaring non-changing variables const, do it even for non-pointer parameters.

When declaring pointers const, remember to declare both data and pointer const as applicable, going deeper for pointers to pointers, if necessary.

Declaring data const looks like: const int *p = &anInteger;.

Declaring pointer const looks like: int * const p = &anInteger;.

Checklist
  • Complete:

    • Verify that the code covers all of the functional requirements.

    • Verify that the code covers all of the design.

  • Includes/Imports:

    • Verify that the includes/imports are complete.

  • Initialization:

    • Check variable, const, and parameter initialization at program start.

    • Check variable, const, and parameter initialization at the start of every loop.

    • Check variable, const, and parameter initialization at function/procedure entry.

  • Names:

    • Check name spelling and uses.

    • Check consistency of names.

    • For each name, check that the name is within the declared scope.

    • For each name, check that the name does not shadow another name.

    • For each struct member reference, verify that the correct operator (-> or .) is used.

    • Check that all non-changing variables are declared const, including all levels for pointers (data and pointer are const), and even further for pointers to pointers, etc.

    • Check for void in the header of every procedure that does not take arguments (C only).

  • Strings:

    • Check that all strings are identified by pointers.

    • Check that all strings have a length prefix.

    • Check that all strings are terminated by a nul character.

  • Assertions:

    • Check that all preconditions exist in the code as complete and correct assertions.

    • Check that all postconditions exist in the code as complete and correct assertions.

    • Check that all loop postconditions exist in the code as complete and correct assertions.

    • Check that all loop preconditions exist in the code as complete and correct assertions.

    • Check that all loop invariants exist in the code as complete and correct assertions.

    • Check that all loop variants exist in the code as complete and correct assertions.

    • Check that all data structure invariants exist in the code as complete and correct assertions.

    • Check that the preconditions for libraries exist in the code as assertions right before every call to those libraries.

    • Check that the postconditions for libraries exist in the code as assertions right after every call to those libraries.

    • Check that all other assumptions exist in the code as complete and correct assertions.

  • Operators:

    • Verify proper use of =.

    • Verify proper use of ==, &&, ||, and ^^ (Yao only).

    • Verify proper use of bitwise operators, including checking for overflow and underflow.

    • Verify absence of divide-by-zero through checks, assertions, or other techniques.

    • Verify proper use of non-wrapping arithmetic operators, including checking for overflow and underflow, asserting its impossibility, or using wrapping operators.

    • Verify proper use of wrapping arithmetic operators or functions.

    • Verify proper use of () to adjust precedence.

  • Calls:

    • Check function call formats for correct usage of pointers.

    • Check function call formats for correct usage of parameters.

    • Check function call formats for correct usage of address-of operator.

  • Recursion:

    • Check for, and document, any indirect recursion. Use a tool if necessary.

    • Check that all recursion has a base case (will terminate).

    • Check that the condition for the base case is complete and correct.

    • Check that the normal case unwinds the stack properly.

  • Loops:

    • For each loop, verify that the loop’s preconditions are met.

    • For each loop, verify that the loop meets its postconditions.

    • For each loop, verify that the loop meets its invariants.

    • For each loop, verify that the loop properly updates its variants.

  • Conditionals:

    • For each conditional, check that the condition is complete and correct.

  • Sequencing:

    • Check for proper sequencing of program statements.

    • Check for any sequence points undefined behavior (C only).

  • Memory Safety:

    • Check that all pointers are initialized NULL.

    • Check that all pointers are not free()’ed before allocation.

    • Check that allocated pointers are always free()’ed (no memory leaks).

    • Check that all allocated pointers are only free()’ed once (no double frees).

    • Check that all allocated pointers are not used after they are free()’ed (no use-after-free).

    • Check for possible stale pointers (pointers to “borrowed” data that might outlive the data).

    • Check that all possible buffer accesses are either protected by bounds checks or that a buffer overrun is impossible.

  • Concurrency:

    • Check for possible data races.

    • Check for possible race conditions.

    • Check for improper thread use (invalid joins, etc.).

  • Output:

    • Check that line stepping is proper.

    • Check that spacing is proper.

  • Files:

    • Verify that all files are properly declared.

    • Verify that all files are properly opened.

    • Verify that all files are properly closed.

  • Cryptography (Where Applicable):

    • Check that all cryptographic operations are constant-time.

    • Check that all buffers are zero’ed after use.

    • Check that the compiler cannot optimize out the buffer zero’ing.

    • If possible, check that all registers are cleared.

    • Check that all secrets are wiped from memory.

  • Other Requirements:

    • For each algorithm in the code, verify that it meets applicable time algorithmic requirements.

    • For each algorithm in the code, verify that it meets applicable space algorithmic requirements.

    • For each security requirement, verify that nothing in the code violates that requirement.

    • For each safety requirement, verify that nothing in the code violates that requirement.

  • Documentation:

    • For each data structure, ensure that the data structure is properly documented, including member names and types, as well as invariants.

    • For each function/procedure, ensure that the function/procedure is properly documented, including parameter names and types, as well as preconditions and postconditions.

  • Syntax:

    • Check that all () are proper and matched.

    • Check that all [] are proper and matched.

    • Check that all {} are proper and matched.

    • Check for proper punctuation.

    • Check for other syntax errors.

  • Standards:

    • For each applicable standard, verify that the code conforms to the standard.

Code Review
Purpose

To allow every programmer on the team to review the code written by every other programmer.

General Instructions

While this step may take a long time, it should not be skipped if there is more than one programmer.

Checklist
  • For Each Programmer:

    • For each other programmer, have the first programmer review the code from the second according to the Code Inspection checklist.

  • Review the code as a team.

Compile

Purpose

To compile the code and remove as many defects as possible before testing.

General Instructions

This phase is designed to remove as many defects as possible before running the code even once.

The instinct of the programmers on the team will be to run the software as soon as it’s compiled, but it should not be run.

Any software execution should wait until the Test phase.

Entry Criteria

  • Complete and reviewed software.

Exit Criteria

  • Compilable software with no statically-detected defects.

Activities

Build Process
Purpose

To create the build process used to compile the software.

General Instructions

This activity takes place after all of the coding so that the shape of the code is well-known.

It also takes place after so that programmers are less tempted to compile the program while in the Code phase.

The above facts are important for creating a build process that is as streamlined as possible.

Checklist
  • Build Options:

    • Using the design, a list of build options should be created.

    • For each build option, add the build option to the build process.

  • Add all material that must be generated to the build process.

  • For Each Programmer:

    • The programmer should ensure that the files that he wrote the majority of are added to the build process.

    • The programmer should ensure that the dependencies of the files are properly resolved.

  • Final:

    • Do the rest of the work needed to create the build process.

    • Review the build process for correctness.

Compile
Purpose

To get the software properly compiling.

General Instructions

Only one programmer on the team can do this step. The rest might be able to start work on creating the test tools.

Checklist
  • Compile the software, logging the location and time spent fixing for all defects.

  • Run one or more linters (like clang-tidy) on the code, logging the location and time spent fixing for all defects.

  • Run one or more static analyzers (like scan-build and/or Frama-C) on the code, logging the location and time spent fixing for all defects.

  • Report all defects to all programmers in order for them to add applicable defects to their PSP data.

Test

Purpose

To test the software and validate that it meets its requirements.

General Instructions

If all phases before this one were done properly, this should not be a terribly expensive phase.

However, if the previous phases were not done properly, this phase will be extraordinarily expensive.

Entry Criteria

  • Completed, compilable software.

  • A fully fleshed-out design for the testing regimen.

Exit Criteria

  • Working, tested software.

  • Reference documentation.

Activities

Create Test Tools
Purpose

To create needed test tools.

General Instructions

Most of the time, this activity should not be necessary.

Checklist
  • For Each Non-Existent Test Tool:

    • Create the test tool with the Code phase checklists.

    • Compile the test tool with the Compile phase checklists.

    • Test the test tool with the Test phase checklists.

Test
Purpose

To create the test suite and test the software thoroughly.

General Instructions

The entire test suite should be able to be run without manual input.

Manually writing tests should be more or less mechanical, no thinking required, if the test regimen design was done well.

However, if the test regimen design was done well, most tests should be automatically generated.

Each programmer should be able to create the tests for their portion of the code, more or less. Some flexibility may be required.

If the testing regimen has been properly designed, then running dynamic analysis on all tests should be trivial.

If the testing regimen has been properly designed, then using the tests as a base for fuzzing should also be trivial.

Unit tests are for exercising postconditions of functions/procedures, as well as the invariants of data structures.

Integration tests are for exercising the proper use of callee’s and other API items.

System tests are for exercising complete system behavior.

Error tests are for ensuring that the software behaves correctly with bad or malicious inputs.

Performance tests are for ensuring that the software meets performance requirements.

Security tests are for ensuring that the software meets security requirements.

Safety tests are for ensuring that the software meets safety requirements.

Cryptography tests are for ensuring that the software does not covertly leak data about secrets.

Performance tests may be accomplished by measuring the performance of all other tests, but they may also require specific tests.

Checklist
  • Unit Tests:

    • Write all unit tests that cannot be generated.

    • Generate unit tests that can be generated (usually with property-based testing).

    • Run all unit tests, removing basic block assertions as necessary, fixing and logging defects as they are found, and saving all generated tests that found one or more defects.

  • Integration Tests:

    • Write all integration tests that cannot be generated.

    • Generate integration tests that can be generated (usually with property-based testing).

    • Run all integration tests, removing basic block assertions as necessary, fixing and logging defects as they are found, and saving all generated tests that found one or more defects.

  • System Tests:

    • Write all system tests that cannot be generated.

    • Generate system tests that can be generated (usually with property-based testing).

    • Run all system tests, removing basic block assertions as necessary, fixing and logging defects as they are found, and saving all generated tests that found one or more defects.

  • Error Tests:

    • Write all error tests that cannot be generated.

    • Generate error tests that can be generated (usually with property-based testing).

    • Run all error tests, removing basic block assertions as necessary, fixing and logging defects as they are found, and saving all generated tests that found one or more defects.

  • Performance Tests:

    • Write all tests for performance requirements that cannot be generated.

    • Generate all tests for performance requirements that can be generated (usually with property-based testing).

    • Run all tests for performance requirements, removing basic block assertions as necessary, fixing and logging defects as they are found, and saving all generated tests that found one or more defects.

  • Security Tests:

    • Write all tests for security requirements that cannot be generated.

    • Generate all tests for security requirements that can be generated (usually with property-based testing).

    • Run all tests for security requirements, removing basic block assertions as necessary, fixing and logging defects as they are found, and saving all generated tests that found one or more defects.

  • Safety Tests:

    • Write all tests for safety requirements that cannot be generated.

    • Generate all tests for safety requirements that can be generated (usually with property-based testing).

    • Run all tests for safety requirements, removing basic block assertions as necessary, fixing and logging defects as they are found, and saving all generated tests that found one or more defects.

  • Cryptography Tests (Where Applicable):

    • Write all tests for constant-time cryptography that cannot be generated.

    • Generate all tests for constant-time cryptography that can be generated (usually with property-based testing).

    • Run all tests for constant-time cryptography multiple times, logging the min, max, mean, median, Q1, Q3, and standard deviation of time.

    • For each test result, calculate the probability of whether the code is constant-time or not, using standard statistical formulas.

    • For each constant-time defect, fix and log the defect as is is found and save all generated tests that found the defect.

  • Coverage:

    • Verify that branch coverage is what was expected with the ratio of paths that were deemed not worth testing.

    • Verify that the line coverage is what was expected with the ratio of paths that were deemed not worth testing.

    • Design tests to cover missing branches and lines and log the test defects.

    • Create and run the tests to cover the missing branches and lines.

    • Run an automated mutation-driven testing tool to identify more coverage problems.

    • Add tests to cover the holes found by the mutation-driven testing tool and log the defects as Test defects.

    • Remove basic block assertions as necessary.

    • Fix and log the defects found by the new tests.

Additional Testing
Purpose

To use the existing test suite to test the software further.

General Instructions

This activity requires the test suite, whereas the previous activity created it.

Possible dynamic analyzers include (but are not limited to): AddressSanitizer, UndefinedBehaviorSanitizer, ThreadSanitizer, MemorySanitizer, Valgrind, and tis-interpreter.

Possible fuzzers include (but are not limited to): afl++, radamsa, and libFuzzer.

Checklist
  • Dynamic Analysis:

    • For each test, decide whether it is possible to run the test under the dynamic analyzers.

    • For each dynamic analyzer, run all possible tests.

  • Fuzzing:

    • For each test, decide whether it is possible to use it as input to fuzzers.

    • For each fuzzer, create starting test cases from the possible test cases.

    • For each fuzzer, run the fuzzer on the software.

Documentation

Purpose

To ensure the documentation is complete, useful, and captures as much knowledge about the system as possible from the heads of the programmers.

General Instructions

This phase assumes that most reference documentation is created while doing the other activities.

In other words, it assumes that reference documentation is created alongside code and tests.

Checking the completeness of the documentation, including documenting all assumptions, is immensely important to accomplish complete Knowledge Capture.

All documentation types (with some extras) should usually be created, but there may be times to skip some.

Introductions should include project health indicators (all green, [tests|passing] and such).

Introductions should include quick general description of the problem the project solves.

Introductions should include a simple code snippet showing how easy it is to use it (not the most complex way of using it).

Introductions should include screenshots and GIF’s if it’s UI-related (very important if it’s UI-related).

Introductions should include quick installation guide if using a common way (or a link to an in-depth guide if it’s not easy to install).

Introductions should include links to other parts (in-depth articles, etc.).

Entry Criteria

  • Working, tested software.

Exit Criteria

  • Completed documentation.

Activities

Documentation Tests
Purpose

To gather requirements for documentation and ensure those requirements will be met.

General Instructions

There should be requirements for documentation.

There should also be some way to verify that the requirements for the documentation are met.

This activity is to provide both of those.

Checklist
  • Documentation Requirements

    • Create the requirements for the reference documentation.

    • Create the requirements for the code samples.

    • Create the requirements for the explanations/discussions.

    • Create the requirements for the tutorials.

    • Create the requirements for the how-to guides.

    • Create the requirements for the introductions.

    • Create the requirements for the case studies.

  • Documentation Tests

    • Implement any possible tests for the requirements of the reference documentation.

    • Implement any possible tests for the requirements of the code samples.

    • Implement any possible tests for the requirements of the explanations/discussions.

    • Implement any possible tests for the requirements of the tutorials.

    • Implement any possible tests for the requirements of the how-to guides.

    • Implement any possible tests for the requirements of the introductions.

    • Implement any possible tests for the requirements of the case studies.

Reference Documentation
Purpose

To complete the reference documentation.

General Instructions

Because most reference documentation should already be done, this activity should not take long.

Checklist
  • Gather all documentation.

  • Sort the Non-Reference Documentation

    • Identify all existing documentation that is not reference documentation.

    • For each bit of non-reference documentation, figure out what kind of documentation it is.

    • For each bit of non-reference documentation, put it with that type of documentation.

  • Write a “Ground Truth” Document

    • Identify what the software actually does, including corner cases.

    • Document what the software actually does.

  • Update pre-existing reference documentation.

  • Reference Documentation Review

    • Review the “Ground Truth” document for correctness and completeness.

    • Review the rest of the reference documentation for correctness and completeness.

    • Ensure that the reference documentation passes the relevant tests.

Code Samples
Purpose

To update and create code samples.

General Instructions

Code samples are extremely important for quick understanding of the “how” of the software.

Checklist
  • Update Code Samples

    • Identify existing code samples.

    • For each existing code sample, update it or remove it, as applicable.

  • Write Code Samples

    • Identify common tasks that users would want to accomplish with the software.

    • For each common user task, write one code sample, if none exist.

  • Review Code Samples

    • Ensure that code samples cover all important and common user tasks.

    • Review all code samples for completeness and correctness.

    • Ensure that the code samples pass the relevant tests.

Explanations/Discussions
Purpose

To update and create explanations/discussions.

General Instructions

Explanations and discussions are understanding-oriented documentation.

These should be focused on helping users comprehend concepts necessary for understanding the reference documentation.

Checklist
  • Update Explanations/Discussions

    • Identify existing explanations and discussions.

    • For each existing explanation or discussion, update it or remove it, as applicable.

  • Write Explanations/Discussions

    • Identify concepts that users must comprehend in order to understand the reference documentation.

    • For each concept, write the requisite explanation and/or discussion.

  • Review Explanations/Discussions

    • Ensure that explanations and discussions cover all concepts necessary to understand the reference documentation.

    • Review all explanations and discussions for completeness and correctness.

    • Ensure that the explanations and discussions pass the relevant tests.

Tutorials
Purpose

To update and create tutorials.

General Instructions

Tutorials are learning-oriented documentation.

They are different from explanations and discussions.

The reason is that each clarifies information for other documentation.

Explanations/discussions are for clearing up the concepts for the reference documentation.

Tutorials are for diving deep into tasks to help users and to clarify the concepts explained in the explanations and discussions.

Tutorials should follow the “Show locked doors before you show a key” principle: show the problem first, then the solution.

Checklist
  • Update Tutorials

    • Identify existing tutorials.

    • For each existing tutorial, update it or remove it, as applicable.

  • Write Tutorials

    • Identify concepts that users must comprehend in order to understand the reference documentation.

    • For each concept, identify the explanation and/or discussion.

    • For each explanation or discussion, identify whether diving deep might help users.

    • For each needed deep dive, write the requisite tutorial.

  • Review Tutorials

    • Ensure that tutorials cover all concepts necessary to understand the reference documentation and explanations/discussions.

    • Review all tutorials for completeness and correctness.

    • Ensure that the tutorials pass the relevant tests.

How-to Guides
Purpose

To update and create how-to guides.

General Instructions

How-to guides are problem-oriented documentation.

They should be directly targeted to solving problems for users.

Checklist
  • Update How-to Guides

    • Identify existing how-to guides.

    • For each existing how-to guide, update it or remove it, as applicable.

  • Write How-to Guides

    • Identify common problems that users might be facing.

    • For each common problem, write a how-to guide showing how to solve it with the software.

  • Review How-to Guides

    • Ensure that how-to guides cover all common problems that users might face.

    • Review all how-to guides for completeness and correctness.

    • Ensure that the how-to guides pass the relevant tests.

Introductions
Purpose

To update and create introductions.

General Instructions

Introductions are why-oriented documentation.

They should be directly targeted to explaining the “why” behind the software.

Checklist
  • Update How-to Introductions

    • Identify existing introductions.

    • For each existing introduction, update it or remove it, as applicable.

  • Write Introductions

    • Identify new features and changes that need justification.

    • For each feature or change, write the requisite introduction.

  • Review Introductions

    • Ensure that introductions cover all new features or changes that need justification.

    • Review all introductions for completeness and correctness.

    • Ensure that the introductions pass the relevant tests.

Case Studies
Purpose

To update and create case studies.

General Instructions

Case studies are persuasion-oriented documentation.

They should be directly targeted to showing users what’s in it for them.

Checklist
  • Update Case Studies

    • Identify existing case studies.

    • For each existing case study, update it or remove it, as applicable.

  • Write Case Studies

    • Identify anything in the software that benefits users and is appropriate for a case study.

    • For each such thing, write the requisite case study.

  • Review Case Studies

    • Ensure that case studies cover at least the best benefits for users.

    • Review all case studies for completeness and correctness.

    • Ensure that the case studies pass the relevant tests.

Announcements
Purpose

To create announcements.

General Instructions

Announcements are for announcing changes.

Checklist
  • Add to the Changelog.

  • Write the Release Notes.

  • Write Announcements

    • Answer the “who”.

    • Answer the “what”.

    • Answer the “when”.

    • Answer the “where”.

    • Answer the “why”.

    • If useful, answer the “how”.

Documentation Clean Up
Purpose

To ensure the documentation is prepared for public release.

General Instructions

Proofreading is especially important because grammar or spelling mistakes reflects poorly on the software itself.

Checklist
  • Proofreading

    • Ensure that the documentation has no grammar, spelling, or other errors.

  • Formatting

    • Ensure that the formatting on the documentation is correct.

  • Final Testing

    • Ensure that the documentation still passes all relevant tests.

Postmortem

Purpose

To wrap up loose ends and gather data about the process.

General Instructions

This should be the easiest phase by far.

Entry Criteria

  • Working, tested software.

  • Reference documentation.

Exit Criteria

  • Complete data for PSP.

Activities

Postmortem
Purpose

To complete the PSP and TSP postmortem.

General Instructions

Both PSP and TSP postmortems should be completed.

Checklist
  • PSP Postmortem

    • Have every programmer complete a postmortem for themselves alone using the data they gathered from psp.py.

  • TSP Postmortem

    • “Gather the programmers.”,

    • “Complete a general postmortem for the subproject.”

Finish Subproject
Purpose

To mark the subproject as completed in psp.py tools and make the software available.

General Instructions

When this subproject was not completed without interruption, or interrupted another subproject, give best effort guesses to size measure questions that psp.py will ask.

It might be best if the subprojects are tracked in separate branches and then merged into master when they are finished.

Checklist
  • Merge (Where Applicable):

    • Merge the subproject into master or the appropriate branch.

  • Finish:

    • Ensure all defects are entered into psp.py.

    • Mark the subproject completed in psp.py.

  • Postmortem:

    • Have every programmer complete a postmortem for themselves alone using the data they gathered from psp.py.

    • Complete a general postmortem for the subproject.

Defect Standard

Documentation

Number: 10

General documentation problems.

Incomplete Documentation

Number: 11

Documentation is incomplete.

Misleading Documentation

Number: 12

Documentation is misleading or wrong.

Out-of-Date Documentation

Number: 13

Documentation is out-of-date.

Confusing Documentation

Number: 14

Documentation is confusing or unusable by its intended audience.

Bloated Documentation

Number: 15

Documentation is bloated; it has more material than needed.

Inaccessible Documentation

Number: 16

Documentation is inaccessible by its intended audience.

Syntax

Number: 20

General syntax problems.

Typos

Number: 21

Spelling, punctuation.

Instruction Formats

Number: 22

General format problems.

Begin-End

Number: 23

Did not properly delimit operation.

Distribution

Number: 30

General packaging and distribution defects.

Package

Number: 31

General package and installer problems.

Version Control

Number: 32

Invalid check-ins to version control, including checking in items that should not be checked in, or forgetting to check in items that should be checked in.

Licensing

Number: 33

License issues, such as failing to abide by licenses.

Assignment

Number: 40

General Assignment problems.

Naming

Number: 41

Declaration problems, duplicates.

Scope

Number: 42

Scope problems, such as shadowing, scope too large, etc.

Initialization

Number: 43

Initialization problems.

Destruction

Number: 44

Destruction (and closing) problems.

Range

Number: 45

Variable limits, array and collection ranges.

Interface

Number: 50

General interface problems.

Internal

Number: 51

Internal procedure calls and references used incorrectly.

I/O

Number: 52

File, display, printer, any communication done wrong.

User

Number: 53

External formats, content, and API’s unusable or difficult to use.

Checking

Number: 60

Errors, inadequate checks.

Error Message

Number: 61

Error message confusing or non-existent.

Input Validation

Number: 62

User or foreign input validation inadequate.

Format String

Number: 63

Misuse of format strings, or failing to properly validate format strings.

Data

Number: 70

Structure and content inadequate.

Structure

Number: 71

Structure problems (i.e., cause performance issues or something similar).

Content

Number: 72

Content problems, such as too much content or not enough.

Function

Number: 80

General logic problems.

Sequencing

Number: 81

Statements not properly sequenced.

Loops

Number: 82

Off-by-one, incrementing, other loop problems.

Recursion

Number: 83

Recursion problems, such as invalid base case, misuse of unwinding.

Conditionals

Number: 84

Invalid use of conditionals, such as bad conditions, invalid cases, etc.

Assertions

Number: 85

An assertion, precondition, postcondition, invariant, or variant was incomplete and/or incorrect.

Algorithms

Number: 86

Violation of algorithmic requirements.

Memory Safety

Number: 90

General memory safety problems.

Buffer Overrun

Number: 91

Overrun buffer bounds.

Pointer

Number: 92

Pointer and string problems, such as stale pointers (pointers not updating when they point into a data structure).

Use-After-Free

Number: 93

Use pointer to memory after freeing the memory.

Double Free

Number: 94

Free memory more than once.

Memory Leak

Number: 95

Failure to properly free memory.

Concurrency

Number: 100

General concurrency problems.

Data Race

Number: 101

Data race problems.

Race Condition

Number: 102

Race conditions that are not data races.

Thread Use

Number: 103

Improper use of threads (failing to properly start, join, or otherwise use).

Design

Number: 110

General design problems.

Functional Design

Number: 111

Functional design problems.

Algorithmic Design

Number: 112

Algorithmic design problems.

Performance Design

Number: 113

Performance design problems.

Security Design

Number: 114

Security design problems.

Safety Design

Number: 115

Safety design problems.

Test Viability

Number: 116

Problems with test viability.

Test

Number: 120

General test problems.

Insufficient Test

Number: 121

Test is insufficient to test the path, by either coverage or some other reason.

Wrong Test Logic

Number: 122

Test logic is wrong.

Flaky Test

Number: 123

Test is flaky (can return different results).

System

Number: 130

General system problems.

Time Performance

Number: 131

Violation of time performance requirements.

Space Performance

Number: 132

Violation of space performance requirements.

Security

Number: 133

Violation of security requirements.

Safety

Number: 134

Violation of safety requirements.

Environment

Number: 140

Problems with support systems.

Build Process

Number: 141

General build process defects.

Dependencies

Number: 142

Failure to check, resolve, or properly use dependencies.