Documented Knowledge
In software engineering it is widely accepted that team and organizational knowledge is one of the most valuable assets of the organization. Documenting this knowledge in a way which makes it scalable and accessible to other teams and the whole organization is of high importance.
As most of us have experienced during our careers, there are constant changes in team structures and in project ownership. When new members join existing teams and existing members move from one team to another, high quality internal documentation allows effective onboarding, overview, and deep understanding of the systems and the processes for which the team is responsible.
When there is no documentation available, a wide variety of issues begin to arise. In context of software systems, without documentation it is challenging to fully understand the behavior of functionality which is developed and maintained by a team. Also it might not be clear what certain sections of code are used for and what business rules they implement. When dealing with distributed systems, without proper documentation it can be extremely difficult to understand the full scope of a system, its components, and the external systems it interacts with.
Lack of documentation also makes it challenging to focus team efforts when implementing new functionality. Many of us have experienced ambiguity in requirements and expected functionality when proper design documentation does not exist. This can result in different team members having different interpretations of intended behavior which leads to wasted development resources and eventually rework. Ultimately, poorly documented knowledge in software engineering leads to wasted resources and bugs in the system.
Although poor documentation is one of the most common complaints of software engineers, and almost everyone recognizes that more effort needs to be done on the matter, many teams find themselves in this situation. There are a few reasons which can explain this gap. One reason is that creating documentation takes effort and time which could be spent by the development team on coding. Another reason is that the value of documentation begins downstream and is not experienced immediately by the author which makes it discouraging to create. So how can we encourage teams to invest more effort into preserving their knowledge by creating and updating documentation?
Ways to Create Internal Documentation
Team culture is a prerequisite to starting this process. Once the team and its leadership have decided that documentation is important and it needs to be brought to an acceptable standard, we can start exploring various ways to create it.
One way to create more documentation is to temporarily stop development efforts of the team and write documentation instead of code. Documentation writing tasks can be periodically prioritized over code each time gaps in documentation are observed.
While this might result in an initial boost, there are a few issues with this approach. Documentation created this way captures scattered knowledge without any design considerations which were originally discussed and additional valuable information which has been forgotten. Furthermore, after a while the team defaults to its original operation mode where no documentation is produced until additional gaps in documentation are observed. Any internal documentation created this way is going to be partial at best, and will get out of sync as new functionality is added and existing functionality is changed.
Another way to create documentation is to treat it as an integral part of the development process, just like code. In this approach, documentation is treated just like any other artifact which is created or updated by the development team during each iteration. Documentation is created and updated along with code, tests, UI design, etc. It is taken into account when estimating effort required for feature completion and is part of "definition of done" criteria.
Furthermore, definition and integration of documentation related tasks in the development process ensures the team can sustain a constant predictable pace over periods of time.
Iterative and Incremental
Agile teams build products piece by piece, in iterative and incremental manner. Although they might differ in size, usually each feature represents a new functionality which is delivered to and can be experienced by the end users.
By closely examining the actual tasks performed by a team for each new feature, it becomes apparent that they align with the natural phases of the software development lifecycle: initiation and planning, design, development, testing, delivery and maintenance. During the naturally occurring initiation and design phases for each new feature, team members and stakeholders exchange information and make design decisions which affect the final behavior of the feature and the system as a whole. And although the information which is communicated during those phases is highly important, it is not always properly recorded.
The key is investing upfront effort in initiation, planning and design phases of each feature. By properly documenting information exchanged in those phases, we achieve two valuable goals: we create better designs and iteratively document the vast majority of team knowledge.
The Many Purposes of Feature Design Documentation
We need to stop looking at design documentation as something which serves a single purpose. High quality design documentation which is actively maintained, becomes much more than just a one time guide for developers during new functionality implementation.
Feature design documentation can be used for:
- New team member onboarding
- Clearing any ambiguity in user requirements
- A clear description of the functionality to be developed, aligned with team and customers
- Better design: writing things down makes us think clearer
- A record of design decisions and considerations
- Focus for team effort during the development phase
- A guide for QA engineers as to what is desired vs actual functionality
- A way for future maintainers to quickly understand the intended logic of a feature when fixing bugs
- Support for operations teams
High quality design documentation becomes a valuable asset which benefits all current and future team members. To better understand how to create this asset in a structured way as part of the development process, let's focus on the practical aspects.
Who Creates and Maintains the Documentation?
Structures of development teams vary greatly from one organization to another. Some teams do not have a dedicated product manager, system analyst or a software architect, roles which can help with writing the documentation. Software engineers on the team have the technical knowledge and the best understanding of the inner workings of the system and in most cases are the ones who need to write and maintain design documentation. By closely collaborating with the business people related to the project, software engineers are able to access any necessary information which is required for the documentation.
The task of creating design and design documentation might seem out of scope of the responsibilities for some if not most software engineers. However by "shifting left" the developer role, developers have the opportunity to exit the comfort zone, learn new skills, grow and create impact.
Which Template to Use?
There are several ways to document the various aspects and requirements of a new functionality. Some teams get by with just writing down user requirements in the description field in Jira (or other planning software), while others implement a full blown documentation process. There are various documents which can be used by teams as part of the development flow. Documents like SRS (Software Requirements Specification) document, PRD (Product Requirements Document), BRD (Business Requirements Document), FRD (Functional Requirements Document), SDD (Software Design Document), SAD (Software Architecture Document) and more.
When dealing with features as building blocks of the final product, one practical approach is to create and maintain a separate dedicated design document per feature. This way the design document can be easily managed as part of feature implementation and referred to when questions about a specific functionality arise. While this approach might not eliminate the need for a high level design document for the system as a whole, in most of the cases it will suffice.
Using templates makes the process of creating documentation easier and more streamlined. In case such template is not available to the team, the team should make it a priority to create one. In the next article I will share a design document template which can be used as part of the process, but for now we focus on the process itself.
To make the workflow of creating and maintaining documentation as familiar as possible for software engineers, we can make use of tools and ecosystem which we as developers already know: Git and Markdown files.
When and What to Document?
A clear structured and repeatable development process has many benefits also for teams which adopt the agile mindset. Having a defined structure allows us to introduce tasks related to documentation at specific locations along the flow. Once introduced, tasks related to documentation become integral part of the feature development life cycle. With proper tasks in place, documentation is created just like code: it is created when none exists, updated whenever changes are required in existing functionality and reviewed just like code at the end of relevant phases.
To better understand what documentation related tasks are performed at each of the phases of the feature development flow (when) and what information is recorded in each of the steps, let's follow Adi, a software engineer who is tasked with leading development of a new small feature.
Initiation and Planning
Adi, a software engineer in product team, is tasked with leading the development of a new feature. Adi creates a feature specification document from an existing template and uses it to capture any information related to the feature.
During feature initiation phase, the business requirements are collected from users and stakeholders to ensure development meets business, customer and market needs. By working closely with management, customers and business stakeholders who request the feature to be developed, Adi has access to all relevant information about the new feature. Because a verbal record is not enough to clearly define, explain and negotiate the scope of the new functionality, Adi uses the feature design documentation to capture important information which is discussed during meetings.
After analyzing user requirements, Adi has a better sense of what is actually requested by the customers, and is able to formulate and write down clear benefit hypothesis for the feature. Adi uses existing design documents to understand existing system behavior and functionality, in order to better assess how new functionality can be developed and integrated in the system. By analyzing the requirements Adi and the team have a better idea about the scope of the feature and any team external dependencies. Clearly understanding feature scope allows Adi and the team to plan feature delivery.
Design
Before development, Adi invests effort in design to ensure that the feature implementation provides optimal value for the system users. During design phase Adi conducts research to uncover any missing information and then proceeds to create functional and technical specifications. Adi uses the feature design document template and fills in the relevant information.
To shift to a more customer centric mindset, Adi adopts Amazon's "Working Backwards" principle, and drafts a fictional press release for the upcoming feature. The press release is documented in a dedicated section of the feature design document.
Adi separates design into two distinct phases: functional and technical and focuses separately on each of them. While the functional specification focuses on the "what", the technical specification covers the "how" aspects of the development.
The functional section of the design documentation is used by Adi to document the various use cases which are part of the feature. Each use case is equivalent to a user story. Adi makes use of UML use case and sequence diagrams to better capture the desired system behavior. Using diagrams allows focused communication during design meetings with other team members who are involved in feature development. Adi closely collaborates with the person responsible for UI/UX and together they create user flows, wireframes and UI design for the new functionality. Links and screenshots of the UI design are added to the feature design document for easy reference. At this stage it is already possible to preliminary validate the proposed solution and user flows with various stakeholders.
During technical specification Adi creates and documents design for API, DB schema and any other technical aspects of the feature. Adi documents technical decisions, considerations, tradeoffs and alternatives for technical solutions including pros and cons.
At the end of the design phase, Adi schedules a design review meeting with any relevant team members and stakeholders for the purpose of receiving feedback, finding gaps in design and getting the final approval before development begins.
Development
Adi and fellow members of the team use the approved design documentation to develop the feature or parts of the feature which are in scope for the current iteration. It is natural that during development new information is uncovered and the design documentation needs to be updated. Adi and the team continuously refer to existing design documents to better understand existing system functionality.
In parallel to development, quality assurance engineers can start working on creating test scenarios based on the functional specification for the feature and develop automated tests by using artifacts from technical specification like API specifications.
If necessary, user documentation can be created for the upcoming feature based on design documentation. By reusing specific parts of the design documentation it takes less effort to create external documentation than creating it from scratch.
Testing
During testing phase, the testing scenarios which are written based on the functional specification are used as a measure of the desired vs actual system behavior. Gaps between feature specification and actual system behavior are bugs which are opened during the testing phase.
During testing phase the design documentation can be reviewed by QA, and just like for code, bugs can be opened for the documentation.
Delivery
Adi and the team deliver the feature or planned MVP to production together with the relevant user documentation.
Maintenance and Operations
Feature design documentation can be provided to any personnel responsible for monitoring the system in production and for maintaining the system in the future. Moreover, the design documentation provides information about who to contact to better understand various system behavior by locating the author or the maintainer of the document. As Winston Royce once said: "Without good documentation, the software must be operated by those who built it". To allow other people to maintain and operate our software, we need to supply them with good documentation.
Conclusions
- Documentation has immense scalable benefits for software development teams and the organization as a whole.
- The benefits of documentation are experienced downstream and not immediately by the author at the moment of creation.
- The key to creating documentation is to incorporate the necessary steps into existing software development life cycle and make it an integral part of feature delivery flow.
- Initiation and design phases are critical phases in the feature delivery flow where important information is verbally exchanged. Documenting this information in a structured way allows teams to capture the vast majority of team knowledge.
- Documentation is treated like code: it is created when it does not exist, modified when there is a change in system functionality and reviewed before proceeding.
- Software engineers are the ones who are responsible for creating the documentation in most of the cases. By "Shifting Left", software engineers collaborate with relevant stakeholders to access information required for feature design and development.
- There is a need for culture, process and tools to make it all happen. Culture means being clear about how things are done and what is important for the team and the organization. Process means defining a development flow with emphasis on design before development and integrating steps related to documentation into the process. Tools in the context of documentation means creating a template which can guide developers in the process of creating documentation and using infrastructure with which developers are already familiar, in this case Git and Markdown files.
Towards More Professional Teams
Your team has decided to invest in structured development process, design and documentation as an integral part of the software development life cycle. It was not easy at the beginning but now you and other members of your team are able to step out of the comfort zone, grow and take on new responsibilities.
When new members join the team, their onboarding is easy: they have documentation which describes each major functionality of the software that the team is responsible for. There is much less ambiguity and miscommunication about new functionality which is being developed in the team. Each new functionality is developed with more attention to actual customer needs and team members are focused on providing value for product users instead of using the latest and greatest technology just for the sake of it.
By investing more time upfront, the quality of the software created by the team rises: there are fewer bugs, and each new bug can be quickly fixed as the design documentation provides a clear expected behavior of the system.
There are no single points of failure with regards to knowledge in the team. If one of the more experienced team members leaves the team, knowledge loss is minimized.
The development team actually becomes and also appears more professional to the external observer: we all naturally assume that high quality documentation also means high quality architecture and code.