A retrospective of a second-year group project.

Our brief was simple: create a system to manage the mentoring of new employees. On paper, it seemed easy enough: a standard user system, a clever algorithm to match mentees to mentors and crud for meetings. In reality, it was a disaster.

We had 8 weeks to develop the system. We spent the first two weeks analysing requirements and producing a system design. Happy with our design, we were looking forward to turning clear, well-defined requirements into our final system.

Except it didn’t go to plan.

Development ran smoothly for most of the project. No issues noticed, tests passing, everyone was happy. The front-end mocks looked great. We were driving down elysium highway on cruise control.

Then, in the closing weeks, everything went wrong at once. We discovered gaping holes in the backend. The final user interface was barely functional. There were too many issues to fix. Overwhelmed with bugs and out of time, we failed to produce an application which met our requirements.

In this retrospective I’m going to take a closer look at our software engineering practices and question our choice of technology.

Project Overview

The first couple of meetings were spent getting everyone up to speed on the project’s specification. From the specification, we decided a web app, split into an interface and an API, was the best solution. In our first meeting we each discussed the technologies we had experience with. The common theme was Java, C and HTML/CSS/JS. Java was taught in first-year so it was a natural choice to power the back-end. We used String Boot to create the API. Two developers had experience with React.js so they formed the front-end team.

Integration

“Integration refers to the software-development activity in which you combine separate software components into a single system” - Steve McConnell

From the start we opted for a phased integration strategy. And we took it to its logical extreme. This was our plan:

  1. Develop the entire front-end
  2. Develop the entire back-end
  3. Combine them
  4. Fix a small number of integration related bugs
  5. Be left with the finished system

Our project died somewhere between steps 3 and 4. During the integration step, we discovered many bugs. The interaction between the front-end and back-end revealed internal issues in both subsystems. Because our interfaces were not well defined, the bugs were interacting with each other and made debugging very hard. Developer morale hit rock-bottom. We had all put a lot of effort into our classes but we had made no progress at the system level. This poor morale, combined with the complex nature of the bugs, made debugging near-impossible.

Looking back to the start of the project, there were multiple paths we could have taken to prevent these errors. We could have continued with our plan of phased integration and improved our documentation, or we could have opted for incremental integration. Let’s take a look at both of these.

Better documentation

When we came to combine our front-end and back-end we found that the API documentation was incomplete. Whilst the front-end team did discuss API endpoints with the back-end team, our definitions for the interface was hazy at best.

Firstly, we did not document all the endpoints. We documented endpoints for our core business logic but supplementary features such as the user account system were not mentioned at all in the documentation. This led to the user system being overlooked during API development. This was brought to the spotlight when we were, comically, trying to integrate the front-end user account system with non-existent endpoints. However, once made aware of the issues, the API team worked very hard to create the missing endpoints. This process took a big toll on their morale. Prior to this, the back-end team believed they were finished and were proud of what they thought was a complete API. During integration, they were given a long to-do list. Understandably, this dropped their morale and resulted in less effective work.

I believe the following factors led to poor API documentation:

  • The API documentation was missing key details
  • Both subteams lost sight of the goal, a singular system
  • No one was assigned responsibility for the interface specification

API Documentation

A leading factor in our integration troubles was the sparse API documentation. The documentation listed endpoints and gave example input/output data. All the examples contained positive cases, there were no examples of the API’s error handling.

So, what should we have included in the documentation?

The State of API 2019 was a survey conducted by Smartbear, creator of Swagger. When they asked developers what they found most important in API documentation, the following were the top five responses:

  • Examples
  • Status and Errors
  • Authentication
  • Error Messages
  • HTTP Requests

From this list, we can see that our API documentation was missing important information. Our sole piece of documentation was examples. This was particularly damaging as it only listed valid examples. There was no discussion about how the system would respond to invalid requests, requests with missing data, or requests with incorrectly configured headers. Also missing were discussions about error handling. The authentication was hastily rolled out in the final days of the project and due to time pressure, was not documented. HTTP verbs were also missing from the documentation. I believe that if our documentation included all five points, we would have designed a more robust API as we would not have been able to overlook any important details.

Losing Sight of the Goal

If we pretend the project had two unrelated objectives: creating a user interface and standalone Java code for the business logic, the project was a success. As fun as this fantasy is, it shows a mistake made by both the front-end team and the back-end team. I believe the issues with documentation came from a dangerous reframing of the project’s objective. Because we only combined the interface and the API in the final stages of the project, I believe it was easy for the team to lose sight of the goal of creating one system. This meant that when writing the API, the back-end team did not consider that the endpoints could be accessed with incorrect data or with invalid requests. This was also shown by the front-end team, lots of time was spent designing an intuitive interface, but less time was spent considering how to manage application state or how to parameterise requests at the application level.

As with the previous point, producing more complete documentation about the interfaces between the two subsystems would have made sure that the purpose of the front-end and back-end was at the forefront of developers’ minds.

A matter of responsibility

There was a lack of accountability for interface specification throughout the project. No-one was assigned the role of specifying the interface until late in the project. At the start of the project, each team believed it was the responsibility of the other team to define the interfaces. To prevent such a situation, a team member could have been assigned the responsibility of building the interfaces. They would have acted as a single-source-of-truth throughout the project and would have helped prevent delays. They would have defined the endpoints and API behaviour to which both the front-end and the back-end would have adhered.

There was also a lack of integration testing. With a team member directly responsible for integration, they would have been in a good position to oversee the application-level testing strategy. Thorough integration tests would have helped debug the complex issues we experienced when integrating at the end of the project.

All these reasons give insight into why the documentation led to severe issues with the final product. Although it is possible that the team was always destined to meet these issues, I believe that our integration strategy meant these issues were hidden until the final stages of the project. In the next section I will explore how we could have encountered these issues at an early stage, when there was still time to pull the project back on track.

Incremental integration

As discussed earlier, we combined our front-end code with the back-end code in one step at the end of the project. I believe that changing our integration strategy to an incremental approach would have increased the survivability of the project.

With incremental integration, you first develop a small, functional part of the system, testing and debugging it. Once complete, you develop another small part of the system, testing, debugging and finally integrating. Once the new, combined system is working as expected, you continue to add, test, debug and integrate small parts of the system.

This results in a wide range of benefits. When adding one small section at a time, you reduce the possible sources of bugs. With phased integration, especially with how we did it in our project, when combining all the classes, errors could come from any of the classes being combined. With incremental integration, errors are restricted to being from either the old system, or the new class being added to it. Another benefit is the psychological boost from seeing the system grow. When you integrate by adding small sections, you are able to see your code working at the system level. The development team are able to see the system coming to life, rather than trusting that it will all work as we did in our project.

There are many ways to approach incremental integration. For this project, quickly developing the user account system would have given us a solid foundation. By focussing on one feature at the start we would have been able to test key assumptions on both the front-end and the back-end. For example, combining both the UI and the API in the early stages of the project would have highlighted the need for thorough documentation as well as the need for integration tests. This would also result in a boost for developer morale, the team would see their code working at the system level, rather than hoping it works at the end of the project.

Combining incremental integration with frequent system builds would have had a great impact on the quality of the codebase. Whilst the codebase did not have frequent daily changes, adopting an approach closer to continuous delivery would have increased both morale and productivity. This is a complex topic, big enough for its own blog post, so I’ll leave it here.

Conclusion

I believe the issues with this project become apparent if you view the objective of the project as creating an unrelated interface and API. We successfully created an intuitive interface, and we created a functional back-end. However, when we tried to combine them we faced an overwhelming amount of issues which we discovered too late to fix. By making the links between the two subsystems explicit, through complete documentation and frequent integration, I believe the team would have been able to deliver a complete, functional system.

Architectural Decisions

The following section is more speculative than the previous one. Here, I will explore the impact of our choice of technologies.

In the early stages of our project we unanimously chose to develop a web application. Then, we decided that creating an API with a web interface was the best option. However, I believe that this decision added unneeded complexity to our application.

From a technology point of view, we chose to use Java Spring Boot for the API and React.js for the web interface. These are both popular choices for this type of project. To develop the front-end and the back-end we divided the team in two. As we have seen in the previous section, having the two subteams increased the need for communication and required more documentation.

Why React.js

Let’s retrace the steps we took before choosing to use React.

In our requirements analysis, we set a very lenient threshold for responding to the user.

The system shall be intuitive and easy to use whilst remaining responsive to the user, responding within 10 seconds of an interaction.

– Our Requirements Analysis

The 10 second response time gave us lots of choice for our architecture, we could pick pretty much any web technology and get a sub-10 second response time.

We explained why we chose React.js in our Design Document, quoted below.

To make the app respond quickly to the user’s requests we have chosen to implement the front end of the website using the React.js framework…

– Our Design Document

We chose React because we believed it would allow us to develop a responsive interface. This is not wrong, React would have allowed us to do that. However, our only requirement was responding within 10 seconds. Therefore our reason for picking React was maybe not the strongest. React performs best when used to develop certain applications, let’s see if ours was one of them.

Writing in “Architecting Modern Web Applications with ASP.NET Core and Azure”, Steve Smith lists the following reasons for developing a Single-Page-Application:

  • Your application must expose a rich user interface with many features
  • Your team is familiar with JavaScript and/or TypeScript development
  • Your application must already expose an API for other (internal or public) clients

Let’s look at each one in depth.

Your application must expose a rich user interface with many features

I think this is one of the main reasons why React was not the best choice. The interface’s goal was to allow the user to login, submit standard text-based forms, and view relevant information. The design of our interface was very simple, it did not require any incremental updates or live reloading. This means that we were not able to take advantage of React’s ability to partially reload pages. We did not require any advanced JavaScript features either. The React code mainly dealt with rendering Bootstrap components.

Your team is familiar with JavaScript and/or TypeScript development

Only two team members had experience with modern JavaScript, by picking React, we reduced the team’s pool of front-end developers. Most team members had experience of traditional web technologies so if we were to pick a more basic front-end solution, we would have been able to increase the size of the front-end team. This would have really helped the team during the final stages of the project.

Your application must already expose an API

The decision to develop a Java API was driven by the decision to use React. As such, if we were to have developed something other than an SPA, we could have explored server-side alternatives, such as templating. This would have simplified the development process as no integration would have been required.

Looking at the above points, it’s easy to see that using modern JavaScript framework was overkill. We had no need for advanced user interfaces and as such I believe that our choice of technology was flawed.

Alternatives

We could have opted for a template-engine based approach. This would have allowed us to write static template files, either in HTML or in a DSL. Then, at runtime, the template engine would replace variables in the template with actual values. Using a templating language would have allowed more developers to be involved in the interface development process. The tighter coupling required by a template engine would have reduced system complexity and would have lessened the need for such a formal integration strategy. I do not believe that a template engine would have limited the functionality of the system. These points, combined with Spring’s native template support, make this a very attractive approach.

Another option could have been to use a slimmed-down JavaScript front-end, such as jQuery. This wouldn’t have necessarily reduced the complexity of the front-end development, but it would have enabled more developers to be involved with the process. Many teammates had basic JavaScript knowledge but had no experience of ES6+ or modern JavaScript frameworks, adopting jQuery would have enabled a larger front-end team.

Claiming that certain technologies would have delivered a better system is a speculative stretch too far for me, so I’m going to leave this discussion here. The main takeaway is that based off our requirements, using a front-end framework was not necessarily the best choice.

Conclusion

We’ve looked into three possible reasons for the project’s failure: quality of documentation, integration strategy, and architectural decisions. The analysis of documentation quality explained the issues we faced in integrating the subsystems. Our analysis of integration strategy revealed mistakes during the construction of the code. Whereas our glimpse into the architecture resulted in a less clear outcome.

I believe it was a combination of those three factors which killed the project, not one factor delivering the fatal blow.

I conclude that adopting an incremental integration from the start would have been the one difference which could have changed this project’s fate. This would have highlighted issues in our documentation as well as potential issues with our architecture from an early stage.

Closing Remarks

When writing this retrospective, I chose to focus on what I believed were the main issues with the project. This piece is not meant to be a comprehensive list of what went wrong, but rather deliver some suggestions to avoid such a situation in the future.

All happy families are alike; each unhappy family is unhappy in its own way

The project was not straightforward, but it was a fantastic learning opportunity. I extend my thanks to my teammates for taking part in this project with me, especially our project manager.

Bibliography

Code Complete 2nd Edition, Steve McConnell (esp. Chapter 29)

The State of API 2019 Report, Smartbear (https://smartbear.com/resources/ebooks/the-state-of-api-2019-report)

Architect Modern Web Applications with ASP.NET Core and Azure, Steve Smith (https://docs.microsoft.com/en-us/dotnet/architecture/modern-web-apps-azure/)