Being working in small team size (5-15 members including PM, BA and testing team), one of my biggest problem is that, in order to meet the delivery schedule, we often have to ‘break’ one or more core principles of microservices. While it works in the end, I can’t help but question where we did go astray from the microservice template, whether the changes are actually a correct solution in our own case and what can we do to improve the system design and implementation process for same set of requirements in the future.

What we have broken

There are too many definitions of microservice, but simply put, microservice architecture revolves around the concept of independent, modular components. Each component is designed for a specific goal, and has its own business logic and database. Development, testing, deployment and scaling are isolated for each component service. Component services can communicate with each other via well-defined APIs.

The main goal of microservice design is not reducing complexity, but to separate original business requirements into smaller subset which can function and be developed, deployed and scaled independently. However, throughout a few projects that I’ve participated in, we often end up with a design that violates the principles of microservice.

1. Shared Database

In some of our design, we use the same database for all services. Some might argue that this is a valid design pattern for microservice, as we can have a single schema for the whole system, maintaining normalization of the database design and enforce data consistency. Also, in our database schema, we often design tables and table dependencies to be bounded within business logic of each module, so to a certain degree, it follows modularility of microservice.

However, this comes at a number of costs which haunt us at the latter phases of the project:

• Crossing the Module’s Contexts: with a single schema, sometimes the module is implemented to access tables and data which are irrelevant to its original defined contexts. This poses a serious issue of data security.

• Development Coupling: to add a new feature to a module, in an orthodox microservice, a new table or columns is introduced to the module’s database schema alone, hence unaffecting any other part of the system. In a single database design, however, these new features would possibly be inserted into some shared tables, which adds more work for the rest of the team to coordinate to update their own parts.

• Runtime Coupling: a single database means a single coupling point for all modules. A module hence can block all others from making transactions by locking the database, and this cannot be solved by scaling alone.

• Database Scaling: this design puts a strain of scaling the database. Each module has its own set of scaling requirements (storage, speed, etc.) and satisfying all by just vertical scaling the database is not optimal. Not only that, the requirements of each module might change overtime in production environment, making scaling the database more troublesome.

2. Business Context’s Overlap

This is perhaps a consequence of single database design: as there is only one source of data, retrieving and updating data bounded to another module might not need to be done via APIs anymore - the module can just access and modify by its own! This type of implementation is quite severe to the data integrity:

• While every database transaction is ACID, it is now unable to tell which module is responsible for a data update. As a result, extra works need to be done to determine data ownership.
• Crossover of data ownership means team cannot function independently and have to spend time communicating to resolve the overlap in design.

3. Chatty design

In some projects, the modules are defined quite small and loose-coupling that, in order to complete some requests, a module has to rely on asynchronous communication to collect all the necessary data. Personally I think it is fine for high-level modules which directly interact with users like frontend, but in this case, almost all modules implement some inter-communication logic, and worst, all are asynchronous.

People call this type of design ‘chatty’ for a reason: it’s complicated to navigate around to figure out full implementation of a request. Also, the main reason asynchronous communication is not recommended is that it destroys the loose-coupling aspect of microservice - if one module fails for any reason, this would create a chain of failure across dependent modules and possibly the whole system. The main benefit of microservice - modularity, hence, no longer retains.

Comes Conway’s law

‘Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.’

Melvin E. Conway

This, perhaps, is the most important theory of software development. The observation is that depending how the organization (team) is divined, there comes a tree of inter-communication between subteams and as a result, the system designed by the team would resemble the communication structure. This actually implies that, by studying the system design, we can possibly trace back and discover the strucure of the team designing and developing it.

I believe Conway’s Law explain a lot about the deviation of our project’s system design compared to a ‘correct’ microservice. Following are some of my observation about our team structure:

• As the team size is small, there are actually not many subteams - it often consists of a PM/BA team, a testing team, a DevOps team, a frontend team and 1-3 backend teams. There might be 2-4 developers at the top of the chain to design the system.

• As the project follows Agile, we focus on 1 feature each sprint and resolve pending issues of last sprint. As a result, after breaking down new feature into tasks, development team often function as 1 frontend team and 1 backend team.

Now let’s compare to the characteristics of microservice’s design and development:

1. Development of each module can run separately

Unshockingly, our team doesn’t need this, as the whole team develop one feature at a time, hence every module are done by one team only.

2. Each team, hence, owns domain knowledge of their own module

More often than not, everyone in the team knows everything about the system, not just to design level, but to implementation level.

3. Modules in microservice must communicate via APIs

This actually creates overhead on top of our development, due to everyone being in a team. If we follow this rule completely, we would need to wait for someone else to update the APIs whenever external modules require new data. Instead, due to moving as a single team, any member can touch the implementation of the other module and update himself/herself.

4. Each module has its own data storage

Again, this adds unneccessary complexity to the development, mainly in local setup. In a large team, the only one taking care of integration part would need to setup everything locally, but in a small team, database per service would require all individuals to not only setup all modules correctly, but also setup solutions to ensure data consistency across multiple databases. The databases might or might not be local, but again, it costs too much for a small team to do so.

5. Communication between modules should be synchronous, not asynchronous

The main issue, in my opinion, is asynchronous communication makes sense in a local development environment. In a single machine, running all services and enforce everything to wait for each other is actually easier to do and easier to understand. As each member of the team can and has to be able to run every module locally, it is understandable that there is no need to think about unreliable connection between modules. However, this does not translate well to production deployment when availability and latency are crucial aspects of the system.

What we can do better as a small team

There is a few things that I would want to do in order to success as a small team:

1. Acknowledge ’true’ microservice might not be our solution

Decoupling business contexts and team communication, in my opinion, is the true main benefit of microservices. As communication between team becomes tree-like instead of graph-like, the leap (subteam) only concerns about their own business contexts and internal communication, leaving integration topics to the top of the communication tree. The API requirements of microservice, hence, is the result of a proper breakdown of communication between sub-teams.

In a small team, however, it is impossible to fully untangle the communication flow. As long as all team member moving together to develop a single feature at a time, everyone bears responsiblity to communicate well with each other. We have to acknowledge that, and remove as much of overhead in design as possible to make the communication go smoothly, even if it breaks some principles of microservices.

2. Retain modularity aspect of microservice

When moving from feature to feature, it is important that the development of the new feature should not modify anything of the old ones, including database schema, code, business logic, deployment logic, and APIs. Any request to change old layer should be a critical issue and need to be reviewed by team leader and acknowledged by the whole team.

3. Enforce data isolation to a certain degree

The best scenario is that each module owns it database, and access to other databases must be done via APIs.

In a shared database design, it should be better if the irrelevant modules can only access the tables via a materialized view for read queries, and call APIs for write queries.

4. Scaling shared database using replica

As some modules’ read demand grows extensively, it is not financial-wise to scale the shared database vertically. We can solve this by creating read-only replicas of the main database and allow read-intensive modules to access these replica. This can keep the original design while allow scaling on production deployment.

5. Avoid chatty microservice design

There are a few solutions to avoid a chatty design, while has low setup cost for local development

• Orchestra: instead of letting all modules chatting around, implement a few or a singular module to organize order of events.
• Choreography: Dapr Pub/sub using redis in local, and reliable event bus on production (Azure Service Bus, AWS SQS)

References

• Conway’s Law
• Don’t Build a Distributed Monolith - Jonathan “J.” Tower - NDC London 2023