Passionate Development

AWS: Modern data architecture for Microservices

I have been thinking about microservices more and more recently.

I have to admit when I heard microservice concept few years back, I was sceptic.

Aaron Patterson sums it best for me back then.

TBH, the main reason I like microservices is that I feel like my method calls are too fast and I’d prefer to throw in some latency.

:))) such a troll.

But time has changed, as I work on bigger projects and organisations, I cannot continue to be ignorant about it.

In fact, the team that I work with at the moment is in the process of splitting our monolithic Rails app into several microservices.

Although I wish that we have had some planning and the necessary upskilling before diving in (see ‘It’s not easy’ point below).

So we have now started reading up and watching some presentations on microservices. So far we have watched 2 presentations:

Here are some of the key points that I got from those presentations, as well as my personal commentary from my short experience with microservices so far.

It’s not easy

Blair from AWS said that microservices is basically distributed computing.

And distributed computing is hard.

You need to consider whether you have the skillsets for managing, program for and debug a distributed computing environment.

You also need to weigh in the skillsets requirements with the benefits from microservices.

If you don’t think you have the skillsets perhaps don’t do it straight away, but instead plan on getting skilled in it. Again only if it makes sense for the business.

As mentioned previoulsy, I think we missed the planning bit and as a result we had few hiccups with one of our new microservice.

No committment to a particular tech stack

You can use multiple tech stacks for each microservices and since they talk to each other via APIs, that won’t matter.

As a developer, that’s a good thing as it will give you opportunity to learn the new shiny such as Elixir, Golang or Crystal.

However Matt from Uber highlight the downside of this, for example:

  • It may create tribalism in the organisation, e.g camp NodeJS vs camp Golang
  • It is harder to move between microservices teams as opposed to one stack organisation

It’s all about trade off of course.

You build it, you run it

Another benefit of microservice is it fast to develop and rapid to deploy, in fact these two can be done in parallel.

You can achieve this because your microservice team has the full control and responsibilty the environment, from the conception, development to deployment.

Blair likes Jeff Bezos’ two-pizza team rule for a microservice team. What this means is, your microservice team need to have enough expertise within the team to deliver the microservice.

This also implies, that within the team - there has to be a capability to dev and to ops, instead of the tradional setup where the devs hand over the project to ops team for deployment and support.

Essentially, you build it, you run it.

Other tips

  • Use poly persistence - one database per microservice is a recommended pattern.
  • Eventual consistency is a challenge when data is passed around from one microservice to another. Use correlation id and propagate this.
  • Implement and expose rollback method for every microservices.
  • Use Transaction Manager that can tell microservices when an error occured and that will trigger rollback for every microservices.
  • Use Master Data Management to keep data consistent across microservices. This can be implemented as an AWS Lambda combined with Cloudwatch.
  • Which database to use? Let the functional requirements drive the choice, for example: if you need geospatial functionality choose Postgres.
  • Whenever you design a distributed system always design for failures which can be network failures, buggy codes etc.
  • Think ahead and plan your analytics requirements as part of the overall architecture.

Two pizza teams -> service teams - dev and ops not separated - devs should do the ops (can’t really remember the rest of the point) “You build it, you run it” (kinda what we sort of achieved with newsletter microservice - minimal OH involvement?)

These are some more presentations on my wish list: