Microservices

This article outlines the main challenges of state propagation across different (micro)services together with architectural patterns for facing those challenges, providing an actual implementation using the Debezium Engine. But first of all, let's define some of the basic concepts that will be used during the article. (Micro)service: I personally do not like this term. I prefer talking about bounded contexts and their internal domain/sub-domains. Running the bounded context and/or its domains as different runtimes (aka (micro)services) is a deployment decision, meaning that domain and sub-domains within a bounded context must be decoupled independently of how they are deployed (microservices vs modular monolith). For the sake of simplicity, in this article, I will use the word "microservice" to represent isolated runtimes that need to react to changes in the state maintained by other microservices. State: Within a microservice, its bounded context, domain, and subdomains define an internal state in the form of aggregates (entities and value objects). In this article, changes in the microservice's state shall be propagated to another runtime. State propagation: In a scenario where the application runtimes (i.e., microservices) are decoupled, independently of how they are deployed (Pod in Kubernetes, Docker container, OS service/process), state propagation represents the mechanism for ensuring that state mutation happened in a bounded context is communicated to the downstream bounded contexts. Setting the Stage As a playground for this article, we will use two bounded contexts defined as part of an e-learning platform: Course Management: Managing courses, including operations for course authoring Rating System: Enable platform users to rate courses Between both contexts, there is an asynchronous, event-driven, relationship, so that whenever a course is created in the Course Management bounded context, an event is published and eventually received by the Rating System, which adapts the inbound event to its internal domain model using an anti-corruption layer (ACL) and creates an empty rating for that Course. The next image outlines this (simplified) context mapping: Bounded Context Mapping of the two services The Challenge The scenario is apparently pretty simple: we can just publish a message (CourseCreated) to a message broker (e.g., RabbitMQ) whenever a Course is created, and by having the Rating System subscribed to that event, it will eventually be received and processed by the downstream service. However, it is not so simple, and we have several "what-if" situations, like: What if the Course creation transaction is eventually rolled back (e.g., the database does not accept the operation) but the message is correctly published to the message broker? What if the Course is created and persisted, but the message broker does not accept the message (or it is not available)? In a nutshell, the main challenge here is how to ensure, in a distributed solution, that both the domain state mutation and the corresponding event publication happen in a consistent and atomic operation so that both or none should happen. There are certainly solutions that can be implemented and message producer or message consumer sides to try to solve these situations, like retries, publishing compensation events, and manually reverting the write operation in the database. However, most of them require the software engineers to have too many scenarios in mind, which is error-prone and reduces codebase maintainability. Another alternative is implementing a 2-phase-commit solution at the infrastructure level, making the deployment and operations of the underlying infrastructure more complex, and most likely, forcing the adoption of expensive commercial solutions. During the rest of the article, we will focus on a solution based on the combination of two important patterns in distributed systems: Transactional Outbox and Change Data Capture, providing a reference implementation that will allow software engineers to focus on what really matters: providing domain value. Applicable Architecture Patterns As described above, we need to ensure that state mutation and the publication of a domain event are atomic operations. This can be achieved by the combination of two patterns which are nicely explained by Chris Richardson in his must-read Microservices Patterns book; therefore, I will not explain them in detail here. Transactional Outbox Pattern Transactional Outbox focuses on persisting both state mutation and corresponding event(s) in an atomic database operation. In our case, we will leverage the ACID capabilities of a relational database, with one transaction that includes two write operations, one in the domain-specific table, and another that persists the events in an outbox (supporting) table. This ensures that we will achieve a consistent state of both domains and the corresponding events. This is shown in the next figure: Transactional Outbox Pattern Change Data Capture With Transaction Log Tailing Once the events are available in the Outbox table, we need a mechanism to detect new events stored in the Outbox (Change Data Capture or CDC) and publish them to external consumers (Transaction Log Tailing Message Relay). The Message Relay is responsible for detecting (i.e., CDC) new events available in the outbox table and publishing those events for external consumers via message broker (e.g., RabbitMQ, SNS + SQS) or event stream (e.g., Kafka, Kinesis). There are different Change Data Capture (CDC) techniques for the Message Relay to detect new events available in the outbox. In this article, we will use the Log Scanners approach, named Transaction Log Tailing by Chris Richardson, where the Message Relay tails the database transaction log to detect the new messages that have been appended to the Outbox table. I personally prefer this approach since it reduces the amount of manual work, but might not be available for all databases. The next image illustrates how the Message Relay integrates with the Transactional Outbox: Transactional Outbox in combination with Transaction Log Tailing CDC One of the main goals of this solution is to ensure that the software engineers of the two bounded contexts only need to focus on the elements with orange color in the diagram above; the grey components are just infrastructure elements that shall be transparent for the developers. So, how do we implement the Transaction Log Tailing Message Relay? Debezium Debezium is a Log Scanner type change data capture solution that provides connectors for several databases, creating a stream of messages out of the changes detected in the database's transaction log. Debezium comes in two flavors: Debezium Server: This is a full-feature version of Debezium which leverages Apache Kafka and Kafka Connectors to stream data from the source database to the target system. Debezium Embedded/Engine: The simplified version can be embedded as a library in your product; it does not require an Apache Kafka service but still makes use of Kafka Connectors to detect changes in the data sources. In this example, we will use Debezium Embedded, due to its simplicity (i.e., no Kafka instance is needed) but at the same time, it is robust enough to provide a suitable solution. The first time a Debezium instance starts to track a database, it takes a snapshot of the current data to be used as a basis, once completed, only the delta of changes from the latest stored offset will be processed. Debezium is highly configurable, making it possible to shape its behavior to meet different expectations, allowing, for instance, to define: The database operations to be tracked (updates, inserts, deletions, schema modifications) The database tables to be tracked Offset backing store solution (in-memory, file-based, Kafka-based) Offset flush internal Some of these properties will be analyzed later in the article. All Pieces Together The next image shows the overall solution from the deployment perspective: Services: In this example, we will use Spring Boot for building the Course Management and Rating System bounded contexts. Each bounded context will be deployed as separate runtimes. The persistence solutions for both services will be PostgreSQL having each of the services a dedicated schema. Persistence: PostgreSQL is also deployed as a Docker container. Message Broker: For the message broker, we will use RabbitMQ, which is also running as a Docker container. Message Relay: Leveraging Spring Boot and Debezium Embedded provides the Change Data Capture (CDC) solution for detecting new records added in the outbox table of the Course Management service. Show Me the Code All the code described in this article can be found in my personal GitHub repository. Overall Project Structure The code provided to implement this example is structured as a multi-module Java Maven project, leveraging Spring Boot and following a hexagonal architecture-like structure. There are three main package groups: 1. Toolkit Supporting Context Modules providing shared components and infrastructure-related elements used by the functional bounded contexts (in this example Course Management and Rating System). For instance, the transactional outbox and the Debezium-based change data capture are shared concerns, and therefore their code belongs to these modules. Where: toolkit-core: Core classes and interfaces, to be used by the functional contexts toolkit-outbox-jpa-postgres: Implementation of the transactional outbox using JPA for Postgres toolkit-cdc-debezium-postgres: Implementation of the message relay as CDC with Debezium embedded for Postgres toolkit-message-publisher-rabbitmq: Message publishers implementation for RabbitMQ toolkit-tx-spring: Provides programmatic transaction management with Spring toolkit-state-propagation-debezium-runtime: Runtime (service) responsible for the CDC and message publication to the RabbitMQ instance 2. Course Management Bounded Context These modules conform to the Course Management bounded context. The module adheres to hexagonal architecture principles, similar to the structure already used in my previous article about repository testing. Where: course-management-domain: Module with the Course Management domain definition, including entities, value objects, domain events, ports, etc; this module has no dependencies with frameworks, being as pure Java as possible. course-management-application: Following hexagonal architecture, this module orchestrates invocations to the domain model using commands and command handlers. course-management-repository-test: Contains the repository test definitions, with no dependencies to frameworks, only verifies the expectation of the repository ports defined in the course-management-domain module course-management-repository-jpa: JPA implementation of the repository interface CourseDefinitionRepository defined in the course-management-domain module; Leverages Spring Boot JPA course-management-repository-jpa-postgres: Specialization of the repository JPA implementation of the previous module, adding postgres specific concerns (e.g., Postgres database migration scripts) course-management-rest: Inbound web-based adapter exposing HTTP endpoints for course creation course-management-runtime: Runtime (service) of the Course Management context 3. Rating System Bounded Context For the sake of simplicity, this bounded context is partially implemented with only an inbound AMPQ-based adapter for receiving the messages created by the Course Management service when a new course is created and published by the CDC service (toolkit-state-propagation-debezium-runtime). Where: rating-system-amqp-listener: AMQP listener leveraging Spring Boot AMQP, subscribes to messages of the Course Management context rating-system-domain: No domain has been defined for the Rating System context. rating-system-application: No application layer has been defined for the Rating System context. rating-system-runtime: Runtime (service) of the Rating System context, starting the AMQP listener defined in rating-system-amqp-listener Request Flow This section outlines the flow of a request for creating a course, starting when the user requests the creation of a course to the backend API and finalizing with the (consistent) publication of the corresponding event in the message broker. This flow has been split into three phases. Phase 1: State Mutation and Domain Events Creation This phase starts with the request for creating a new Course Definition. The HTTP POST request is mapped to a domain command and processed by its corresponding command handler defined in course-management-application. Command handlers are automatically injected into the provided CommandBus implementation; in this example, the CommandBusInProcess defined in the toolkit-core module: Java import io.twba.tk.command.CommandHandler; import jakarta.inject.Inject; import jakarta.inject.Named; @Named public class CreateCourseDefinitionCommandHandler implements CommandHandler<CreateCourseDefinitionCommand> { private final CourseDefinitionRepository courseDefinitionRepository; @Inject public CreateCourseDefinitionCommandHandler(CourseDefinitionRepository courseDefinitionRepository) { this.courseDefinitionRepository = courseDefinitionRepository; } @Override public void handle(CreateCourseDefinitionCommand command) { if(!courseDefinitionRepository.existsCourseDefinitionWith(command.getTenantId(), command.getCourseDescription().title())) { courseDefinitionRepository.save(CourseDefinition.builder(command.getTenantId()) .withCourseDates(command.getCourseDates()) .withCourseDescription(command.getCourseDescription()) .withCourseObjective(command.getCourseObjective()) .withDuration(command.getCourseDuration()) .withTeacherId(command.getTeacherId()) .withPreRequirements(command.getPreRequirements()) .withCourseId(command.getCourseId()) .createNew()); } else { throw new IllegalStateException("Course definition with value " + command.getCourseDescription().title() + " already exists"); } } } The command handler creates an instance of the CourseDefinition entity. The business logic and invariants (if any) of creating a Course Definition are encapsulated within the domain entity. The creation of a new instance of the domain entity also comes with the corresponding CourseDefinitionCreated domain event: Java @Getter public class CourseDefinition extends MultiTenantEntity { /* ... */ public static class CourseDefinitionBuilder { /* ... */ public CourseDefinition createNew() { //new instance, generate domain event CourseDefinition courseDefinition = new CourseDefinition(tenantId, 0L, id, courseDescription, courseObjective, preRequirements, duration, teacherId, courseDates, CourseStatus.PENDING_TO_REVIEW); var courseDefinitionCreatedEvent = CourseDefinitionCreatedEvent.triggeredFrom(courseDefinition); courseDefinition.record(courseDefinitionCreatedEvent); //record event in memory return courseDefinition; } } } The event is "recorded" into the created course definition instance. This method is defined in the abstract class Entity of the toolkit-core module: Java public abstract class Entity extends ModelValidator implements ConcurrencyAware { @NotNull @Valid protected final List<@NotNull Event<? extends DomainEventPayload>> events; private Long version; public Entity(Long version) { this.events = new ArrayList<>(); this.version = version; } /*...*/ protected void record(Event<? extends DomainEventPayload> event) { event.setAggregateType(aggregateType()); event.setAggregateId(aggregateId()); event.setEventStreamVersion(Objects.isNull(version)?0:version + events.size()); this.events.add(event); } public List<Event<? extends DomainEventPayload>> getDomainEvents() { return Collections.unmodifiableList(events); } } Once the course definition instance is in place, the command handler will persist the instance in the course definition repository, starting the second phase of the processing flow. Phase 2, Persisting the State: Course Definition and Events in the Outbox Whenever a domain entity is saved in the repository, the domain events associated with the domain state mutation (in this example, the creating of a CourseDefinition entity) are temporarily appended to an in-memory, ThreadLocal, buffer. This buffer resides in the DomainEventAppender of toolkit-core. Java public class DomainEventAppender { private final ThreadLocal<List<Event<? extends DomainEventPayload>>> eventsToPublish = new ThreadLocal<>(); /*...*/ public void append(List<Event<? extends DomainEventPayload>> events) { //add the event to the buffer, later this event will be published to other bounded contexts if(isNull(eventsToPublish.get())) { resetBuffer(); } //ensure event is not already in buffer events.stream().filter(this::notInBuffer).map(this::addEventSourceMetadata).forEach(event -> eventsToPublish.get().add(event)); } /*...*/ } Events are placed in this buffer from an aspect executed around methods annotated with AppendEvents. The pointcut and aspect (both in toolkit-core) look like: Java public class CrossPointcuts { @Pointcut("@annotation(io.twba.tk.core.AppendEvents)") public void shouldAppendEvents() {} } @Aspect @Named public class DomainEventAppenderConcern { private final DomainEventAppender domainEventAppender; @Inject public DomainEventAppenderConcern(DomainEventAppender domainEventAppender) { this.domainEventAppender = domainEventAppender; } @After(value = "io.twba.tk.aspects.CrossPointcuts.shouldAppendEvents()") public void appendEventsToBuffer(JoinPoint jp) { if(Entity.class.isAssignableFrom(jp.getArgs()[0].getClass())) { Entity entity = (Entity)jp.getArgs()[0]; domainEventAppender.append(entity.getDomainEvents()); } } } The command handlers are automatically decorated before being "injected" into the command bus. One of the decorators ensures the command handlers are transactional, and another ensures when the transaction completes the events in the in-memory-thread-local buffer are published to the outbox table consistently with the ongoing transaction. The next sequence diagram shows the decorators applied to the domain-specific command handler. High-Level Processing Flow The outbox is an append-only buffer (as a Postgres table in this example) where a new entry for each event is added. The outbox entry has the following structure: Java public record OutboxMessage(String uuid, String header, String payload, String type, long epoch, String partitionKey, String tenantId, String correlationId, String source, String aggregateId) { } Where the actual event payload is serialized as a JSON string in the payload property. The outbox interface is straightforward: Java public interface Outbox { void appendMessage(OutboxMessage outboxMessage); int partitionFor(String partitionKey); } The Postgres implementation of the Outbox interface is placed in the toolkit-outbox-jpa-postgres module: Java public class OutboxJpa implements Outbox { private final OutboxProperties outboxProperties; private final OutboxMessageRepositoryJpaHelper helper; @Autowired public OutboxJpa(OutboxProperties outboxProperties, OutboxMessageRepositoryJpaHelper helper) { this.outboxProperties = outboxProperties; this.helper = helper; } @Override public void appendMessage(OutboxMessage outboxMessage) { helper.save(toJpa(outboxMessage)); } @Override public int partitionFor(String partitionKey) { return MurmurHash3.hash32x86(partitionKey.getBytes()) % outboxProperties.getNumPartitions(); } /*...*/ } Phase 2 is completed, and now both the domain state and the corresponding event consistently persist under the same transaction in our Postgres database. Phase 3: Events Publication In order to make the domain events generated in the previous phase available to external consumers, the message relay implementation based on Debezium Embedded is monitoring the outbox table, so that whenever a new record is added to the outbox, the message relay creates a Cloud Event and publishes it to the RabbitMQ instance following the Cloud Event AMQP binding specification. The following code snippet shows the Message Relay specification as defined in the toolkit-core module: Java public class DebeziumMessageRelay implements MessageRelay { private static final Logger log = LoggerFactory.getLogger(DebeziumMessageRelay.class); private final Executor executor = Executors.newSingleThreadExecutor(r -> new Thread(r, "debezium-message-relay")); private final CdcRecordChangeConsumer recordChangeConsumer; private final DebeziumEngine<RecordChangeEvent<SourceRecord>> debeziumEngine; public DebeziumMessageRelay(DebeziumProperties debeziumProperties, CdcRecordChangeConsumer recordChangeConsumer) { this.debeziumEngine = DebeziumEngine.create(ChangeEventFormat.of(Connect.class)) .using(DebeziumConfigurationProvider.outboxConnectorConfig(debeziumProperties).asProperties()) .notifying(this::handleChangeEvent) .build(); this.recordChangeConsumer = recordChangeConsumer; } private void handleChangeEvent(RecordChangeEvent<SourceRecord> sourceRecordRecordChangeEvent) { SourceRecord sourceRecord = sourceRecordRecordChangeEvent.record(); Struct sourceRecordChangeValue= (Struct) sourceRecord.value(); log.info("Received record - Key = '{}' value = '{}'", sourceRecord.key(), sourceRecord.value()); Struct struct = (Struct) sourceRecordChangeValue.get(AFTER); recordChangeConsumer.accept(DebeziumCdcRecord.of(struct)); } @Override public void start() { this.executor.execute(debeziumEngine); log.info("Debezium CDC started"); } @Override public void stop() throws IOException { if (this.debeziumEngine != null) { this.debeziumEngine.close(); } } @Override public void close() throws Exception { stop(); } } As can be seen in the code snippet above, the DebeziumEngine is configured to notify a private method handleChangeEvent when a change in the database is detected. In this method, a Consumer of CdcRecord is used as a wrapper of the internal Debezium model represented by the Struct class. Initial configuration must be provided to the Debezium Engine; in the example, this is done with the DebeziumConfigurationProvider class: Java public class DebeziumConfigurationProvider { public static io.debezium.config.Configuration outboxConnectorConfig(DebeziumProperties properties) { return withCustomProps(withStorageProps(io.debezium.config.Configuration.create() .with("name", "outbox-connector") .with("connector.class", properties.getConnectorClass()) .with("offset.storage", properties.getOffsetStorage().getType()), properties.getOffsetStorage()) .with("offset.flush.interval.ms", properties.getOffsetStorage().getFlushInterval()) .with("database.hostname", properties.getSourceDatabaseProperties().getHostname()) .with("database.port", properties.getSourceDatabaseProperties().getPort()) .with("database.user", properties.getSourceDatabaseProperties().getUser()) .with("database.password", properties.getSourceDatabaseProperties().getPassword()) .with("database.dbname", properties.getSourceDatabaseProperties().getDbName()), properties) .with("database.server.id", properties.getSourceDatabaseProperties().getServerId()) .with("database.server.name", properties.getSourceDatabaseProperties().getServerName()) .with("skipped.operations", "u,d,t") .with("include.schema.changes", "false") .with("table.include.list", properties.getSourceDatabaseProperties().getOutboxTable()) .with("snapshot.include.collection.list", properties.getSourceDatabaseProperties().getOutboxTable()) .build(); } private static Configuration.Builder withStorageProps(Configuration.Builder debeziumConfigBuilder, DebeziumProperties.OffsetProperties offsetProperties) { offsetProperties.getOffsetProps().forEach(debeziumConfigBuilder::with); return debeziumConfigBuilder; } private static Configuration.Builder withCustomProps(Configuration.Builder debeziumConfigBuilder, DebeziumProperties debeziumProperties) { debeziumProperties.getCustomProps().forEach(debeziumConfigBuilder::with); return debeziumConfigBuilder; } } The most relevant properties are outlined below: connector.class: The name of the connector to use; usually this is related to the database technology being tracked for changes. In this example, we are using io.debezium.connector.postgresql.PostgresConnector. offset.storage: Type of storage for maintaining the offset; in this example, we are using org.apache.kafka.connect.storage.MemoryOffsetBackingStore, so that offsets are lost after restarting the service. See below. offset.flush.interval. ms: Number of milliseconds for the offsets to be persisted in the offset store database.*: These properties refer to the database being tracked (CDC) for changes by Debezium. skipped.operations: If not specified, all the operations will be tracked. For our example, since we only want to detect newly created events, all the operations but inserts are skipped. table.include.list: List of the tables to include for the CDC; in this example, only the table where events are stored during Phase 2 is relevant (i.e., outbox_schema.outbox) The first thing Debezium will do after starting is take a snapshot of the current data and generate the corresponding events. After that, the offset is updated to the latest record, and the deltas (newly added, updated, or deleted records) are processed, updating the offset accordingly. Since in the provided example we are using an in-memory-based offset store, the snapshot is performed always after starting the service. Therefore, since this is not a production-ready implementation yet, there are two options: Use a durable offset store (file-based or Kafka-based are supported by Debezium Embedded) Delete the outbox table after the events are being processed, ensuring the delete operations are skipped by Debezium in its configuration. The Message Relay is configured and initialized in the toolkit-state-propagation-debezium-runtime module. The values of the configuration properties needed by Debezium Embedded are defined in the Spring Boot properties.yaml file: YAML server: port: 9091 management: endpoints: web: exposure: include: prometheus, health, flyway, info debezium: connector-class: "io.debezium.connector.postgresql.PostgresConnector" custom-props: "[topic.prefix]": "embedded-debezium" "[debezium.source.plugin.name]": "pgoutput" "[plugin.name]": "pgoutput" source-database-properties: db-name: "${CDC_DB_NAME}" hostname: "${CDC_HOST}" user: "${CDC_DB_USER}" password: "${CDC_DB_PASSWORD}" port: 5432 server-name: "debezium-message-relay" server-id: "debezium-message-relay-1" outbox-table: "${CDC_OUTBOX_TABLE}:outbox_schema.outbox" outbox-schema: "" offset-storage: type: "org.apache.kafka.connect.storage.MemoryOffsetBackingStore" flush-interval: 3000 offset-props: "[offset.flush.timeout.ms]": 1000 "[max.poll.records]": 1000 The engine is started using the Spring Boot lifecycle events: Java @Component public class MessageRelayInitializer implements ApplicationListener<ApplicationReadyEvent> { private final MessageRelay messageRelay; @Autowired public MessageRelayInitializer(MessageRelay messageRelay) { this.messageRelay = messageRelay; } @Override public void onApplicationEvent(ApplicationReadyEvent event) { messageRelay.start(); } } The change data capture records (CdcRecord) are processed by the CloudEventRecordChangeConsumer which creates the Cloud Event representation of the CDC record and publishes it through the MessagePublisher. Java public class CloudEventRecordChangeConsumer implements CdcRecordChangeConsumer { /*...*/ @Override public void accept(CdcRecord cdcRecord) { final CloudEvent event; try { String payload = cdcRecord.valueOf("payload"); String uuid = cdcRecord.valueOf("uuid"); String type = cdcRecord.valueOf("type"); String tenantId = cdcRecord.valueOf("tenant_id"); String aggregateId = cdcRecord.valueOf("aggregate_id"); long epoch = cdcRecord.valueOf("epoch"); String partitionKey = cdcRecord.valueOf("partition_key"); String source = cdcRecord.valueOf("source"); String correlationId = cdcRecord.valueOf("correlation_id"); event = new CloudEventBuilder() .withId(uuid) .withType(type) .withSubject(aggregateId) .withExtension(TwbaCloudEvent.CLOUD_EVENT_TENANT_ID, tenantId) .withExtension(TwbaCloudEvent.CLOUD_EVENT_TIMESTAMP, epoch) .withExtension(TwbaCloudEvent.CLOUD_EVENT_PARTITION_KEY, partitionKey) .withExtension(TwbaCloudEvent.CLOUD_EVENT_CORRELATION_ID, correlationId) .withExtension(TwbaCloudEvent.CLOUD_EVENT_GENERATING_APP_NAME, source) .withSource(URI.create("https://thewhiteboardarchitect.com/" + source)) .withData("application/json",payload.getBytes("UTF-8")) .build(); messagePublisher.publish(event); } catch (Exception e) { throw new RuntimeException(e); } } } The provided MessagePublisher is a simple RabbitMQ outbound adapter converting the Cloud Event to the corresponding AMQP message as per the Cloud Event AMQP protocol binding. Java public class MessagePublisherRabbitMq implements MessagePublisher { /*...*/ @Override public boolean publish(CloudEvent dispatchedMessage) { MessageProperties messageProperties = new MessageProperties(); messageProperties.setContentType(MessageProperties.CONTENT_TYPE_JSON); rabbitTemplate.send("__MR__" + dispatchedMessage.getExtension(CLOUD_EVENT_GENERATING_APP_NAME), //custom extension for message routing dispatchedMessage.getType(), toAmqpMessage(dispatchedMessage)); return true; } private static Message toAmqpMessage(CloudEvent dispatchedMessage) { return MessageBuilder.withBody(Objects.nonNull(dispatchedMessage.getData()) ? dispatchedMessage.getData().toBytes() : new byte[0]) .setContentType(MessageProperties.CONTENT_TYPE_JSON) .setMessageId(dispatchedMessage.getId()) .setHeader(CLOUD_EVENT_AMQP_BINDING_PREFIX + CLOUD_EVENT_TENANT_ID, dispatchedMessage.getExtension(CLOUD_EVENT_TENANT_ID)) .setHeader(CLOUD_EVENT_AMQP_BINDING_PREFIX + CLOUD_EVENT_TIMESTAMP, dispatchedMessage.getExtension(CLOUD_EVENT_TIMESTAMP)) .setHeader(CLOUD_EVENT_AMQP_BINDING_PREFIX + CLOUD_EVENT_PARTITION_KEY, dispatchedMessage.getExtension(CLOUD_EVENT_PARTITION_KEY)) .setHeader(CLOUD_EVENT_AMQP_BINDING_PREFIX + CLOUD_EVENT_SUBJECT, dispatchedMessage.getSubject()) .setHeader(CLOUD_EVENT_AMQP_BINDING_PREFIX + CLOUD_EVENT_SOURCE, dispatchedMessage.getSource().toString()) .build(); } } After Phase 3, the events are published to the producer's (Course Management Service) message relay RabbitMQ exchange defined under the convention __MR__<APP_NAME>. In our example, __MR__course-management, messages are routed to the right exchange based on a custom cloud event extension as shown in the previous code snippet. Visit my GitHub repository and check the readme file to see how to spin up the example. Alternative Solutions This example makes use of Debezium Embedded to provide a change data capture and message relay solution. This works fine for technologies supported by Debezium through its connectors. For non-supported providers, alternative approaches can be applied: DynamoDB Streams: Suitable for DynamoDB databases; in combination with Kinesis, it can be used for subscribing to changes in a DynamoDB (outbox) table Custom database polling: This could be implemented for supporting databases with no connectors for Debezium. Adding any of those alternatives in this example would simply just provide specific implementations of the MessageRelay interface, without additional changes in any of the other services. Conclusion Ensuring consistency in the state propagation and data exchange between services is key for providing a reliable distributed solution. Usually, this is not carefully considered when designing distributed, event-driven software, leading to undesired states and situations especially when those systems are already in production. By the combination of the transactional outbox pattern and message relay, we have seen how this consistency can be enforced, and by using the hexagonal architecture style, the additional complexity of implementing those patterns can be easily hidden and reused across bounded context implementations. The code of this article is not yet production-ready, as concerns like observability, retries, scalability (e.g., with partitioning), proper container orchestration, etc. are still pending. Subsequent articles will go through those concerns using the provided code as a basis.

Back in 2018, I decided to use my free time to help modernize a family member’s business. Along the way, I wanted to gain some experience and understanding of AWS. Ultimately, I discovered that nearly all my free time was spent learning AWS cloud infrastructure concepts. I had only a small fraction of my time left to focus on building the modern, cloud-based solution that I originally had in mind. As I planned more feature requests for the app, I realized that I needed a better approach. In early 2020, I discovered Heroku. Since I didn’t need to worry about underlying cloud configurations, I could focus my time on adding new features. The Heroku ecosystem worked out great for my simple use case, but I began to wonder about more complex use cases. What about the scenario where a collection of secure and private services needs to interact with one another for a payment processing solution? Would this use case force me to live in the ecosystem of one of the big three cloud service providers? I’m going to find out. The Current State of Secure Microservices For several years, I was fortunate to work in an environment that valued the DevOps lifecycle. The DevOps team handled all things cloud for me, so I could focus on architecting and building microservices to meet my customers’ needs. During that time of my life, this environment was the exception, not the norm. I just did a search for “companies lacking cloud infrastructure knowledge” in my browser, and the results yielded some pretty surprising conclusions: There is a serious shortage of cloud expertise. The lack of cloud skills is leading to significant performance impacts with cloud-native services. Cloud security is a challenge for 1 in 4 companies. The top search results talked about a lack of understanding of the core cloud concepts and the need for crucial training for teams to be effective. The training most teams need usually falls by the wayside as customer demands and deliverables take higher priority. With this current approach, most cloud implementations are forced to move at a slower pace, and they’re often exposed to unknown vulnerabilities. The current state of securing your microservices in the cloud is not a happy one. The Ideal State for Secure Microservices The ideal state for cloud-native solutions would adhere to a personal mission statement I established several years ago: “Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” – J. Vester In this context, those with a directive to drive toward cloud-native solutions should be able to move at a pace that aligns with corporate objectives. They shouldn’t be slowed by the learning curve associated with underlying cloud infrastructure. So, what does this look like when we’re facing a cloud solution encompassing multiple microservices, all of which need to be isolated from the public and adhere to compliance regulations (like SOC, ISO, PCI, or HIPAA)? About Private Spaces My 2020 Heroku experience was positive. So I wanted to see how it would work with this complex use case. That’s when I discovered Private Spaces. Private Spaces are available as a part of Heroku Enterprise. They’re dedicated environments for running microservices within an isolated network. This approach allows teams to deploy their services into a network that’s not exposed to the public internet. Under the hood, these services function exactly the same as in my basic use case. I can set them up through the Heroku CLI, and simple Git-based commands can trigger deployments. For the regulatory compliance needs, I can lean on Heroku Shield to help me comply with PCI DSS, HIPAA, ISO (27001, 27017, and 27018), and SOC (1, 2, and 3). At a high level, Heroku lets me implement a secure cloud-native design that can be illustrated like this: Here, we have an implementation that leverages Heroku Shield all within a Private Space. This allows a collection of microservices — utilizing several different programming languages — to interact with all the major primary and secondary card networks, all while adhering to various regulatory compliance requirements. Additionally, I get secure communications with the Salesforce platform and GitLab. Heroku in Action Using the Heroku CLI, I can get my Private Space and Heroku Shield up and running. In Heroku, this is called a Shield Private Space. Here are some high-level examples to work through the process. To create a new Shield Private Space, we use spaces:create and add the --shield option. Shell $ heroku spaces:create payment-network --shield --team payments-team --region oregon Creating space payment-network in team payments-team... done === payment-network Team: payments-team Region: oregon State: allocating If the use case requires Classless Inter-Domain Routing (CIDR) ranges, I can use --cidr and --data-cidr flags. You’ll notice that I created my Private Space in the oregon region. You can create a Private Space in one of 10 available regions (in the U.S., Europe, Asia, and Australia). For a list of available regions, do the following: Shell $ heroku regions ID Location Runtime ───────── ─────────────────────── ────────────── eu Europe Common Runtime us United States Common Runtime dublin Dublin, Ireland Private Spaces frankfurt Frankfurt, Germany Private Spaces london London, United Kingdom Private Spaces montreal Montreal, Canada Private Spaces mumbai Mumbai, India Private Spaces oregon Oregon, United States Private Spaces singapore Singapore Private Spaces sydney Sydney, Australia Private Spaces tokyo Tokyo, Japan Private Spaces virginia Virginia, United States Private Spaces For each microservice that needs to run in the payment-network Private Space, I simply add the --space option when running the apps:create command: Shell $ heroku apps:create clearing-service --space payment-network Creating app... done, clearing-service To grant consumers access to the payment-network space, I can maintain the allowed list of trusted IPs: Shell $ heroku trusted-ips:add 192.0.2.128/26 --space payment-network Added 192.0.2.128/26 to trusted IP ranges on payment-network - WARNING: It may take a few moments for the changes to take effect. Conclusion Teams are often given a directive from above to adopt a cloud-native approach. However, many teams have a serious gap in understanding when it comes to deploying secure cloud architectures. If you’re using one of the big three cloud providers, then bridging this gap will come at a price—likely missed timelines expected by the product owner. Is there a better option for secure cloud deployment? I think Private Spaces combined with Heroku Shield represents a better option. For me personally, it also matters that Heroku is part of the solutions platform from Salesforce, which has a history of dedication to providing a cloud adoption alternative focused on the success of its customers. So I felt like this was a long-term strategy for consideration. Have a really great day!

By John Vester CORE

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Cloud Native: Championing Cloud Development Across the SDLC. When it comes to software engineering and application development, cloud native has become commonplace in many teams' vernacular. When people survey the world of cloud native, they often come away with the perspective that the entire process of cloud native is for the large enterprise applications. A few years ago, that may have been the case, but with the advancement of tooling and services surrounding systems such as Kubernetes, the barrier to entry has been substantially lowered. Even so, does adopting cloud-native practices for applications consisting of a few microservices make a difference? Just as cloud native has become commonplace, the shift-left movement has made inroads into many organizations' processes. Shifting left is a focus on application delivery from the outset of a project, where software engineers are just as focused on the delivery process as they are on writing application code. Shifting left implies that software engineers understand deployment patterns and technologies as well as implement them earlier in the SDLC. Shifting left using cloud native with microservices development may sound like a definition containing a string of contemporary buzzwords, but there's real benefit to be gained in combining these closely related topics. Fostering a Deployment-First Culture Process is necessary within any organization. Processes are broken down into manageable tasks across multiple teams with the objective being an efficient path by which an organization sets out to reach a goal. Unfortunately, organizations can get lost in their processes. Teams and individuals focus on doing their tasks as best as possible, and at times, so much so that the goal for which the process is defined gets lost. Software development lifecycle (SDLC) processes are not immune to this problem. Teams and individuals focus on doing their tasks as best as possible. However, in any given organization, if individuals on application development teams are asked how they perceive their objectives, responses can include: "Completing stories" "Staying up to date on recent tech stack updates" "Ensuring their components meet security standards" "Writing thorough tests" Most of the answers provided would demonstrate a commitment to the process, which is good. However, what is the goal? The goal of the SDLC is to build software and deploy it. Whether it be an internal or SaaS application, deploying software helps an organization meet an objective. When presented with the statement that the goal of the SDLC is to deliver and deploy software, just about anyone who participates in the process would say, "Well, of course it is." Teams often lose sight of this "obvious" directive because they're far removed from the actual deployment process. A strategic investment in the process can close that gap. Cloud-native abstractions bring a common domain and dialogue across disciplines within the SDLC. Kubernetes is a good basis upon which cloud-native abstractions can be leveraged. Not only does Kubernetes' usefulness span applications of many shapes and sizes, but when it comes to the SDLC, Kubernetes can also be the environment used on systems ranging from local engineering workstations, though the entire delivery cycle, and on to production. Bringing the deployment platform all the way "left" to an engineer's workstation has everyone in the process speaking the same language, and deployment becomes a focus from the beginning of the process. Various teams in the SDLC may look at "Kubernetes Everywhere" with skepticism. Work done on Kubernetes in reducing its footprint for systems such as edge devices has made running Kubernetes on a workstation very manageable. Introducing teams to Kubernetes through automation allows them to iteratively absorb the platform. The most important thing is building a deployment-first culture. Plan for Your Deployment Artifacts With all teams and individuals focused on the goal of getting their applications to production as efficiently and effectively as possible, how does the evolution of application development shift? The shift is subtle. With a shift-left mindset, there aren't necessarily a lot of new tasks, so the shift is where the tasks take place within the overall process. When a detailed discussion of application deployment begins with the first line of code, existing processes may need to be updated. Build Process If software engineers are to deploy to their personal Kubernetes clusters, are they able to build and deploy enough of an application that they're not reliant on code running on a system beyond their workstation? And there is more to consider than just application code. Is a database required? Does the application use a caching system? It can be challenging to review an existing build process and refactor it for workstation use. The CI/CD build process may need to be re-examined to consider how it can be invoked on a workstation. For most applications, refactoring the build process can be accomplished in such a way that the goal of local build and deployment is met while also using the refactored process in the existing CI/CD pipeline. For new projects, begin by designing the build process for the workstation. The build process can then be added to a CI/CD pipeline. The local build and CI/CD build processes should strive to share as much code as possible. This will keep the entire team up to date on how the application is built and deployed. Build Artifacts The primary deliverables for a build process are the build artifacts. For cloud-native applications, this includes container images (e.g., Docker images) and deployment packages (e.g., Helm charts). When an engineer is executing the build process on their workstation, the artifacts will likely need to be published to a repository, such as a container registry or chart repository. The build process must be aware of context. Existing processes may already be aware of their context with various settings for environments ranging from test and staging to production. Workstation builds become an additional context. Given the awareness of context, build processes can publish artifacts to workstation-specific registries and repositories. For cloud-native development, and in keeping with the local workstation paradigm, container registries and chart repositories are deployed as part of the workstation Kubernetes cluster. As the process moves from build to deploy, maintaining build context includes accessing resources within the current context. Parameterization Central to this entire process is that key components of the build and deployment process definition cannot be duplicated based on a runtime environment. For example, if a container image is built and published one way on the local workstation and another way in the CI/CD pipeline. How long will it be before they diverge? Most likely, they diverge sooner than expected. Divergence in a build process will create a divergence across environments, which leads to divergence in teams and results in the eroding of the deployment-first culture. That may sound a bit dramatic, but as soon as any code forks — without a deliberate plan to merge the forks — the code eventually becomes, for all intents and purposes, unmergeable. Parameterizing the build and deployment process is required to maintain a single set of build and deployment components. Parameters define build context such as the registries and repositories to use. Parameters define deployment context as well, such as the number of pod replicas to deploy or resource constraints. As the process is created, lean toward over-parameterization. It's easier to maintain a parameter as a constant rather than extract a parameter from an existing process. Figure 1. Local development cluster Cloud-Native Microservices Development in Action In addition to the deployment-first culture, cloud-native microservices development requires tooling support that doesn't impede the day-to-day tasks performed by an engineer. If engineers can be shown a new pattern for development that allows them to be more productive with only a minimum-to-moderate level of understanding of new concepts, while still using their favorite tools, the engineers will embrace the paradigm. While engineers may push back or be skeptical about a new process, once the impact on their productivity is tangible, they will be energized to adopt the new pattern. Easing Development Teams Into the Process Changing culture is about getting teams on board with adopting a new way of doing something. The next step is execution. Shifting left requires that software engineers move from designing and writing application code to becoming an integral part of the design and implementation of the entire build and deployment process. This means learning new tools and exploring areas in which they may not have a great deal of experience. Human nature tends to resist change. Software engineers may look at this entire process and think, "How can I absorb this new process and these new tools while trying to maintain a schedule?" It's a valid question. However, software engineers are typically fine with incorporating a new development tool or process that helps them and the team without drastically disrupting their daily routine. Whether beginning a new project or refactoring an existing one, adoption of a shift-left engineering process requires introducing new tools in a way that allows software engineers to remain productive while iteratively learning the new tooling. This starts with automating and documenting the build out of their new development environment — their local Kubernetes cluster. It also requires listening to the team's concerns and suggestions as this will be their daily environment. Dev(elopment) Containers The Development Containers specification is a relatively new advancement based on an existing concept in supporting development environments. Many engineering teams have leveraged virtual desktop infrastructure (VDI) systems, where a developer's workstation is hosted on a virtualized infrastructure. Companies that implement VDI environments like the centralized control of environments, and software engineers like the idea of a pre-packaged environment that contains all the components required to develop, debug, and build an application. What software engineers do not like about VDI environments is network issues where their IDEs become sluggish and frustrating to use. Development containers leverage the same concept as VDI environments but bring it to a local workstation, allowing engineers to use their locally installed IDE while being remotely connected to a running container. This way, the engineer has the experience of local development while connected to a running container. Development containers do require an IDE that supports the pattern. What makes the use of development containers so attractive is that engineers can attach to a container running within a Kubernetes cluster and access services as configured for an actual deployment. In addition, development containers support a first-class development experience, including all the tools a developer would expect to be available in a development environment. From a broader perspective, development containers aren't limited to local deployments. When configured for access, cloud environments can provide the same first-class development experience. Here, the deployment abstraction provided by containerized orchestration layers really shines. Figure 2. Microservice development container configured with dev containers The Synergistic Evolution of Cloud-Native Development Continues There's a synergy across shift-left, cloud-native, and microservices development. They present a pattern for application development that can be adopted by teams of any size. Tooling continues to evolve, making practical use of the technologies involved in cloud-native environments accessible to all involved in the application delivery process. It is a culture change that entails a change in mindset while learning new processes and technologies. It's important that teams aren't burdened with a collection of manual processes where they feel their productivity is being lost. Automation helps ease teams into the adoption of the pattern and technologies. As with any other organizational change, upfront planning and preparation is important. Just as important is involving the teams in the plan. When individuals have a say in change, ownership and adoption become a natural outcome. This is an excerpt from DZone's 2024 Trend Report, Cloud Native: Championing Cloud Development Across the SDLC.Read the Free Report

By Ray Elenteny CORE

In any microservice, managing database interactions with precision is crucial for maintaining application performance and reliability. Usually, we will unravel weird issues with database connection during performance testing. Recently, a critical issue surfaced within the repository layer of a Spring microservice application, where improper exception handling led to unexpected failures and service disruptions during performance testing. This article delves into the specifics of the issue and also highlights the pivotal role of the @Transactional annotation, which remedied the issue. Spring microservice applications rely heavily on stable and efficient database interactions, often managed through the Java Persistence API (JPA). Properly managing database connections, particularly preventing connection leaks, is critical to ensuring these interactions do not negatively impact application performance. Issue Background During a recent round of performance testing, a critical issue emerged within one of our essential microservices, which was designated for sending client communications. This service began to experience repeated Gateway time-out errors. The underlying problem was rooted in our database operations at the repository layer. An investigation into these time-out errors revealed that a stored procedure was consistently failing. The failure was triggered by an invalid parameter passed to the procedure, which raised a business exception from the stored procedure. The repository layer did not handle this exception efficiently; it bubbled up. Below is the source code for the stored procedure call: Java public long createInboxMessage(String notifCode, String acctId, String userId, String s3KeyName, List<Notif> notifList, String attributes, String notifTitle, String notifSubject, String notifPreviewText, String contentType, boolean doNotDelete, boolean isLetter, String groupId) throws EDeliveryException { try { StoredProcedureQuery query = entityManager.createStoredProcedureQuery("p_create_notification"); DbUtility.setParameter(query, "v_notif_code", notifCode); DbUtility.setParameter(query, "v_user_uuid", userId); DbUtility.setNullParameter(query, "v_user_id", Integer.class); DbUtility.setParameter(query, "v_acct_id", acctId); DbUtility.setParameter(query, "v_message_url", s3KeyName); DbUtility.setParameter(query, "v_ecomm_attributes", attributes); DbUtility.setParameter(query, "v_notif_title", notifTitle); DbUtility.setParameter(query, "v_notif_subject", notifSubject); DbUtility.setParameter(query, "v_notif_preview_text", notifPreviewText); DbUtility.setParameter(query, "v_content_type", contentType); DbUtility.setParameter(query, "v_do_not_delete", doNotDelete); DbUtility.setParameter(query, "v_hard_copy_comm", isLetter); DbUtility.setParameter(query, "v_group_id", groupId); DbUtility.setOutParameter(query, "v_notif_id", BigInteger.class); query.execute(); BigInteger notifId = (BigInteger) query.getOutputParameterValue("v_notif_id"); return notifId.longValue(); } catch (PersistenceException ex) { logger.error("DbRepository::createInboxMessage - Error creating notification", ex); throw new EDeliveryException(ex.getMessage(), ex); } } Issue Analysis As illustrated in our scenario, when a stored procedure encountered an error, the resulting exception would propagate upward from the repository layer to the service layer and finally to the controller. This propagation was problematic, causing our API to respond with non-200 HTTP status codes—typically 500 or 400. Following several such incidents, the service container reached a point where it could no longer handle incoming requests, ultimately resulting in a 502 Gateway Timeout error. This critical state was reflected in our monitoring systems, with Kibana logs indicating the issue: `HikariPool-1 - Connection is not available, request timed out after 30000ms.` The issue was improper exception handling, as exceptions bubbled up through the system layers without being properly managed. This prevented the release of database connections back into the connection pool, leading to the depletion of available connections. Consequently, after exhausting all connections, the container was unable to process new requests, resulting in the error reported in the Kibana logs and a non-200 HTTP error. Resolution To resolve this issue, we could handle the exception gracefully and not bubble up further, letting JPA and Spring context release the connection to the pool. Another alternative is to use @Transactional annotation for the method. Below is the same method with annotation: Java @Transactional public long createInboxMessage(String notifCode, String acctId, String userId, String s3KeyName, List<Notif> notifList, String attributes, String notifTitle, String notifSubject, String notifPreviewText, String contentType, boolean doNotDelete, boolean isLetter, String groupId) throws EDeliveryException { ……… } The implementation of the method below demonstrates an approach to exception handling that prevents exceptions from propagating further up the stack by catching and logging them within the method itself: Java public long createInboxMessage(String notifCode, String acctId, String userId, String s3KeyName, List<Notif> notifList, String attributes, String notifTitle, String notifSubject, String notifPreviewText, String contentType, boolean doNotDelete, boolean isLetter, String loanGroupId) { try { ....... query.execute(); BigInteger notifId = (BigInteger) query.getOutputParameterValue("v_notif_id"); return notifId.longValue(); } catch (PersistenceException ex) { logger.error("DbRepository::createInboxMessage - Error creating notification", ex); } return -1; } With @Transactional The @Transactional annotation in Spring frameworks manages transaction boundaries. It begins a transaction when the annotated method starts and commits or rolls it back when the method completes. When an exception occurs, @Transactional ensures that the transaction is rolled back, which helps appropriately release database connections back to the connection pool. Without @Transactional If a repository method that calls a stored procedure is not annotated with @Transactional, Spring does not manage the transaction boundaries for that method. The transaction handling must be manually implemented if the stored procedure throws an exception. If not properly managed, this can result in the database connection not being closed and not being returned to the pool, leading to a connection leak. Best Practices Always use @Transactional when the method's operations should be executed within a transaction scope. This is especially important for operations involving stored procedures that can modify the database state. Ensure exception handling within the method includes proper transaction rollback and closing of any database connections, mainly when not using @Transactional. Conclusion Effective transaction management is pivotal in maintaining the health and performance of Spring Microservice applications using JPA. By employing the @Transactional annotation, we can safeguard against connection leaks and ensure that database interactions do not degrade application performance or stability. Adhering to these guidelines can enhance the reliability and efficiency of our Spring Microservices, providing stable and responsive services to the consuming applications or end users.

By Amol Gote CORE

We have a somewhat bare-bones chat service in our series so far. Our service exposes endpoints for managing topics and letting users post messages in topics. For a demo, we have been using a makeshift in-memory store that shamelessly provides no durability guarantees. A basic and essential building block in any (web) service is a data store (for storing, organizing, and retrieving data securely and efficiently). In this tutorial, we will improve the durability, organization, and persistence of data by introducing a database. There are several choices of databases: in-memory (a very basic form of which we have used earlier), object-oriented databases, key-value stores, relational databases, and more. We will not repeat an in-depth comparison of these here and instead defer to others. Furthermore, in this article, we will use a relational (SQL) database as our underlying data store. We will use the popular GORM library (an ORM framework) to simplify access to our database. There are several relational databases available, both free as well as commercial. We will use Postgres (a very popular, free, lightweight, and easy-to-manage database) for our service. Postgres is also an ideal choice for a primary source-of-truth data store because of the strong durability and consistency guarantees it provides. Setting Up the Database A typical pattern when using a database in a service is: |---------------| |-----------| |------------| |------| | Request Proto | <-> | Service | <-> | ORM/SQL | <-> | DB | |---------------| |-----------| |------------| |------| A gRPC request is received by the service (we have not shown the REST Gateway here). The service converts the model proto (e.g., Topic) contained in the request (e.g., CreateTopicRequest) into the ORM library. The ORM library generates the necessary SQL and executes it on the DB (and returns any results). Setting Up Postgres We could go the traditional way of installing Postgres (by downloading and installing its binaries for the specific platforms). However, this is complicated and brittle. Instead, we will start using Docker (and Docker Compose) going forward for a compact developer-friendly setup. Set Up Docker Set up Docker Desktop for your platform following the instructions. Add Postgres to Docker Compose Now that Docker is set up, we can add different containers to this so we can build out the various components and services OneHub requires. docker-compose.yml: version: '3.9' services: pgadmin: image: dpage/pgadmin4 ports: - ${PGADMIN_LISTEN_PORT}:${PGADMIN_LISTEN_PORT} environment: PGADMIN_LISTEN_PORT: ${PGADMIN_LISTEN_PORT} PGADMIN_DEFAULT_EMAIL: ${PGADMIN_DEFAULT_EMAIL} PGADMIN_DEFAULT_PASSWORD: ${PGADMIN_DEFAULT_PASSWORD} volumes: - ./.pgadmin:/var/lib/pgadmin postgres: image: postgres:15.3 environment: POSTGRES_DB: ${POSTGRES_DB} POSTGRES_USER: ${POSTGRES_USER} POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} volumes: - ./.pgdata:/var/lib/postgresql/data ports: - 5432:5432 healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] interval: 5s timeout: 5s retries: 5 That's it. A few key things to note are: The Docker Compose file is an easy way to get started with containers - especially on a single host without needing complicated orchestration engines (hint: Kubernetes). The main part of Docker Compose files are the service sections that describe the containers for each of the services that Docker Compose will be executing as a "single unit in a private network." This is a great way to package multiple related services needed for an application and bring them all up and down in one step instead of having to manage them one by one individually. The latter is not just cumbersome, but also error-prone (manual dependency management, logging, port checking, etc). For now, we have added one service - postgres - running on port 5432. Since the services are running in an isolated context, environment variables can be set to initialize/control the behavior of the services. These environment variables are read from a specific .env file (below). This file can also be passed as a CLI flag or as a parameter, but for now, we are using the default .env file. Some configuration parameters here are the Postgres username, password, and database name. .env: POSTGRES_DB=onehubdb POSTGRES_USER=postgres POSTGRES_PASSWORD=docker ONEHUB_DB_ENDOINT=postgres://postgres:docker@postgres:5432/onehubdb PGADMIN_LISTEN_PORT=5480 PGADMIN_DEFAULT_EMAIL=admin@onehub.com PGADMIN_DEFAULT_PASSWORD=password All data in a container is transient and is lost when the container is shut down. In order to make our database durable, we will store the data outside the container and map it as a volume. This way from within the container, Postgres will read/write to its local directory (/var/lib/postgresql/data) even though all reads/writes are sent to the host's file system (./.pgdata) Another great benefit of using Docker is that all the ports used by the different services are "internal" to the network that Docker creates. This means the same postgres service (which runs on port 5432) can be run on multiple Docker environments without having their ports changed or checked for conflicts. This works because, by default, ports used inside a Docker environment are not exposed outside the Docker environment. Here we have chosen to expose port 5432 explicitly in the ports section of docker-compose.yml. That's it. Go ahead and bring it up: docker compose up If all goes well, you should see a new Postgres database created and initialized with our username, password, and DB parameters from the .env file. The database is now ready: onehub-postgres-1 | 2023-07-28 22:52:32.199 UTC [1] LOG: starting PostgreSQL 15.3 (Debian 15.3-1.pgdg120+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit onehub-postgres-1 | 2023-07-28 22:52:32.204 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 onehub-postgres-1 | 2023-07-28 22:52:32.204 UTC [1] LOG: listening on IPv6 address "::", port 5432 onehub-postgres-1 | 2023-07-28 22:52:32.209 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" onehub-postgres-1 | 2023-07-28 22:52:32.235 UTC [78] LOG: database system was shut down at 2023-07-28 22:52:32 UTC onehub-postgres-1 | 2023-07-28 22:52:32.253 UTC [1] LOG: database system is ready to accept connections The OneHub Docker application should now show up on the Docker desktop and should look something like this: (Optional) Setup a DB Admin Interface If you would like to query or interact with the database (outside code), pgAdmin and adminer are great tools. They can be downloaded as native application binaries, installed locally, and played. This is a great option if you would like to manage multiple databases (e.g., across multiple Docker environments). ... Alternatively ... If it is for this single project and downloading yet another (native app) binary is undesirable, why not just include it as a service within Docker itself!? With that added, our docker-compose.yml now looks like this: docker-compose.yml: version: '3.9' services: pgadmin: image: dpage/pgadmin4 ports: - ${PGADMIN_LISTEN_PORT}:${PGADMIN_LISTEN_PORT} environment: PGADMIN_LISTEN_PORT: ${PGADMIN_LISTEN_PORT} PGADMIN_DEFAULT_EMAIL: ${PGADMIN_DEFAULT_EMAIL} PGADMIN_DEFAULT_PASSWORD: ${PGADMIN_DEFAULT_PASSWORD} volumes: - ./.pgadmin:/var/lib/pgadmin postgres: image: postgres:15.3 environment: POSTGRES_DB: ${POSTGRES_DB} POSTGRES_USER: ${POSTGRES_USER} POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} volumes: - ./.pgdata:/var/lib/postgresql/data ports: - 5432:5432 healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] interval: 5s timeout: 5s retries: 5 The accompanying environment variables are in our .env file: .env: POSTGRES_DB=onehubdb POSTGRES_USER=postgres POSTGRES_PASSWORD=docker ONEHUB_DB_ENDOINT=postgres://postgres:docker@postgres:5432/onehubdb PGADMIN_LISTEN_PORT=5480 PGADMIN_DEFAULT_EMAIL=admin@onehub.com PGADMIN_DEFAULT_PASSWORD=password Now you can simply visit the pgAdmin web console on your browser. Use the email and password specified in the .env file and off you go! To connect to the Postgres instance running in the Docker environment, simply create a connection to postgres (NOTE: container local DNS names within the Docker environment are the service names themselves). On the left-side Object Explorer panel, (right) click on Servers >> Register >> Server... and give a name to your server ("postgres"). In the Connection tab, use the hostname "postgres" and set the names of the database, username, and password as set in the .env file for the POSTGRES_DB, POSTGRES_USER, and POSTGRES_PASSWORD variables respectively. Click Save, and off you go! Introducing Object Relational Mappers (ORMs) Before we start updating our service code to access the database, you may be wondering why the gRPC service itself is not packaged in our docker-compose.yml file. Without this, we would still have to start our service from the command line (or a debugger). This will be detailed in a future post. In a typical database, initialization (after the user and DB setup) would entail creating and running SQL scripts to create tables, checking for new versions, and so on. One example of a table creation statement (that can be executed via psql or pgadmin) is: CREATE TABLE topics ( id STRING NOT NULL PRIMARY KEY, created_at DATETIME DEFAULT CURRENT_TIMESTAMP, updated_at DATETIME DEFAULT CURRENT_TIMESTAMP, name STRING NOT NULL, users TEXT[], ); Similarly, an insertion would also have been manual construction of SQL statements, e.g.: INSERT INTO topics ( id, name ) VALUES ( "1", "Taylor Swift" ); ... followed by a verification of the saved results: select * from topics ; This can get pretty tedious (and error-prone with vulnerability to SQL injection attacks). SQL expertise is highly valuable but seldom feasible - especially being fluent with the different standards, different vendors, etc. Even though Postgres does a great job in being as standards-compliant as possible - for developers - some ease of use with databases is highly desirable. Here ORM libraries are indispensable, especially for developers not dealing with SQL on a regular basis (e.g., yours truly). ORM (Object Relational Mappers) provide an object-like interface to a relational database. This simplifies access to data in our tables (i.e., rows) as application-level classes (Data Access Objects). Table creations and migrations can also be managed by ORM libraries. Behind the scenes, ORM libraries are generating and executing SQL queries on the underlying databases they accessing. There are downsides to using an ORM: ORMs still incur a learning cost for developers during adoption. Interface design choices can play a role in impacting developer productivity. ORMs can be thought of as a schema compiler. The underlying SQL generated by them may not be straightforward or efficient. This results in ORM access to a database being slower than raw SQL, especially for complex queries. However, for complex queries or complex data pattern accesses, other scalability techniques may need to be applied (e.g., sharding, denormalization, etc.). The queries generated by ORMs may not be clear or straightforward, resulting in increased debugging times on slow or complex queries. Despite these downsides, ORMs can be put to good use when not overly relied upon. We shall use a popular ORM library, GORM. GORM comes with a great set of examples and documentation and the quick start is a great starting point. Create DB Models GORM models are our DB models. GORM models are simple Golang structs with struct tags on each member to identify the member's database type. Our User, Topic and Message models are simply this: Topic, Message, User Models package datastore import ( "time" "github.com/lib/pq" ) type BaseModel struct { CreatedAt time.Time UpdatedAt time.Time Id string `gorm:"primaryKey"` Version int // used for optimistic locking } type User struct { BaseModel Name string Avatar string ProfileData map[string]interface{} `gorm:"type:json"` } type Topic struct { BaseModel CreatorId string Name string `gorm:"index:SortedByName"` Users pq.StringArray `gorm:"type:text[]"` } type Message struct { BaseModel ParentId string TopicId string `gorm:"index:SortedByTopicAndCreation,priority:1"` CreatedAt time.Time `gorm:"index:SortedByTopicAndCreation,priority:2"` SourceId string UserId string ContentType string ContentText string ContentData map[string]interface{} `gorm:"type:json"` } Why are these models needed when we have already defined models in our .proto files? Recall that the models we use need to reflect the domain they are operating in. For example, our gRPC structs (in .proto files) reflect the models and programming models from the application's perspective. If/When we build a UI, view-models would reflect the UI/view perspectives (e.g., a FrontPage view model could be a merge of multiple data models). Similarly, when storing data in a database, the models need to convey intent and type information that can be understood and processed by the database. This is why GORM expects data models to have annotations on its (struct) member variables to convey database-specific information like column types, index definitions, index column orderings, etc. A good example of this in our data model is the SortByTopicAndCreation index (which, as the name suggests, helps us list topics sorted by their creation timestamp). Database indexes are one or more (re)organizations of data in a database that speed up retrievals of certain queries (at the cost of increased write times and storage space). We won't go into indexes deeply. There are fantastic resources that offer a deep dive into the various internals of database systems in great detail (and would be highly recommended). The increased writes and storage space must be considered when creating more indexes in a database. We have (in our service) been mindful about creating more indexes and kept these to the bare minimum (to suit certain types of queries). As we scale our services (in future posts) we will revisit how to address these costs by exploring asynchronous and distributed index-building techniques. Data Access Layer Conventions We now have DB models. We could at this point directly call the GORM APIs from our service implementation to read and write data from our (Postgres) database; but first, a brief detail on the conventions we have decided to choose. Motivations Database use can be thought of as being in two extreme spectrums: On the one hand, a "database" can be treated as a better filesystem with objects written by some key to prevent data loss. Any structure, consistency guarantees, optimization, or indexes are fully the responsibility of the application layer. This gets very complicated, error-prone, and hard very fast. On the other extreme, use the database engine as the undisputed brain (the kitchen sink) of your application. Every data access for every view in your application is offered (only) by one or very few (possibly complex) queries. This view, while localizing data access in a single place, also makes the database a bottleneck and its scalability daunting. In reality, vertical scaling (provisioning beefier machines) is the easiest, but most expensive solution - which most vendors will happily recommend in such cases. Horizontal scaling (getting more machines) is hard as increased data coupling and probabilities of node failures (network partitions) mean more complicated and careful tradeoffs between consistency and availability. Our sweet spot is somewhere in between. While ORMs (like GORM) provide an almost 1:1 interface compatibility between SQL and the application needs, being judicious with SQL remains advantageous and should be based on the (data and operational) needs of the application. For our chat application, some desirable (data) traits are: Messages from users must not be lost (durability). Ordering of messages is important (within a topic). Few standard query types: CRUD on Users, Topics, and Messages Message ordering by timestamp but limited to either within a topic or by a user (for last N messages) Given our data "shapes" are simple and given the read usage of our system is much higher especially given the read/write application (i.e .,1 message posted is read by many participants on a Topic), we are choosing to optimize for write consistency, simplicity and read availability, within a reasonable latency). Now we are ready to look at the query patterns/conventions. Unified Database Object First, we will add a simple data access layer that will encapsulate all the calls to the database for each particular model (topic, messages, users). Let us create an overarching "DB" object that represents our Postgres DB (in db/db.go): type OneHubDB struct { storage *gorm.DB } This tells GORM that we have a database object (possibly with a connection) to the underlying DB. The Topic Store, User Store, and Message Store modules all operate on this single DB instance (via GORM) to read/write data from their respective tables (topics, users, messages). Note that this is just one possible convention. We could have instead used three different DB (gorm.DB) instances, one for each entity type: e.g., TopicDB, UserDB, and MessageDB. Use Custom IDs Instead of Auto-Generated Ones We are choosing to generate our own primary key (IDs) for topics, users, and messages instead of depending on the auto-increment (or auto-id) generation by the database engine. This was for the following reasons: An auto-generated key is localized to the database instance that generates it. This means if/when we add more partitions to our databases (for horizontal scaling) these keys will need to be synchronized and migrating existing keys to avoid duplications at a global level is much harder. Auto increment keys offer reduced randomness, making it easy for attackers to "iterate" through all entities. Sometimes we may simply want string keys that are custom assignable if they are available (for SEO purposes). Lack of attribution to keys (e.g., a central/global key server can also allow attribution/annotation to keys for analytics purposes). For these purposes, we have added a GenId table that keeps track of all used IDs so we can perform collision detection, etc: type GenId struct { Class string `gorm:"primaryKey"` Id string `gorm:"primaryKey"` CreatedAt time.Time } Naturally, this is not a scalable solution when the data volume is large, but suffices for our demo and when needed, we can move this table to a different DB and still preserve the keys/IDs. Note that GenId itself is also managed by GORM and uses a combination of Class + Id as its primary key. An example of this is Class=Topic and Id=123. Random IDs are assigned by the application in a simple manner: func randid(maxlen int) string { max_id := int64(math.Pow(36, maxlen)) randval := rand.Int63() % max_id return strconv.FormatInt(randval, 36) } func (tdb *OneHubDB) NextId(cls string) string { for { gid := GenId{Id: randid(), Class: cls, CreatedAt: time.Now()} err := tdb.storage.Create(gid).Error log.Println("ID Create Error: ", err) if err == nil { return gid.Id } } } The method randid generates a maxlen-sized string of random characters. This is as simple as (2^63) mod maxid where maxid = 36 ^ maxlen. The NextId method is used by the different entity create methods (below) to repeatedly generate random IDs if collisions exist. In case you are worried about excessive collisions or are interested in understanding their probabilities, you can learn about them here. Judicious Use of Indexes Indexes are very beneficial to speed up certain data retrieval operations at the expense of increased writes and storage. We have limited our use of indexes to a very handful of cases where strong consistency was needed (and could be scaled easily): Topics sorted by name (for an alphabetical sorting of topics) Messages sorted by the topic and creation time stamps (for the message list natural ordering) What is the impact of this on our application? Let us find out. Topic Creations and Indexes When a topic is created (or it is updated) an index write would be required. Topic creations/updates are relatively low-frequency operations (compared to message postings). So a slightly increased write latency is acceptable. In a more realistic chat application, a topic creation is a bit more heavyweight due to the need to check permissions, apply compliance rules, etc. So this latency hit is acceptable. Furthermore, this index would only be needed when "searching" for topics and even an asynchronous index update would have sufficed. Message Related Indexes To consider the usefulness of indexes related to messages, let us look at some usage numbers. This is a very simple application, so these scalability issues most likely won't be a concern (so feel free to skip this section). If your goals are a bit more lofty, looking at Slack's usage numbers we can estimate/project some usage numbers for our own demo to make it interesting: Number of daily active topics: 100 Number of active users per topic: 10 Message sent by an active user in a topic: Every 5 minutes (assume time to type, read other messages, research, think, etc.) Thus, the number of messages created each day is: = 100 * 10 * (1400 minutes in a day / 5 minutes)= 280k messages per day~ 3 messages per second In the context of these numbers, if we were to create a message every 3 seconds, even with an extra index (or three), we can handle this load comfortably in a typical database that can handle 10k IOPS, which is rather modest. It is easy to wonder if this scales as the number of topics or active users per topic or the creation frenzy increases. Let us consider a more intense setup (in a larger or busier organization). Instead of the numbers above, if we had 10k topics and 100 active users with a message every minute (instead of 5 minutes), our write QPS would be: WriteQPS: = 10000 * 100 * 1400 / 1= 1.4B messages per day~ 14k messages per second That is quite a considerable blow-up. We can solve this in a couple of ways: Accept a higher latency on writes - For example, instead of requiring a write to happen in a few milliseconds, accept an SLO of, say, 500ms. Update indexes asynchronously - This doesn't get us that much further, as the number of writes in a system has not changed - only the when has changed. Shard our data Let us look at sharding! Our write QPS is in aggregate. On a per-topic level, it is quite low (14k/10000 = 1.4 qps). However, user behavior for our application is that such activities on a topic are fairly isolated. We only want our messages to be consistent and ordered within a topic - not globally. We now have the opportunity to dynamically scale our databases (or the Messages tables) to be partitioned by topic IDs. In fact, we could build a layer (a control plane) that dynamically spins up database shards and moves topics around reacting to load as and when needed. We will not go that extreme here, but this series is tending towards just that especially in the context of SaaS applications. The _annoyed_ reader might be wondering if this deep dive was needed right now! Perhaps not - but by understanding our data and user experience needs, we can make careful tradeoffs. Going forward, such mini-dives will benefit us immensely to quickly evaluate tradeoffs (e.g., when building/adding new features). Store Specific Implementations Now that we have our basic DB and common methods, we can go to each of the entity methods' implementations. For each of our entity methods, we will create the basic CRUD methods: Create Update Get Delete List/Search The Create and Update methods are combined into a single "Save" method to do the following: If an ID is not provided then treat it as a create. If an ID is provided treat it as an update-or-insert (upsert) operation by using the NextId method if necessary. Since we have a base model, Create and Update will set CreatedAt and UpdatedAt fields respectively. The delete method is straightforward. The only key thing here is instead of leveraging GORM's cascading delete capabilities, we also delete the related entities in a separate call. We will not worry about consistency issues resulting from this (e.g., errors in subsequent delete methods). For the Get method, we will fetch using a standard GORM get-query-pattern based on a common id column we use for all models. If an entity does not exist, then we return a nil. Users DB Our user entity methods are pretty straightforward using the above conventions. The Delete method additionally also deletes all Messages for/by the user first before deleting the user itself. This ordering is to ensure that if the deletion of topics fails, then the user deletion won't proceed giving the caller to retry. package datastore import ( "errors" "log" "strings" "time" "gorm.io/gorm" ) func (tdb *OneHubDB) SaveUser(topic *User) (err error) { db := tdb.storage topic.UpdatedAt = time.Now() if strings.Trim(topic.Id, " ") == "" { return InvalidIDError // create a new one } result := db.Save(topic) err = result.Error if err == nil && result.RowsAffected == 0 { topic.CreatedAt = time.Now() err = tdb.storage.Create(topic).Error } return } func (tdb *OneHubDB) DeleteUser(topicId string) (err error) { err = tdb.storage.Where("topic_id = ?", topicId).Delete(&Message{}).Error if err == nil { err = tdb.storage.Where("id = ?", topicId).Delete(&User{}).Error } return } func (tdb *OneHubDB) GetUser(id string) (*User, error) { var out User err := tdb.storage.First(&out, "id = ?", id).Error if err != nil { log.Println("GetUser Error: ", id, err) if errors.Is(err, gorm.ErrRecordNotFound) { return nil, nil } else { return nil, err } } return &out, err } func (tdb *OneHubDB) ListUsers(pageKey string, pageSize int) (out []*User, err error) { query := tdb.storage.Model(&User{}).Order("name asc") if pageKey != "" { count := 0 query = query.Offset(count) } if pageSize <= 0 || pageSize > tdb.MaxPageSize { pageSize = tdb.MaxPageSize } query = query.Limit(pageSize) err = query.Find(&out).Error return out, err } Topics DB Our topic entity methods are also pretty straightforward using the above conventions. The Delete method additionally also deletes all messages in the topic first before deleting the user itself. This ordering is to ensure that if the deletion of topics fails then the user deletion won't proceed giving the caller a chance to retry. Topic entity methods: package datastore import ( "errors" "log" "strings" "time" "gorm.io/gorm" ) /////////////////////// Topic DB func (tdb *OneHubDB) SaveTopic(topic *Topic) (err error) { db := tdb.storage topic.UpdatedAt = time.Now() if strings.Trim(topic.Id, " ") == "" { return InvalidIDError // create a new one } result := db.Save(topic) err = result.Error if err == nil && result.RowsAffected == 0 { topic.CreatedAt = time.Now() err = tdb.storage.Create(topic).Error } return } func (tdb *OneHubDB) DeleteTopic(topicId string) (err error) { err = tdb.storage.Where("topic_id = ?", topicId).Delete(&Message{}).Error if err == nil { err = tdb.storage.Where("id = ?", topicId).Delete(&Topic{}).Error } return } func (tdb *OneHubDB) GetTopic(id string) (*Topic, error) { var out Topic err := tdb.storage.First(&out, "id = ?", id).Error if err != nil { log.Println("GetTopic Error: ", id, err) if errors.Is(err, gorm.ErrRecordNotFound) { return nil, nil } else { return nil, err } } return &out, err } func (tdb *OneHubDB) ListTopics(pageKey string, pageSize int) (out []*Topic, err error) { query := tdb.storage.Model(&Topic{}).Order("name asc") if pageKey != "" { count := 0 query = query.Offset(count) } if pageSize <= 0 || pageSize > tdb.MaxPageSize { pageSize = tdb.MaxPageSize } query = query.Limit(pageSize) err = query.Find(&out).Error return out, err } Messages DB Message entity methods: package datastore import ( "errors" "strings" "time" "gorm.io/gorm" ) func (tdb *OneHubDB) GetMessages(topic_id string, user_id string, pageKey string, pageSize int) (out []*Message, err error) { user_id = strings.Trim(user_id, " ") topic_id = strings.Trim(topic_id, " ") if user_id == "" && topic_id == "" { return nil, errors.New("Either topic_id or user_id or both must be provided") } query := tdb.storage if topic_id != "" { query = query.Where("topic_id = ?", topic_id) } if user_id != "" { query = query.Where("user_id = ?", user_id) } if pageKey != "" { offset := 0 query = query.Offset(offset) } if pageSize <= 0 || pageSize > 10000 { pageSize = 10000 } query = query.Limit(pageSize) err = query.Find(&out).Error return out, err } // Get messages in a topic paginated and ordered by creation time stamp func (tdb *OneHubDB) ListMessagesInTopic(topic_id string, pageKey string, pageSize int) (out []*Topic, err error) { err = tdb.storage.Where("topic_id= ?", topic_id).Find(&out).Error return } func (tdb *OneHubDB) GetMessage(msgid string) (*Message, error) { var out Message err := tdb.storage.First(&out, "id = ?", msgid).Error if err != nil { if errors.Is(err, gorm.ErrRecordNotFound) { return nil, nil } else { return nil, err } } return &out, err } func (tdb *OneHubDB) ListMessages(topic_id string, pageKey string, pageSize int) (out []*Message, err error) { query := tdb.storage.Where("topic_id = ?").Order("created_at asc") if pageKey != "" { count := 0 query = query.Offset(count) } if pageSize <= 0 || pageSize > tdb.MaxPageSize { pageSize = tdb.MaxPageSize } query = query.Limit(pageSize) err = query.Find(&out).Error return out, err } func (tdb *OneHubDB) CreateMessage(msg *Message) (err error) { msg.CreatedAt = time.Now() msg.UpdatedAt = time.Now() result := tdb.storage.Model(&Message{}).Create(msg) err = result.Error return } func (tdb *OneHubDB) DeleteMessage(msgId string) (err error) { err = tdb.storage.Where("id = ?", msgId).Delete(&Message{}).Error return } func (tdb *OneHubDB) SaveMessage(msg *Message) (err error) { db := tdb.storage q := db.Model(msg).Where("id = ? and version = ?", msg.Id, msg.Version) msg.UpdatedAt = time.Now() result := q.UpdateColumns(map[string]interface{}{ "updated_at": msg.UpdatedAt, "content_type": msg.ContentType, "content_text": msg.ContentText, "content_data": msg.ContentData, "user_id": msg.SourceId, "source_id": msg.SourceId, "parent_id": msg.ParentId, "version": msg.Version + 1, }) err = result.Error if err == nil && result.RowsAffected == 0 { // Must have failed due to versioning err = MessageUpdateFailed } return } The Messages entity methods are slightly more involved. Unlike the other two, Messages entity methods also include Searching by Topic and Searching by User (for ease). This is done in the GetMessages method that provides paginated (and ordered) retrieval of messages for a topic or by a user. Write Converters To/From Service/DB Models We are almost there. Our database is ready to read/write data. It just needs to be invoked by the service. Going back to our original plan: |---------------| |-----------| |--------| |------| | Request Proto | <-> | Service | <-> | GORM | <-> | DB | |---------------| |-----------| |--------| |------| We have our service models (generated by protobuf tools) and we have our DB models that GORM understands. We will now add converters to convert between the two. Converters for entity X will follow these conventions: A method XToProto of type func(input *datastore.X) (out *protos.X) A method XFromProto of type func(input *protos.X) (out *datastore.X) With that one of our converters (for Topics) is quite simply (and boringly): package services import ( "log" "github.com/lib/pq" ds "github.com/panyam/onehub/datastore" protos "github.com/panyam/onehub/gen/go/onehub/v1" "google.golang.org/protobuf/types/known/structpb" tspb "google.golang.org/protobuf/types/known/timestamppb" ) func TopicToProto(input *ds.Topic) (out *protos.Topic) { var userIds map[string]bool = make(map[string]bool) for _, userId := range input.Users { userIds[userId] = true } out = &protos.Topic{ CreatedAt: tspb.New(input.BaseModel.CreatedAt), UpdatedAt: tspb.New(input.BaseModel.UpdatedAt), Name: input.Name, Id: input.BaseModel.Id, CreatorId: input.CreatorId, Users: userIds, } return } func TopicFromProto(input *protos.Topic) (out *ds.Topic) { out = &ds.Topic{ BaseModel: ds.BaseModel{ CreatedAt: input.CreatedAt.AsTime(), UpdatedAt: input.UpdatedAt.AsTime(), Id: input.Id, }, Name: input.Name, CreatorId: input.CreatorId, } if input.Users != nil { var userIds []string for userId := range input.Users { userIds = append(userIds, userId) } out.Users = pq.StringArray(userIds) } return } The full set of converters can be found here - Service/DB Models Converters. Hook Up the Converters in the Service Definitions Our last step is to invoke the converters above in the service implementation. The methods are pretty straightforward. For example, for the TopicService we have: CreateTopic During creation we allow custom IDs to be passed in. If an entity with the ID exists the request is rejected. If an ID is not passed in, a random one is assigned. Creator and Name parameters are required fields. The topic is converted to a "DBTopic" model and saved by calling the SaveTopic method. UpdateTopic All our Update<Entity> methods follow a similar pattern: Fetch the existing entity (by ID) from the DB. Update the entity fields based on fields marked in the update_mask (so patches are allowed). Update with any extra entity-specific operations (e.g., AddUsers, RemoveUsers, etc.) - these are just for convenience so the caller would not have to provide an entire "final" users list each time. Convert the updated proto to a "DB Model." Call SaveTopic on the DB. SaveTopic uses the "version" field in our DB to perform an optimistically concurrent write. This ensures that by the time the model is loaded and it is being written, a write by another request/thread will not be overwritten. The Delete, List and Get methods are fairly straightforward. The UserService and MessageService also are implemented in a very similar way with minor differences to suit specific requirements. Testing It All Out We have a database up and running (go ahead and start it with docker compose up). We have converters to/from service and database models. We have implemented our service code to access the database. We just need to connect to this (running) database and pass a connection object to our services in our runner binary (cmd/server.go): Add an extra flag to accept a path to the DB. This can be used to change the DB path if needed. var ( addr = flag.String("addr", ":9000", "Address to start the onehub grpc server on.") gw_addr = flag.String("gw_addr", ":8080", "Address to start the grpc gateway server on.") db_endpoint = flag.String("db_endpoint", "", fmt.Sprintf("Endpoint of DB where all topics/messages state are persisted. Default value: ONEHUB_DB_ENDPOINT environment variable or %s", DEFAULT_DB_ENDPOINT)) ) Create *gorm.DB instance from the db_endpoint value. We have already created a little utility method for opening a GORM-compatible SQL DB given an address: cmd/utils/db.go: package utils import ( // "github.com/panyam/goutils/utils" "log" "strings" "github.com/panyam/goutils/utils" "gorm.io/driver/postgres" "gorm.io/driver/sqlite" "gorm.io/gorm" ) func OpenDB(db_endpoint string) (db *gorm.DB, err error) { log.Println("Connecting to DB: ", db_endpoint) if strings.HasPrefix(db_endpoint, "sqlite://") { dbpath := utils.ExpandUserPath((db_endpoint)[len("sqlite://"):]) db, err = gorm.Open(sqlite.Open(dbpath), &gorm.Config{}) } else if strings.HasPrefix(db_endpoint, "postgres://") { db, err = gorm.Open(postgres.Open(db_endpoint), &gorm.Config{}) } if err != nil { log.Println("Cannot connect DB: ", db_endpoint, err) } else { log.Println("Successfully connected DB: ", db_endpoint) } return } Now let us create the method OpenOHDB, which is a simple wrapper that also checks for a db_endpoint value from an environment variable (if it is not provided) and subsequently opens a gorm.DB instance needed for a OneHubDB instance: func OpenOHDB() *ds.OneHubDB { if *db_endpoint == "" { *db_endpoint = cmdutils.GetEnvOrDefault("ONEHUB_DB_ENDPOINT", DEFAULT_DB_ENDPOINT) } db, err := cmdutils.OpenDB(*db_endpoint) if err != nil { log.Fatal(err) panic(err) } return ds.NewOneHubDB(db) } With the above two, we need a simple change to our main method: func main() { flag.Parse() ohdb := OpenOHDB() go startGRPCServer(*addr, ohdb) startGatewayServer(*gw_addr, *addr) } Now we shall also pass the ohdb instance to the GRPC service creation methods. And we are ready to test our durability! Remember we set up auth in a previous part, so we need to pass login credentials, albeit fake ones (where password = login + "123"). Create a Topic curl localhost:8080/v1/topics -u auser:auser123 | json_pp { "nextPageKey" : "", "topics" : [] } That's right. We do not have any topics yet so let us create some. curl -X POST localhost:8080/v1/topics \ -u auser:auser123 \ -H 'Content-Type: application/json' \ -d '{"topic": {"name": "First Topic"}' | json_pp Yielding: { "topic" : { "createdAt" : "1970-01-01T00:00:00Z", "creatorId" : "auser", "id" : "q43u", "name" : "First Topic", "updatedAt" : "2023-08-04T08:14:56.413050Z", "users" : {} } } Let us create a couple more: curl -X POST localhost:8080/v1/topics \ -u auser:auser123 \ -H 'Content-Type: application/json' \ -d '{"topic": {"name": "First Topic", "id": "1"}' | json_pp curl -X POST localhost:8080/v1/topics \ -u auser:auser123 \ -H 'Content-Type: application/json' \ -d '{"topic": {"name": "Second Topic", "id": "2"}' | json_pp curl -X POST localhost:8080/v1/topics \ -u auser:auser123 \ -H 'Content-Type: application/json' \ -d '{"topic": {"name": "Third Topic", "id": "3"}' | json_pp With a list query returning: { "nextPageKey" : "", "topics" : [ { "createdAt" : "1970-01-01T00:00:00Z", "creatorId" : "auser", "id" : "q43u", "name" : "First Topic", "updatedAt" : "2023-08-04T08:14:56.413050Z", "users" : {} }, { "createdAt" : "1970-01-01T00:00:00Z", "creatorId" : "auser", "id" : "dejc", "name" : "Second Topic", "updatedAt" : "2023-08-05T06:52:33.923076Z", "users" : {} }, { "createdAt" : "1970-01-01T00:00:00Z", "creatorId" : "auser", "id" : "zuoz", "name" : "Third Topic", "updatedAt" : "2023-08-05T06:52:35.100552Z", "users" : {} } ] } Get Topic by ID We can do a listing as in the previous section. We can also obtain individual topics: curl localhost:8080/v1/topics/q43u -u auser:auser123 | json_pp { "topic" : { "createdAt" : "1970-01-01T00:00:00Z", "creatorId" : "auser", "id" : "q43u", "name" : "First Topic", "updatedAt" : "2023-08-04T08:14:56.413050Z", "users" : {} } } Send and List Messages on a Topic Let us send a few messages on the "First Topic" (id = "q43u"): curl -X POST localhost:8080/v1/topics/q43u/messages -u 'auser:auser123' -H 'Content-Type: application/json' -d '{"message": {"content_text": "Message 1"}' curl -X POST localhost:8080/v1/topics/q43u/messages -u 'auser:auser123' -H 'Content-Type: application/json' -d '{"message": {"content_text": "Message 2"}' curl -X POST localhost:8080/v1/topics/q43u/messages -u 'auser:auser123' -H 'Content-Type: application/json' -d '{"message": {"content_text": "Message 3"}' Now to list them: curl localhost:8080/v1/topics/q43u/messages -u 'auser:auser123' | json_pp { "messages" : [ { "contentData" : null, "contentText" : "Message 1", "contentType" : "", "createdAt" : "0001-01-01T00:00:00Z", "id" : "hlso", "topicId" : "q43u", "updatedAt" : "2023-08-07T05:00:36.547072Z", "userId" : "auser" }, { "contentData" : null, "contentText" : "Message 2", "contentType" : "", "createdAt" : "0001-01-01T00:00:00Z", "id" : "t3lr", "topicId" : "q43u", "updatedAt" : "2023-08-07T05:00:39.504294Z", "userId" : "auser" }, { "contentData" : null, "contentText" : "Message 3", "contentType" : "", "createdAt" : "0001-01-01T00:00:00Z", "id" : "8ohi", "topicId" : "q43u", "updatedAt" : "2023-08-07T05:00:42.598521Z", "userId" : "auser" } ], "nextPageKey" : "" } Conclusion Who would have thought setting up and using a database would have been such a meaty topic? We covered a lot of ground here that will both give us a good "functioning" service as well as a foundation when implementing new ideas in the future: We chose a relational database - Postgres - for its strong modeling capabilities, consistency guarantees, performance, and versatility. We also chose an ORM library (GORM) to improve our velocity and portability if we need to switch to another relational data store. We wrote data models that GORM could use to read/write from the database. We eased the setup by hosting both Postgres and its admin UI (pgAdmin) in a Docker Compose file. We decided to use GORM carefully and judiciously to balance velocity with minimal reliance on complex queries. We discussed some conventions that will help us along in our application design and extensions. We also addressed a way to assess, analyze, and address scalability challenges as they might arise and use that to guide our tradeoff decisions (e.g., type and number of indexes, etc). We wrote converter methods to convert between service and data models. We finally used the converters in our service to offer a "real" persistent implementation of a chat service where messages can be posted and read. Now that we have a "minimum usable app," there are a lot of useful features to add to our service and make it more and more realistic (and hopefully production-ready). Take a breather and see you soon in continuing the exciting adventure! In the next post, we will look at also including our main binary (with gRPC service and REST Gateways) in the Docker Compose environment without sacrificing hot reloading and debugging.

Readers of my publications are likely familiar with the idea of employing an API First approach to developing microservices. Countless times I have realized the benefits of describing the anticipated URIs and underlying object models before any development begins. In my 30+ years of navigating technology, however, I’ve come to expect the realities of alternate flows. In other words, I fully expect there to be situations where API First is just not possible. For this article, I wanted to walk through an example of how teams producing microservices can still be successful at providing an OpenAPI specification for others to consume without manually defining an openapi.json file. I also wanted to step outside my comfort zone and do this without using Java, .NET, or even JavaScript. Discovering FastAPI At the conclusion of most of my articles I often mention my personal mission statement: “Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” – J. Vester My point in this mission statement is to make myself accountable for making the best use of my time when trying to reach goals and objectives set at a higher level. Basically, if our focus is to sell more widgets, my time should be spent finding ways to make that possible – steering clear of challenges that have already been solved by existing frameworks, products, or services. I picked Python as the programming language for my new microservice. To date, 99% of the Python code I’ve written for my prior articles has been the result of either Stack Overflow Driven Development (SODD) or ChatGPT-driven answers. Clearly, Python falls outside my comfort zone. Now that I’ve level-set where things stand, I wanted to create a new Python-based RESTful microservice that adheres to my personal mission statement with minimal experience in the source language. That’s when I found FastAPI. FastAPI has been around since 2018 and is a framework focused on delivering RESTful APIs using Python-type hints. The best part about FastAPI is the ability to automatically generate OpenAPI 3 specifications without any additional effort from the developer’s perspective. The Article API Use Case For this article, the idea of an Article API came to mind, providing a RESTful API that allows consumers to retrieve a list of my recently published articles. To keep things simple, let’s assume a given Article contains the following properties: id : Simple, unique identifier property (number) title : The title of the article (string) url : The full URL to the article (string) year : The year the article was published (number) The Article API will include the following URIs: GET /articles : Will retrieve a list of articles GET /articles/{article_id} : Will retrieve a single article by the id property POST /articles : Adds a new article FastAPI in Action In my terminal, I created a new Python project called fast-api-demo and then executed the following commands: Shell $ pip install --upgrade pip $ pip install fastapi $ pip install uvicorn I created a new Python file called api.py and added some imports, plus established an app variable: Python from fastapi import FastAPI, HTTPException from pydantic import BaseModel app = FastAPI() if __name__ == "__main__": import uvicorn uvicorn.run(app, host="localhost", port=8000) Next, I defined an Article object to match the Article API use case: Python class Article(BaseModel): id: int title: str url: str year: int With the model established, I needed to add the URIs…which turned out to be quite easy: Python # Route to add a new article @app.post("/articles") def create_article(article: Article): articles.append(article) return article # Route to get all articles @app.get("/articles") def get_articles(): return articles # Route to get a specific article by ID @app.get("/articles/{article_id}") def get_article(article_id: int): for article in articles: if article.id == article_id: return article raise HTTPException(status_code=404, detail="Article not found") To save myself from involving an external data store, I decided to add some of my recently published articles programmatically: Python articles = [ Article(id=1, title="Distributed Cloud Architecture for Resilient Systems: Rethink Your Approach To Resilient Cloud Services", url="https://dzone.com/articles/distributed-cloud-architecture-for-resilient-syste", year=2023), Article(id=2, title="Using Unblocked to Fix a Service That Nobody Owns", url="https://dzone.com/articles/using-unblocked-to-fix-a-service-that-nobody-owns", year=2023), Article(id=3, title="Exploring the Horizon of Microservices With KubeMQ's New Control Center", url="https://dzone.com/articles/exploring-the-horizon-of-microservices-with-kubemq", year=2024), Article(id=4, title="Build a Digital Collectibles Portal Using Flow and Cadence (Part 1)", url="https://dzone.com/articles/build-a-digital-collectibles-portal-using-flow-and-1", year=2024), Article(id=5, title="Build a Flow Collectibles Portal Using Cadence (Part 2)", url="https://dzone.com/articles/build-a-flow-collectibles-portal-using-cadence-par-1", year=2024), Article(id=6, title="Eliminate Human-Based Actions With Automated Deployments: Improving Commit-to-Deploy Ratios Along the Way", url="https://dzone.com/articles/eliminate-human-based-actions-with-automated-deplo", year=2024), Article(id=7, title="Vector Tutorial: Conducting Similarity Search in Enterprise Data", url="https://dzone.com/articles/using-pgvector-to-locate-similarities-in-enterpris", year=2024), Article(id=8, title="DevSecOps: It's Time To Pay for Your Demand, Not Ingestion", url="https://dzone.com/articles/devsecops-its-time-to-pay-for-your-demand", year=2024), ] Believe it or not, that completes the development for the Article API microservice. For a quick sanity check, I spun up my API service locally: Shell $ python api.py INFO: Started server process [320774] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://localhost:8000 (Press CTRL+C to quit) Then, in another terminal window, I sent a curl request (and piped it to json_pp): Shell $ curl localhost:8000/articles/1 | json_pp { "id": 1, "title": "Distributed Cloud Architecture for Resilient Systems: Rethink Your Approach To Resilient Cloud Services", "url": "https://dzone.com/articles/distributed-cloud-architecture-for-resilient-syste", "year": 2023 } Preparing To Deploy Rather than just run the Article API locally, I thought I would see how easily I could deploy the microservice. Since I had never deployed a Python microservice to Heroku before, I felt like now would be a great time to try. Before diving into Heroku, I needed to create a requirements.txt file to describe the dependencies for the service. To do this, I installed and executed pipreqs: Shell $ pip install pipreqs $ pipreqs This created a requirements.txt file for me, with the following information: Plain Text fastapi==0.110.1 pydantic==2.6.4 uvicorn==0.29.0 I also needed a file called Procfile which tells Heroku how to spin up my microservice with uvicorn. Its contents looked like this: Shell web: uvicorn api:app --host=0.0.0.0 --port=${PORT} Let’s Deploy to Heroku For those of you who are new to Python (as I am), I used the Getting Started on Heroku with Python documentation as a helpful guide. Since I already had the Heroku CLI installed, I just needed to log in to the Heroku ecosystem from my terminal: Shell $ heroku login I made sure to check all of my updates in my repository on GitLab. Next, the creation of a new app in Heroku can be accomplished using the CLI via the following command: Shell $ heroku create The CLI responded with a unique app name, along with the URL for app and the git-based repository associated with the app: Shell Creating app... done, powerful-bayou-23686 https://powerful-bayou-23686-2d5be7cf118b.herokuapp.com/ | https://git.heroku.com/powerful-bayou-23686.git Please note – by the time you read this article, my app will no longer be online. Check this out. When I issue a git remote command, I can see that a remote was automatically added to the Heroku ecosystem: Shell $ git remote heroku origin To deploy the fast-api-demo app to Heroku, all I have to do is use the following command: Shell $ git push heroku main With everything set, I was able to validate that my new Python-based service is up and running in the Heroku dashboard: With the service running, it is possible to retrieve the Article with id = 1 from the Article API by issuing the following curl command: Shell $ curl --location 'https://powerful-bayou-23686-2d5be7cf118b.herokuapp.com/articles/1' The curl command returns a 200 OK response and the following JSON payload: JSON { "id": 1, "title": "Distributed Cloud Architecture for Resilient Systems: Rethink Your Approach To Resilient Cloud Services", "url": "https://dzone.com/articles/distributed-cloud-architecture-for-resilient-syste", "year": 2023 } Delivering OpenAPI 3 Specifications Automatically Leveraging FastAPI’s built-in OpenAPI functionality allows consumers to receive a fully functional v3 specification by navigating to the automatically generated /docs URI: Shell https://powerful-bayou-23686-2d5be7cf118b.herokuapp.com/docs Calling this URL returns the Article API microservice using the widely adopted Swagger UI: For those looking for an openapi.json file to generate clients to consume the Article API, the /openapi.json URI can be used: Shell https://powerful-bayou-23686-2d5be7cf118b.herokuapp.com/openapi.json For my example, the JSON-based OpenAPI v3 specification appears as shown below: JSON { "openapi": "3.1.0", "info": { "title": "FastAPI", "version": "0.1.0" }, "paths": { "/articles": { "get": { "summary": "Get Articles", "operationId": "get_articles_articles_get", "responses": { "200": { "description": "Successful Response", "content": { "application/json": { "schema": { } } } } } }, "post": { "summary": "Create Article", "operationId": "create_article_articles_post", "requestBody": { "content": { "application/json": { "schema": { "$ref": "#/components/schemas/Article" } } }, "required": true }, "responses": { "200": { "description": "Successful Response", "content": { "application/json": { "schema": { } } } }, "422": { "description": "Validation Error", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } } } }, "/articles/{article_id}": { "get": { "summary": "Get Article", "operationId": "get_article_articles__article_id__get", "parameters": [ { "name": "article_id", "in": "path", "required": true, "schema": { "type": "integer", "title": "Article Id" } } ], "responses": { "200": { "description": "Successful Response", "content": { "application/json": { "schema": { } } } }, "422": { "description": "Validation Error", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } } } } } } }, "components": { "schemas": { "Article": { "properties": { "id": { "type": "integer", "title": "Id" }, "title": { "type": "string", "title": "Title" }, "url": { "type": "string", "title": "Url" }, "year": { "type": "integer", "title": "Year" } }, "type": "object", "required": [ "id", "title", "url", "year" ], "title": "Article" }, "HTTPValidationError": { "properties": { "detail": { "items": { "$ref": "#/components/schemas/ValidationError" }, "type": "array", "title": "Detail" } }, "type": "object", "title": "HTTPValidationError" }, "ValidationError": { "properties": { "loc": { "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "type": "array", "title": "Location" }, "msg": { "type": "string", "title": "Message" }, "type": { "type": "string", "title": "Error Type" } }, "type": "object", "required": [ "loc", "msg", "type" ], "title": "ValidationError" } } } } As a result, the following specification can be used to generate clients in a number of different languages via OpenAPI Generator. Conclusion At the start of this article, I was ready to go to battle and face anyone not interested in using an API First approach. What I learned from this exercise is that a product like FastAPI can help define and produce a working RESTful microservice quickly while also including a fully consumable OpenAPI v3 specification…automatically. Turns out, FastAPI allows teams to stay focused on their goals and objectives by leveraging a framework that yields a standardized contract for others to rely on. As a result, another path has emerged to adhere to my personal mission statement. Along the way, I used Heroku for the first time to deploy a Python-based service. This turned out to require little effort on my part, other than reviewing some well-written documentation. So another mission statement bonus needs to be mentioned for the Heroku platform as well. If you are interested in the source code for this article you can find it on GitLab. Have a really great day!

By John Vester CORE

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Modern API Management: Connecting Data-Driven Architectures Alongside AI, Automation, and Microservices. Microservices-based applications are distributed in nature, and each service can run on a different machine or in a container. However, splitting business logic into smaller units and deploying them in a distributed manner is just the first step. We then must understand the best way to make them communicate with each other. Microservices Communication Challenges Communication between microservices should be robust and efficient. When several small microservices are interacting to complete a single business scenario, it can be a challenge. Here are some of the main challenges arising from microservice-to-microservice communication. Resiliency There may be multiple instances of microservices, and an instance may fail due to several reasons — for example, it may crash or be overwhelmed with too many requests and thus unable to process requests. There are two design patterns that make communication between microservices more resilient: retry and circuit breakers. Retry In a microservices architecture, transient failures are unavoidable due to communication between multiple services within the application, especially on a cloud platform. These failures could occur due to various scenarios such as a momentary connection loss, response time-out, service unavailability, slow network connections, etc. (Shrivastava, Shrivastav 2022). Normally, these errors resolve by themselves by retrying the request either immediately or after a delay, depending on the type of error that occurred. The retry is carried out for a preconfigured number of times until it times out. However, a point of note is that the logical consistency of the operation must be maintained during the request to obtain repeatable responses and avoid potential side effects outside of our expectations. Circuit Breaker In a microservices architecture, as discussed in the previous section, failures can occur due to several reasons and are typically self-resolving. However, this may not always be the case since a situation of varying severity may arise where the errors take longer than estimated to be resolved or may not be resolved at all. The circuit breaker pattern, as the name implies, causes a break in a function operation when the errors reach a certain threshold. Usually, this break also triggers an alert that can be monitored. As opposed to the retry pattern, a circuit breaker prevents an operation that’s likely to result in failure from being performed. This prevents congestion due to failed requests and the escalation of failures downstream. The operation can be continued with the persisting error enabling the efficient use of computing resources. The error does not stall the completion of other operations that are using the same resource, which is inherently limited (Shrivastava, Shrivastav 2022). Distributed Tracing Modern-day microservices-architecture-based applications are made up of distributed systems that are exceedingly complex to design, and monitoring and debugging them becomes even more complicated. Due to the large number of microservices involved in an application that spans multiple development teams, systems, and infrastructures, even a single request involves a complex network of communication. While this complex distributed system enables a scalable, efficient, and reliable system, it also makes system observability more challenging to achieve, thereby creating issues with troubleshooting. Distributed tracing helps us overcome this observability challenge by using a request-centric view. As a request is processed by the components of a distributed system, distributed tracing captures the detailed execution of the request and its causally related actions across the system's components (Shkuro 2019). Load Balancing Load balancing is the method used to utilize resources optimally and to ensure smooth operational performance. In order to be efficient and scalable, more than one instance of a service is used, and the incoming requests are distributed across these instances for a smooth process flow. In Kubernetes, load balancing algorithms are implemented in a more effective manner using a service mesh, which is based on recorded metrics such as latency. Service meshes mainly manage the traffic between services on the network, ensuring that inter-service communications are safe and reliable by enabling the services to detect and communicate with each other. The use of a service mesh improves observability and aids in monitoring highly distributed systems. Security Each service must be secured individually, and the communication between services must be secure. In addition, there needs to be a centralized way to manage access controls and authentication across all services. One of the most popular ways for securing microservices is to use API gateways, which act as proxies between the clients and the microservices. API gateways can perform authentication and authorization checks, rate limiting, and traffic management. Service Versioning The deployment of a microservice version update often leads to unexpected issues and breaking errors between the new version of the microservice and other microservices in the system, or even external clients using that microservice. While the team deploying the new version attempts to mitigate and reduce these breaks, multiple versions of the same microservice can be run simultaneously, thereby allowing requests to be routed to the appropriate version of the microservice. This is done using API versioning for API contracts. Communication Patterns Communication between microservices can be designed by using two main patterns: synchronous and asynchronous. In Figure 1, we see a basic overview of these communication patterns along with their respective implementation styles and choices. Figure 1. Synchronous and asynchronous communication with common implementation technologies Synchronous Pattern Synchronous communication between microservices is one-to-one communication. The microservice that generates the request is blocked until a response is received from the other service. This is done using HTTP requests or gRPC — a high-performance remote procedure call (RPC) framework. In synchronous communication, the microservices are tightly coupled, which is advantageous for less distributed architectures where communication happens in real time, thereby reducing the complexity of debugging (Newman 2021). Figure 2. Synchronous communication depicting the request-response model The following table shows a comparison between technologies that are commonly used to implement the synchronous communication pattern. Table 1. REST vs. gRPC vs. GraphQL REST gRPC GraphQL Architectural principles Uses a stateless client-server architecture; relies on URIs and HTTP methods for a layered system with a uniform interface Uses the client-server method of remote procedure call; methods are directly called by the client and behave like local methods, although they are on the server side Uses client-driven architecture principles; relies on queries, mutations, and subscriptions via APIs to request, modify, and update data from/on the server HTTP methods POST, GET, PUT, DELETE Custom methods POST Payload data structure to send/receive data JSON- and XML-based payload Protocol Buffers-based serialized payloads JSON-based payloads Request/response caching Natively supported on client and server side Unsupported by default Supported but complex as all requests have a common endpoint Code generation Natively unsupported; requires third-party tools like Swagger Natively supported Natively unsupported; requires third-party tools like GraphQL code generator Asynchronous Pattern In asynchronous communication, as opposed to synchronous, the microservice that initiates the request is not blocked until the response is received. It can proceed with other processes without receiving a response from the microservice it sends the request to. In the case of a more complex distributed microservices architecture, where the services are not tightly coupled, asynchronous message-based communication is more advantageous as it improves scalability and enables continued background operations without affecting critical processes (Newman 2021). Figure 3. Asynchronous communication Event-Driven Communication The event-driven communication pattern leverages events to facilitate communication between microservices. Rather than sending a request, microservices generate events without any knowledge of the other microservices' intents. These events can then be used by other microservices as required. The event-driven pattern is asynchronous communication as the microservices listening to these events have their own processes to execute. The principle behind events is entirely different from the request-response model. The microservice emitting the event leaves the recipient fully responsible for handling the event, while the microservice itself has no idea about the consequences of the generated event. This approach enables loose coupling between microservices (Newman 2021). Figure 4. Producers emit events that some consumers subscribe to Common Data Communication through common data is asynchronous in nature and is achieved by having a microservice store data at a specific location where another microservice can then access that data. The data's location must be persistent storage, such as data lakes or data warehouses. Although common data is frequently used as a method of communication between microservices, it is often not considered a communication protocol because the coupling between microservices is not always observable when it is used. This communication style finds its best use case in situations that involve large volumes of data as a common data location prevents redundancy, makes data processing more efficient, and is easily scalable (Newman 2021). Figure 5. An example of communication through common data Request-Response Communication The request-response communication model is similar to the synchronous communication that was previously discussed — where a microservice provides a request to another microservice and has to await a response. Along with the previously discussed protocols (HTTP, gPRC, etc.), message queues are used as well. Request-response is implemented as one of the following two methods: Blocking synchronous – Microservice A opens a network connection and sends a request to Microservice B along this connection. The established connection stays open while Microservice A waits for Microservice B to respond. Non-blocking asynchronous – Microservice A sends a request to Microservice B, and Microservice Bneeds to know implicitly where to route the response. Also, message queues can be used; they provide an added benefit of buffering multiple requests in the queue to await processing. This method is helpful in situations where the rate of requests received exceeds the rate of handling these requests. Rather than trying to handle more requests than its capacity, the microservice can take its time generating a response before moving on to handle the next request (Newman 2021). Figure 6. An example of request-response non-blocking asynchronous communication Conclusion In recent years, we have observed a paradigm shift from designing large, clunky, monolithic applications that are complex to scale and maintain to using microservices-based architectures that enable the design of distributed applications — ones that can integrate multiple communication patterns and protocols across systems. These complex distributed systems can be developed, deployed, scaled, and maintained independently by different teams with fewer conflicts, resulting in a more robust, reliable, and resilient application. Using the most optimal communication pattern and protocol for the exact operation that a microservice must achieve is a crucial task and has a huge impact on the functionality and performance of an application. The aim is to make the communication between microservices as seamless as possible to establish an efficient system. In-depth knowledge regarding the available communication patterns and protocols is an essential aspect of modern-day cloud-based application design that is not only dynamic but also highly competitive with multiple contenders providing identical applications and services. Speed, scalability, efficiency, security, and other additional features are often crucial in determining the overall quality of an application, and proper microservices communication is the backbone to achieving those capabilities. References: Shrivastava, Saurabh. Shrivastav, Neelanjali. 2022. Solutions Architect's Handbook, 2nd Edition. Packt. Shkuro, Yuri. 2019. Mastering Distributed Tracing. Packt. Newman, Sam. 2021. Building Microservices, 2nd Edition. O'Reilly. This is an excerpt from DZone's 2024 Trend Report, Modern API Management: Connecting Data-Driven Architectures Alongside AI, Automation, and Microservices.Read the Free Report

The monolithic architecture was historically used by developers for a long time — and for a long time, it worked. Unfortunately, these architectures use fewer parts that are larger, thus meaning they were more likely to fail in entirety if a single part failed. Often, these applications ran as a singular process, which only exacerbated the issue. Microservices solve these specific issues by having each microservice run as a separate process. If one cog goes down, it doesn’t necessarily mean the whole machine stops running. Plus, diagnosing and fixing defects in smaller, highly cohesive services is often easier than in larger monolithic ones. Microservices design patterns provide tried-and-true fundamental building blocks that can help write code for microservices. By utilizing patterns during the development process, you save time and ensure a higher level of accuracy versus writing code for your microservices app from scratch. In this article, we cover a comprehensive overview of microservices design patterns you need to know, as well as when to apply them. Key Benefits of Using Microservices Design Patterns Microservices design patterns offer several key benefits, including: Scalability: Microservices allow applications to be broken down into smaller, independent services, each responsible for a specific function or feature. This modular architecture enables individual services to be scaled independently based on demand, improving overall system scalability and resource utilization. Flexibility and agility: Microservices promote flexibility and agility by decoupling different parts of the application. Each service can be developed, deployed, and updated independently, allowing teams to work autonomously and release new features more frequently. This flexibility enables faster time-to-market and easier adaptation to changing business requirements. Resilience and fault isolation: Microservices improve system resilience and fault isolation by isolating failures to specific services. If one service experiences an issue or failure, it does not necessarily impact the entire application. This isolation minimizes downtime and improves system reliability, ensuring that the application remains available and responsive. Technology diversity: Microservices enable technology diversity by allowing each service to be built using the most suitable technology stack for its specific requirements. This flexibility enables teams to choose the right tools and technologies for each service, optimizing performance, development speed, and maintenance. Improved development and deployment processes: Microservices streamline development and deployment processes by breaking down complex applications into smaller, manageable components. This modular architecture simplifies testing, debugging, and maintenance tasks, making it easier for development teams to collaborate and iterate on software updates. Scalability and cost efficiency: Microservices enable organizations to scale their applications more efficiently by allocating resources only to the services that require them. This granular approach to resource allocation helps optimize costs and ensures that resources are used effectively, especially in cloud environments where resources are billed based on usage. Enhanced fault tolerance: Microservices architecture allows for better fault tolerance as services can be designed to gracefully degrade or fail independently without impacting the overall system. This ensures that critical functionalities remain available even in the event of failures or disruptions. Easier maintenance and updates: Microservices simplify maintenance and updates by allowing changes to be made to individual services without affecting the entire application. This reduces the risk of unintended side effects and makes it easier to roll back changes if necessary, improving overall system stability and reliability. Let's go ahead and look for different Microservices Design Patterns. Database per Service Pattern The database is one of the most important components of microservices architecture, but it isn’t uncommon for developers to overlook the database per service pattern when building their services. Database organization will affect the efficiency and complexity of the application. The most common options that a developer can use when determining the organizational architecture of an application are: Dedicated Database for Each Service A database dedicated to one service can’t be accessed by other services. This is one of the reasons that makes it much easier to scale and understand from a whole end-to-end business aspect. Picture a scenario where your databases have different needs or access requirements. The data owned by one service may be largely relational, while a second service might be better served by a NoSQL solution and a third service may require a vector database. In this scenario, using dedicated services for each database could help you manage them more easily. This structure also reduces coupling as one service can’t tie itself to the tables of another. Services are forced to communicate via published interfaces. The downside is that dedicated databases require a failure protection mechanism for events where communication fails. Single Database Shared by All Services A single shared database isn’t the standard for microservices architecture but bears mentioning as an alternative nonetheless. Here, the issue is that microservices using a single shared database lose many of the key benefits developers rely on, including scalability, robustness, and independence. Still, sharing a physical database may be appropriate in some situations. When a single database is shared by all services, though, it’s very important to enforce logical boundaries within it. For example, each service should own its have schema, and read/write access should be restricted to ensure that services can’t poke around where they don’t belong. Saga Pattern A saga is a series of local transactions. In microservices applications, a saga pattern can help maintain data consistency during distributed transactions. The saga pattern is an alternative solution to other design patterns that allow for multiple transactions by giving rollback opportunities. A common scenario is an e-commerce application that allows customers to purchase products using credit. Data may be stored in two different databases: One for orders and one for customers. The purchase amount can’t exceed the credit limit. To implement the Saga pattern, developers can choose between two common approaches. 1. Choreography Using the choreography approach, a service will perform a transaction and then publish an event. In some instances, other services will respond to those published events and perform tasks according to their coded instructions. These secondary tasks may or may not also publish events, according to presets. In the example above, you could use a choreography approach so that each local e-commerce transaction publishes an event that triggers a local transaction in the credit service. Benefits of Choreography After having explained the term itself let us take a closer look at the benefits of using a choreographed pattern for a microservice architecture. The most important ones are outlined in the bulleted list below: Loose coupling: Choreography allows microservices to be loosely coupled, which means they can operate independently and asynchronously without depending on a central coordinator. This can make the system more scalable and resilient, as the failure of one microservice will not necessarily affect the other microservices. Ease of maintenance: Choreography allows microservices to be developed and maintained independently, which can make it easier to update and evolve the system. Decentralized control: Choreography allows control to be decentralized, which can make the system more resilient and less prone to failure. Asynchronous communication: Choreography allows microservices to communicate asynchronously, which can be more efficient and scalable than synchronous communication. Overall, choreography can be a useful design pattern for building scalable, resilient, and maintainable microservice architectures. Though some of these benefits can actually turn into drawbacks. 2. Orchestration An orchestration approach will perform transactions and publish events using an object to orchestrate the events, triggering other services to respond by completing their tasks. The orchestrator tells the participants what local transactions to execute. Saga is a complex design pattern that requires a high level of skill to successfully implement. However, the benefit of proper implementation is maintained data consistency across multiple services without tight coupling. Benefits of Orchestration Orchestration in microservice architectures can lead to some nice benefits which compensate for the drawbacks of a choreographed system. A few of them are explained below: Simplicity: Orchestration can be simpler to implement and maintain than choreography, as it relies on a central coordinator to manage and coordinate the interactions between the microservices. Centralized control: With a central coordinator, it is easier to monitor and manage the interactions between the microservices in an orchestrated system. Visibility: Orchestration allows for a holistic view of the system, as the central coordinator has visibility into all of the interactions between the microservices. Ease of troubleshooting: With a central coordinator, it is easier to troubleshoot issues in an orchestrated system. When to use Orchestration vs Choreography Whether you want to use choreography or orchestration in your microservice architecture should always be a well-thought-out choice. Both approaches bring their advantages but also downsides. API Gateway Pattern For large applications with multiple clients, implementing an API gateway pattern is a compelling option One of the largest benefits is that it insulates the client from needing to know how services have been partitioned. However, different teams will value the API gateway pattern for different reasons. One of these possible reasons is that it grants a single entry point for a group of microservices by working as a reverse proxy between client apps and the services. Another is that clients don’t need to know how services are partitioned, and service boundaries can evolve independently since the client knows nothing about them. The client also doesn’t need to know how to find or communicate with a multitude of ever-changing services. You can also create a gateway for specific types of clients (for example, backends for frontends) which improves ergonomics and reduces the number of roundtrips needed to fetch data. Plus, an API gateway pattern can take care of crucial tasks like authentication, SSL termination, and caching, which makes your app more secure and user-friendly. Another advantage is that the pattern insulates the client from needing to know how services have been partitioned. Before moving on to the next pattern, there’s one more benefit to cover: Security. The primary way the pattern improves security is by reducing the attack surface area. By providing a single entry point, the API endpoints aren’t directly exposed to clients, and authorization and SSL can be efficiently implemented. Developers can use this design pattern to decouple internal microservices from client apps so a partially failed request can be utilized. This ensures a whole request won’t fail because a single microservice is unresponsive. To do this, the encoded API gateway utilizes the cache to provide an empty response or return a valid error code. Circuit Breaker Design Pattern This pattern is usually applied between services that are communicating synchronously. A developer might decide to utilize the circuit breaker when a service is exhibiting high latency or is completely unresponsive. The utility here is that failure across multiple systems is prevented when a single microservice is unresponsive. Therefore, calls won’t be piling up and using the system resources, which could cause significant delays within the app or even a string of service failures. Implementing this pattern as a function in a circuit breaker design requires an object to be called to monitor failure conditions. When a failure condition is detected, the circuit breaker will trip. Once this has been tripped, all calls to the circuit breaker will result in an error and be directed to a different service. Alternatively, calls can result in a default error message being retrieved. There are three states of the circuit breaker pattern functions that developers should be aware of. These are: Open: A circuit breaker pattern is open when the number of failures has exceeded the threshold. When in this state, the microservice gives errors for the calls without executing the desired function. Closed: When a circuit breaker is closed, it’s in the default state and all calls are responded to normally. This is the ideal state developers want a circuit breaker microservice to remain in — in a perfect world, of course. Half-open: When a circuit breaker is checking for underlying problems, it remains in a half-open state. Some calls may be responded to normally, but some may not be. It depends on why the circuit breaker switched to this state initially. Command Query Responsibility Segregation (CQRS) A developer might use a command query responsibility segregation (CQRS) design pattern if they want a solution to traditional database issues like data contention risk. CQRS can also be used for situations when app performance and security are complex and objects are exposed to both reading and writing transactions. The way this works is that CQRS is responsible for either changing the state of the entity or returning the result in a transaction. Multiple views can be provided for query purposes, and the read side of the system can be optimized separately from the write side. This shift allows for a reduction in the complexity of all apps by separately querying models and commands so: The write side of the model handles persistence events and acts as a data source for the read side The read side of the model generates projections of the data, which are highly denormalized views Asynchronous Messaging If a service doesn’t need to wait for a response and can continue running its code post-failure, asynchronous messaging can be used. Using this design pattern, microservices can communicate in a way that’s fast and responsive. Sometimes this pattern is referred to as event-driven communication. To achieve the fastest, most responsive app, developers can use a message queue to maximize efficiency while minimizing response delays. This pattern can help connect multiple microservices without creating dependencies or tightly coupling them. While there are tradeoffs one makes with async communication (such as eventual consistency), it’s still a flexible, scalable approach to designing a microservices architecture. Event Sourcing The event-sourcing design pattern is used in microservices when a developer wants to capture all changes in an entity’s state. Using event stores like Kafka or alternatives will help keep track of event changes and can even function as a message broker. A message broker helps with the communication between different microservices, monitoring messages and ensuring communication is reliable and stable. To facilitate this function, the event sourcing pattern stores a series of state-changing events and can reconstruct the current state by replaying the occurrences of an entity. Using event sourcing is a viable option in microservices when transactions are critical to the application. This also works well when changes to the existing data layer codebase need to be avoided. Strangler-Fig Pattern Developers mostly use the strangler design pattern to incrementally transform a monolith application to microservices. This is accomplished by replacing old functionality with a new service — and, consequently, this is how the pattern receives its name. Once the new service is ready to be executed, the old service is “strangled” so the new one can take over. To accomplish this successful transfer from monolith to microservices, a facade interface is used by developers that allows them to expose individual services and functions. The targeted functions are broken free from the monolith so they can be “strangled” and replaced. Utilizing Design Patterns To Make Organization More Manageable Setting up the proper architecture and process tooling will help you create a successful microservice workflow. Use the design patterns described above and learn more about microservices in my blog to create a robust, functional app.

Origin of Cell-Based Architecture In the rapidly evolving domain of digital services, the need for scalable and resilient architectures (the ability of the system to recover from a failure quickly) has peaked. The introduction of cell-based architecture marks a pivotal shift tailored to meet the surging demands of hyper-scaling (architecture's ability for rapid scaling in response to fluctuating demand). This methodology, essential for rapid scaling in response to fluctuating demands, has become the foundation for digital success. It's a strategy that empowers tech behemoths like Amazon and Facebook, along with service platforms such as DoorDash, to skillfully navigate the tidal waves of digital traffic during peak moments and ensure service to millions of users worldwide without a hitch. Consider the surge Amazon faces on Prime Day or the global traffic spike Facebook navigates during significant events. Similarly, DoorDash's quest to flawlessly handle a flood of orders showcases a recurring theme: the critical need for an architecture that scales vertically and horizontally — expanding capacity without sacrificing system integrity or the user experience. In the current landscape, where startups frequently encounter unprecedented growth rates, the dream of scaling quickly can become a nightmare of scalability issues. Hypergrowth — a rapid expansion that surpasses expectations — presents a formidable challenge, risking a company's collapse if it fails to scale efficiently. This challenge birthed the concept of hyperscaling, emphasizing an architecture's nimbleness in adapting and growing to meet dynamic demands. Essential to this strategy is extensive parallelization and rigorous fault isolation, ensuring companies can scale without succumbing to the pitfalls of rapid growth. Cell-based architecture emerges as a beacon for applications and services where downtime is not an option. In scenarios where every second of inactivity spells significant reputational or financial loss, this architectural paradigm proves invaluable. It is especially crucial for: Applications requiring uninterrupted operation to ensure customer satisfaction and maintain business continuity. Financial services vital for maintaining economic stability. Ultra-scale systems where failure is an unthinkable option. Multi-tenant services requiring segregated resources for specific clients. This architectural innovation was developed in direct response to the increasing need for modern, rapidly expanding digital services. It provides a scalable, resilient framework supporting continuous service delivery and operational superiority. Understanding Cell-Based Architecture What Exactly Is Cell-Based Architecture? Cell-based architecture is a modern approach to creating digital services that are both scalable and resilient, taking cues from the principles of distributed systems and microservices design patterns. This architecture breaks down an extensive system into smaller, independent units called cells. Each cell is self-sufficient, containing a specific segment of the system's functionality, data storage, compute, application logic, and dependencies. This modular setup allows each cell to be scaled, deployed, and managed independently, enhancing the system's ability to grow and recover from failures without widespread impact. Drawing an analogy to urban planning, consider cell-based architecture akin to a well-designed metropolis where each neighborhood operates autonomously, equipped with its services and amenities, yet contributes to the city's overall prosperity. In times of disruption, such as a power outage or a water main break, only the affected neighborhood experiences downtime while the rest of the city thrives. Just as a single neighborhood can experience disruption without paralyzing the entire city, a cell encountering an issue in this architectural framework does not trigger a system-wide failure. This ensures the digital service remains robust and reliable, maintaining high uptime and resilience. Cell-based architecture builds scalable and robust digital services by breaking down an extensive system into smaller, independent units called cells. Each cell is self-contained with its own data storage and computing power similar to how neighborhoods work in a city. They operate independently, so if one cell has a problem, it doesn't affect the rest of the system. This design helps improve the system's stability and ability to grow without causing widespread issues. Fig. 1: Cell-Based Architecture Key Components Cell: Akin to neighborhoods, cells are the foundational building blocks of this architecture. Each cell is an autonomous microservice cluster with resources capable of handling a subset of service responsibilities. A cell is a stand-alone version of the application with its own computing power, load balancer, and databases. This setup allows each cell to operate independently, making it possible to deploy, monitor, and maintain them separately. This independence means that if one cell runs into problems, it doesn't affect the others, which helps the system to scale effectively and stay robust. Cell Router: Cell Routers play a critical role similar to a city's traffic management system. They dynamically route requests to the most appropriate cell based on factors such as load, geographic location, or specific service requirements. By efficiently balancing the load across various cells, cell routers ensure that each request is processed by the cell best suited to handle it, optimizing system performance and the user experience, much like how traffic lights and signs direct the flow of vehicles to ensure smooth transit within a city. Inter-Cell Communication Layer: Despite the autonomy of individual cells, cooperation between them is essential for handling tasks across the system. The Inter-Cell Communication Layer facilitates secure and efficient message exchange between cells. This layer acts as the public transportation system of our city analogy, connecting different neighborhoods (cells) to ensure seamless collaboration and unified service delivery across the entire architecture. It ensures that even as cells operate independently, they can still work together effectively, mirroring how different parts of a city are connected yet function cohesively. Control Plane: The control plane is a critical component of cell-based architecture, acting as the central hub for administrative operations. It oversees tasks such as setting up new cells (provisioning), shutting down existing cells (de-provisioning), and moving customers between cells (migrating). This ensures that the infrastructure remains responsive to the system's and its users' needs, allowing for dynamic resource allocation and seamless service continuity. Why and When to Use Cell-Based Architecture? Why Use It? Cell-based architecture offers a robust framework for efficiently scaling digital services, guaranteeing their resilience and adaptability during expansion. Below is a breakdown of its advantages: Higher Scalability: By defining and managing the capacity of each cell, you can add more cells to scale out (handle growth by adding more system components, such as databases and servers, and spreading the workload evenly). This avoids hitting the resource limits that come with scaling up (accommodating growth by increasing the size of a system's component, such as a database, server, or subsystem). As demand grows, you add more cells, each a contained unit with known capacities, making the system inherently scalable. Safer Deployments: Deployments and rollbacks are smoother with cells. You can deploy changes to one cell at a time, minimizing the impact of any issues. Canary cells can be used to test new deployments under actual conditions with minimal risk, providing a safety net for broader deployment. Easy Testability: Testing large, spread-out systems can be challenging, especially as they get bigger. However, with cell-based architecture, each cell is kept to a manageable size, making it much simpler to test how they behave at their largest capacity. Testing a whole big service can be too expensive and complex. However, testing just one cell is doable because you can simulate the most significant amount of work the cell can handle, similar to the most crucial job a single customer might give your application. This makes it practical and cost-effective to ensure each cell runs smoothly. Lower Blast Radius: Cell-based architecture limits the spread of failures by isolating issues within individual cells, much like neighborhoods in a city. This division ensures that a problem in one cell doesn't affect the entire system, maintaining overall functionality. Each cell operates independently, minimizing any single incident's impact area, or "blast radius," akin to the regional isolation seen in large-scale services. This setup enhances system resilience by keeping disruptions contained and preventing widespread outages.Fig. 2: Cell-based architecture services exhibit enhanced resilience to failures and feature a reduced blast radius compared to traditional services Improved Reliability and Recovery Higher Mean Time Between Failure (MTBF): Cell-based architecture increases the system's reliability by reducing how often problems occur. This design keeps each cell small and manageable, allowing for regular checks and maintenance, smoothing operations and making them more predictable. With customers distributed across different cells, any issues affect only a limited set of requests and users. Changes are tested on just a few cells at a time, making it easy to revert without widespread impact. For example, if you have customers divided across ten cells, a problem in one cell affects only 10% of your customers. This controlled approach to managing changes and addressing issues quickly means the system experiences fewer disruptions, leading to a more stable and reliable service. Lower Mean Time to Recovery (MTTR): Recovery is quicker and more straightforward with cells since you deal with a more minor, contained issue rather than a system-wide problem. Higher Availability: Cell-based architecture can lead to fewer and shorter failures, improving the overall uptime of your service. Even though there might be more potential points of failure (each cell could theoretically fail), the impact of each failure is significantly reduced, and they're easier to fix. When to Use It? Here's a brief guide to help you understand when it's advantageous to use this architectural strategy: High-Stakes Applications: If downtime could severely impact your customers, tarnish your reputation, or result in substantial financial loss, a cell-based approach can safeguard against widespread disruptions. Critical Economic Infrastructure: Cell-based architecture ensures continuous operation for financial services industries (FSI), where workloads are pivotal to economic stability. Ultra-Scale Systems: Systems too large or critical to fail—those that must maintain operation under almost any circumstance—are prime candidates for cell-based design. Stringent Recovery Objectives: Cell-based architecture offers quick recovery capabilities for workloads requiring a Recovery Point Objective (RPO) of less than 5 seconds and a Recovery Time Objective (RTO) of less than 30 seconds. Multi-Tenant Services with Dedicated Needs: For services where tenants demand fully dedicated resources, assigning them their cell ensures isolation and dedicated performance. Although cell-based architecture brings considerable benefits to handling critical workloads, it also comes with its own hurdles, such as heightened complexity, elevated costs, the necessity for specialized tools and practices, and the need for investment in a routing layer. For a more in-depth analysis of these challenges, please see the "Weighing the Scales: Benefits and Challenges." Implementing Cell-Based Architecture This section highlights critical design factors that come into play while designing and implementing a cell-based architecture. Designing a Cell Cell design is a foundational aspect of cell-based architecture, where a system is divided into smaller, self-contained units known as cells. Each cell operates independently with its resources, making the entire system more scalable and resilient. To embark on cell design, identify distinct functionalities within your system that can be isolated into individual cells. This might involve grouping services by their operational needs or user base. Once you've defined these boundaries, equip each cell with the necessary resources, such as databases and application logic, to ensure it can function autonomously. This setup facilitates targeted scaling and recovery and minimizes the impact of failures, as issues in one cell won't spill over to others. Implementing effective communication channels between cells and establishing comprehensive monitoring are crucial steps to maintain system cohesion and oversee cell performance. By systematically organizing your architecture into cells, you create a robust framework that enhances the manageability and adaptability of your system. Here are a few ideas on cell design that can be leveraged to bolster system resilience: Distribute Cells Across Availability Zones: By positioning cells across different availability zones (AZs), you can protect your system against the failure of a single data center or geographic location. This geographical distribution ensures that even if one AZ encounters issues, other cells in different AZs can continue to operate, maintaining overall system availability and reducing the risk of complete service downtime. Implement Redundant Cell Configurations: Creating redundant copies of cells within and across AZs can further enhance resilience. This redundancy means that if one cell fails, its responsibilities can be immediately taken over by a duplicate cell, minimizing service disruption. This approach requires careful synchronization between cells to ensure data consistency but significantly improves fault tolerance. Design Cells for Autonomous Operation: Ensuring that each cell can operate independently, with its own set of resources, databases, and application logic, is crucial. This independence allows cells to be isolated from failures elsewhere in the system. Even if one cell experiences a problem, it won't spread to others, localizing the impact and making it easier to identify and rectify issues. Use Load Balancers and Cell Routers Strategically: Integrating load balancers and cell routers that are aware of cell locations and health statuses can help efficiently redirect traffic away from troubled cells or AZs. This dynamic routing capability allows for real-time adjustments to traffic flow, directing users to the healthiest available cells and balancing the load to prevent overburdening any single cell or AZ. Facilitate Easy Cell Replication and Deployment: Design cells with replication and redeployment in mind. In case of a cell or AZ failure, having mechanisms for quickly spinning up new cells in alternative locations can be invaluable. Automation tools and templates for cell deployment can expedite this process, reducing recovery times and enhancing overall system resilience. Regularly Test Failover Processes: Regular testing of cell failover processes, including simulated failures and recovery drills, can ensure that your system responds as expected during actual outages. These tests can reveal potential weaknesses in your cell design and failover strategies, allowing for continuous improvement of system resilience. By incorporating these ideas into your cell design, you can create a more resilient system capable of withstanding various failure scenarios while minimizing the impact on service availability and performance. Cell Partitioning Cell partitioning is a crucial technique in cell-based architecture. It focuses on dividing a system's workload among distinct cells to optimize performance, scalability, and resilience. It involves categorizing and directing user requests or data to specific cells based on predefined criteria. This process ensures no cell becomes overwhelmed, enhancing system reliability and efficiency. How Cell Partitioning Can Be Done: Identify Partition Criteria: Determine the basis for distributing workloads among cells. Typical criteria include geographic location, user ID, request type, or date range. This step is pivotal in defining how the system categorizes and routes requests to the appropriate cells. Implement Routing Logic: Develop a routing mechanism within the cell router or API gateway that uses the identified criteria to direct incoming requests to the correct cell. This might involve dynamic decision-making algorithms that consider current cell load and availability. Continuous Monitoring and Adjustment: Regularly monitor the performance and load distribution across cells. Use this data to adjust partitioning criteria and routing logic to maintain optimal system performance and scalability. Partitioning Algorithms: Several algorithms can be utilized for effective cell partitioning, each with its strengths and tailored to different types of workloads and system requirements: Consistent Hashing: Requests are distributed based on the hash values of the partition key (e.g., user ID), ensuring even workload distribution and minimal reorganization when cells are added or removed. Range-Based Partitioning: Divides data into ranges (e.g., alphabetical or numerical) and assigns each range to a specific cell. This is ideal for ordered data, allowing efficient query operations. Round Robin: This method distributes requests evenly across all available cells in a cyclic manner. It is straightforward and helpful in achieving a basic level of load balancing. Sharding: Similar to range-based partitioning but more complex, sharding involves splitting large databases into smaller, faster, more easily managed parts, or "shards," each handled by a separate cell. Dynamic Partitioning: Adjusts partitioning in real-time based on workload characteristics or system performance metrics. This approach requires advanced algorithms capable of analyzing system states and making immediate adjustments. By thoughtfully implementing cell partitioning and choosing the appropriate algorithm, you can significantly enhance your cell-based architecture's performance, scalability, and resilience. Regular review and adjustment of your partitioning strategy ensures it continues to meet your system's evolving needs. Implementing a Cell Router In cell-based architecture, the cell router is crucial for steering traffic to the correct cells, ensuring efficient workload management and scalability. An effective cell router hinges on two key elements: traffic routing logic and failover strategies, which maintain system reliability and optimize performance. Implementing Traffic Routing Logic: Start by defining the criteria for how requests are directed to various cells, including the users' geographic location, the type of request, and the specific services needed. The aim is to reduce latency and evenly distribute the load. Employ dynamic routing that adapts to cell availability and workload changes in real time, possibly through integration with a service discovery tool that monitors each cell's status and location. Establishing Failover Strategies: Solid failover processes are essential for the cell router to ensure the system's dependability. Should any cell become unreachable, the router must automatically reroute traffic to the next available cell, requiring minimal manual intervention. This is achieved by implementing health checks across cells to swiftly identify and respond to failures, thus keeping the user experience smooth and the service highly available, even during cell outages. Fig 3. The cell router ensures a smooth user experience by redirecting traffic to healthy cells during outages, maintaining uninterrupted service availability For the practical implementation of a cell router, you can take one of the following approaches: Load Balancers: Use cloud-based load balancers that dynamically direct traffic based on specific request attributes, such as URL paths or headers, according to set rules. API Gateways: An API gateway can serve as the primary entry for all incoming requests and route them to the appropriate cell based on configured logic. Service Mesh: A service mesh offers a network layer that facilitates efficient service-to-service communications and routing requests based on policies, service discovery, and health status. Custom Router Service: Developing a custom service allows routing decisions based on detailed request content, current cell load, or bespoke business logic, offering tailored control over traffic management. Choosing the right implementation strategy for a cell router depends on specific needs, such as the granularity of routing decisions, integration capabilities with existing systems, and management simplicity. Each method provides varying degrees of control, complexity, and adaptability to cater to distinct architectural requirements. Cell Sizing Cell sizing in a cell-based architecture refers to determining each cell's optimal size and capacity to ensure it can handle its designated workload effectively without overburdening. Proper cell sizing is crucial for several reasons: Balanced Load Distribution: Correctly sized cells help achieve a balanced distribution of workloads across the system, preventing any single cell from becoming a bottleneck. Scalability: Well-sized cells can scale more efficiently. As demand increases, the system can add more cells or adjust resources within existing cells to accommodate growth. Resilience and Recovery: Smaller, well-defined cells can isolate failures more effectively, limiting the impact of any single point of failure. This makes the system more resilient and simplifies recovery processes. Cost Efficiency: Optimizing cell size helps utilize resources more efficiently, avoiding unnecessary expenditure on underutilized capacities. How Cell Sizing Is Done? Cell sizing involves a careful analysis of several factors: Workload Analysis: Understand the nature and volume of each cell's workload. This includes peak demand times, data throughput, and processing requirements. Resource Requirements: Based on the workload analysis, estimate the resources (CPU, memory, storage) each cell needs to operate effectively under various conditions. Performance Metrics: Consider key performance indicators (KPIs) that define successful cell operation. This could include response times, error rates, and throughput. Scalability Goals: Define how the system should scale in response to increased demand. This will influence whether cells should be designed to scale up (increase resources in a cell) or scale out (add more cells). Testing and Adjustment: Validate cell size assumptions by testing under simulated workload conditions. Monitoring real-world performance and adjusting as needed is a continuous part of cell sizing. Effective cell sizing often involves a combination of theoretical analysis and empirical testing. Starting with a best-guess estimate based on workload characteristics and adjusting based on observed performance ensures that cells remain efficient, responsive, and cost-effective as the system evolves. Cell Deployment Cell deployment in a cell-based architecture is the process of distributing and managing your application's workload across multiple self-contained units called cells. This strategy ensures scalability, resilience, and efficient resource use. Here's a concise guide on how it's typically done and the technology choices available for effective implementation. How Is Cell Deployment Done? Automated Deployment Pipelines: Start by setting up automated deployment pipelines. These pipelines handle your application's packaging, testing, and deployment to various cells. Automation ensures consistency, reduces errors, and enables rapid deployment across cells. Blue/Green Deployments: Use blue/green deployment strategies to minimize downtime and reduce risk. By deploying the new version of your application to a separate environment (green) while keeping the current version (blue) running, you can switch traffic to the latest version once it's fully ready and tested. Canary Releases: Gradually roll out updates to a small subset of cells or users before making them available system-wide. This allows you to monitor the impact of changes and roll them back if necessary without affecting all users. Technology Choices for Cell Deployment: Container Orchestration Tools: Tools such as Kubernetes, AWS ECS, and Docker Swarm are crucial for orchestrating cell deployments, enabling the encapsulation of applications into containers for streamlined deployment, scaling, and management across various cells. CI/CD Tools: Continuous Integration and Continuous Deployment (CI/CD) tools such as Jenkins, GitLab CI, CircleCI, and AWS Pipeline facilitate the automation of testing and deployment processes, ensuring that new code changes can be efficiently rolled out. Infrastructure as Code (IaC): Tools like Terraform and AWS CloudFormation allow you to define your infrastructure in code, making it easier to replicate and deploy cells across different environments or cloud providers. Service Meshes: Service meshes like Istio or Linkerd provide advanced traffic management capabilities, including canary deployments and service discovery, which are crucial for managing communication and cell updates. By leveraging these deployment strategies and technologies, you can achieve a high degree of automation and control in your cell deployments, ensuring your application remains scalable, reliable, and easy to manage. Cell Observability Cell observability is crucial in a cell-based architecture to ensure you have comprehensive visibility into each cell's health, performance, and operational metrics. It allows you to monitor, troubleshoot, and optimize the system effectively, enhancing overall reliability and user experience. Implementing Cell Observability: To achieve thorough cell observability, focus on three key areas: logging, monitoring, and tracing. Logging captures detailed events and operations within each cell. Monitoring tracks key performance indicators and health metrics in real time. Tracing follows requests as they move through the cells, identifying bottlenecks or failures in the workflow. Technology Choices for Cell Observability: Logging Tools: Solutions like Elasticsearch, Logstash, Kibana (ELK Stack), or Splunk provide powerful logging capabilities, allowing you to aggregate and analyze logs from all cells centrally. Monitoring Solutions: Prometheus, coupled with Grafana for visualization, offers robust monitoring capabilities with support for custom metrics. Cloud-native services like Amazon CloudWatch or Google Operations (formerly Stackdriver) provide integrated monitoring solutions tailored for applications deployed on their respective cloud platforms. Distributed Tracing Systems: Tools like Jaeger, Zipkin, and AWS XRay enable distributed tracing, helping you to understand the flow of requests across cells and identify latency issues or failures in microservices interactions. Service Meshes: Service meshes such as Istio or Linkerd inherently offer observability features, including monitoring, logging, and tracing requests between cells without requiring changes to your application code. By leveraging these tools and focusing on comprehensive observability, you can ensure that your cell-based architecture remains performant, resilient, and capable of supporting your application's dynamic needs. Weighing the Scales: Benefits and Challenges Adopting Cell-Based Architecture transforms the structural and operational dynamics of digital services. Breaking down a service into independently scalable and resilient units (cells) offers a robust framework for managing complexity and ensuring system availability. However, this architectural paradigm also introduces new challenges and complexities. Here's a deeper dive into the technical advantages and considerations. Benefits Horizontal Scalability: Unlike traditional scale-up approaches, Cell-Based Architecture enables horizontal scaling by adding more cells. This method alleviates common bottlenecks associated with centralized databases or shared resources, allowing for linear scalability as user demand increases. Fault Isolation and Resilience: The architecture's compartmentalized design ensures that failures are contained within individual cells, significantly reducing the system's overall blast radius. This isolation enhances the system's resilience, as issues in one cell can be mitigated or repaired without impacting the entire service. Deployment Agility: Leveraging cells allows for incremental deployments and feature rollouts, akin to implementing rolling updates across microservices. This granularity in deployment strategy minimizes downtime and enables a more flexible response to market or user demands. Simplified Operational Complexity: While the initial setup is complex, the ongoing operation and management of cells can be more straightforward than monolithic architectures. Each cell's autonomy simplifies monitoring, troubleshooting, and scaling efforts, as operational tasks can be executed in parallel across cells. Challenges (Considerations) Architectural Complexity: Transitioning to or implementing Cell-Based Architecture demands a meticulous design phase, focusing on defining cell boundaries, data partitioning strategies, and inter-cell communication protocols. This complexity requires a deep understanding of distributed systems principles and may necessitate a development and operational practices shift. Resource and Infrastructure Overhead (Higher Cost): Each cell operates with its set of resources and infrastructure, potentially leading to increased overhead compared to shared-resource models. Optimizing resource utilization and cost-efficiency becomes paramount, especially as the number of cells grows. Inter-Cell Communication Management: Ensuring coherent and efficient communication between cells without introducing tight coupling or significant latency is a critical challenge. Developers must design a communication layer that supports the necessary interactions while maintaining cells' independence and avoiding performance bottlenecks. Data Consistency and Synchronization: Maintaining data consistency across cells, especially in scenarios requiring global state or real-time data synchronization, adds another layer of complexity. Implementing strategies like event sourcing, CQRS (Command Query Responsibility Segregation), or distributed sagas may be necessary to address these challenges. Specialized Tools and Practices: Operating a cell-based architecture requires specialized operational tools and practices to effectively manage multiple instances of workloads. Routing Layer Investment: A robust cell routing layer is essential for directing traffic appropriately across cells, necessitating additional investment in technology and expertise. Navigating the Trade-offs Opting for Cell-Based Architecture involves navigating these trade-offs and evaluating whether scalability, resilience, and operational agility benefits outweigh the complexities of implementation and management. It is most suitable for services requiring high availability, those undergoing rapid expansion, or systems where modular scaling and failure isolation are critical. Best Practices and Pitfalls Best Practices Adopting a cell-based architecture can significantly enhance the scalability and resilience of your applications. Here are streamlined best practices for implementing this approach effectively: Begin With a Solid Foundation Treat Your Current Setup as Cell Zero: Viewing your existing system as the initial cell, gradually introducing traffic routing and distribution across new cells. Launch with Multiple Cells: Implement more than one cell from the beginning to quickly learn and adapt to the operational dynamics of a cell-based environment. Plan for Flexibility and Growth Implement a Cell Migration Mechanism Early: Prepare for the need to move customers between cells, ensuring you can scale and adjust without disruption. Focus on Reliability Conduct a Failure Mode Analysis: Identify and assess potential failures within each cell and their impact, developing strategies to ensure robustness and minimize cross-cell effects. Ensure Independence and Security Maintain Cell Autonomy: Design cells to be self-sufficient, with dedicated resources and clear ownership, possibly by a single team. Secure Communication: Use versioned, well-defined APIs for cell interactions and enforce security policies at the API gateway level. Minimize Dependencies: Keep inter-cell dependencies low to preserve the architecture's benefits, such as fault isolation. Optimize Deployment and Operations Avoid Shared Resources: Each cell should have its data storage to eliminate global state dependencies. Deploy in Waves: Introduce updates and deployments in phases across cells for better change management and quick rollback capabilities. By following these practices, you can leverage cell-based architecture to create scalable, resilient, but also manageable, and secure systems ready to meet the challenges of modern digital demands. Common Pitfalls While cell-based architecture offers significant advantages for scalability and resilience, it also introduces specific challenges and pitfalls that organizations need to be aware of when adopting this approach: Complexity in Management and Operations Increased Operational Overhead: Managing multiple cells can introduce complexity in deployment, monitoring, and operations, requiring robust automation and orchestration tools to maintain efficiency. Consistency Management: Ensuring data consistency across cells, especially in stateful applications, can be challenging and might require sophisticated synchronization mechanisms. Initial Setup and Migration Challenges Complex Migration Process: Transitioning to a cell-based architecture from a traditional setup can be complex, requiring careful planning to avoid service disruption and data loss. Steep Learning Curve: Teams may face a learning curve in understanding cell-based concepts and best practices, necessitating training and potentially slowing initial progress. Design and Architectural Considerations Cell Isolation: Properly isolating cells to prevent failure propagation requires meticulous design, failing which the system might not fully realize the benefits of fault isolation. Optimal Cell Size: Determining the optimal size for cells can be tricky, as overly small cells may lead to inefficiencies, while huge cells might compromise scalability and resilience. Resource Utilization and Cost Implications Potential for Increased Costs: If not carefully managed, the duplication of resources across cells can lead to increased operational costs. Underutilization of Resources: Balancing resource allocation to prevent underutilization while avoiding over-provisioning requires continuous monitoring and adjustment. Networking and Communication Overhead Network Complexity: The cell-based architecture may introduce additional network complexity, including the need for sophisticated routing and load-balancing strategies. Inter-Cell Communication: Ensuring efficient and secure communication between cells, especially in geographically distributed setups, can introduce latency and requires safe, reliable networking solutions. Security and Compliance Security Configuration: Each cell's need for individual security configurations can complicate enforcing consistent security policies across the architecture. Compliance Verification: Verifying that each cell complies with regulatory requirements can be more challenging in a distributed architecture, requiring robust auditing mechanisms. Scalability vs. Cohesion Trade-Off Dependency Management: While minimizing dependencies between cells enhances fault tolerance, it can also lead to challenges in maintaining application cohesion and consistency. Data Duplication: Avoiding shared resources may result in data duplication and synchronization challenges, impacting system performance and consistency. Organizations should invest in robust planning, adopt comprehensive automation and monitoring tools, and ensure ongoing team training to mitigate these pitfalls. Understanding these challenges upfront can help design a more resilient, scalable, and efficient cell-based architecture. Cell-Based Wins in the Real World Cell-based architecture has become essential for managing scalability and ensuring system resilience, from high-growth startups to tech giants like Amazon and Facebook. This architectural model has been adopted across various industries, reflecting its effectiveness in handling large-scale, critical workloads. Here's a brief look at how DoorDash and Slack have implemented cell-based architecture to address their unique challenges. DoorDash's Transition to Cell-Based Architecture Faced with the demands of hypergrowth, DoorDash migrated from a monolithic system to a cell-based architecture, marking a pivotal shift in its operational strategy. This transition, known as Project SuperCell, was driven by the need to efficiently manage fluctuating demand and maintain consistent service reliability across diverse markets. By leveraging AWS's cloud infrastructure, DoorDash was able to isolate failures within individual cells, preventing widespread system disruptions. It significantly enhanced their ability to scale resources and maintain service reliability, even during peak times, demonstrating the transformative potential of adopting a cell-based approach. Slack's Migration to Cell-Based Architecture Slack underwent a major shift to a cell-based architecture to lessen the impact of gray failures and boost service redundancy. Prompted by a review of a network outage, this move revealed the risks of depending solely on a single availability zone. The new cellular structure aims to confine failures more effectively and minimize the extent of potential site outages. With the adoption of isolated services in each availability zone, Slack has enabled its internal services to function independently within each zone, curtailing the fallout from outages and speeding up the recovery process. This significant redesign has markedly improved Slack's system resilience, underscoring cell-based architecture's role in ensuring high service availability and quality. Roblox's Strategic Shift to Cellular Infrastructure Roblox's shift to a cell-based architecture showcases its response to rapid growth and the need to support over 70 million daily active users with reliable, low-latency experiences. Roblox created isolated clusters within their data centers by adopting a cellular infrastructure, enhancing system resilience through service replication across cells. This setup allowed for the deactivation of non-functional cells without disrupting service, effectively containing failures. The move to cellular infrastructure has significantly boosted Roblox's system reliability, enabling the platform to offer always-on, immersive experiences worldwide. This strategy highlights the effectiveness of cell-based architecture in managing large-scale, dynamic workloads and maintaining high service quality as platforms expand. These examples from DoorDash, Slack, and Roblox illustrate the strategic value of cell-based architecture in addressing the challenges of scale and reliability. By isolating workloads into independent cells, these companies have achieved greater scalability, fault tolerance, and operational efficiency, showcasing the effectiveness of this approach in supporting dynamic, high-demand services. Key Takeaways Cell-based architecture represents a transformative approach for organizations aiming to achieve hyper-scalability and resilience in the digital era. Companies like Amazon, Facebook, DoorDash, and Slack have demonstrated their efficacy in managing hypergrowth and ensuring uninterrupted service by segmenting systems into independent, self-sufficient cells. This architectural strategy facilitates dynamic scaling and robust fault isolation and demands careful consideration of increased complexity, resource allocation, and the need for specialized operational tools. As businesses continue to navigate the demands of digital growth, the adoption of cell-based architecture emerges as a strategic solution for sustaining operational integrity and delivering consistent user experiences amidst the ever-evolving digital landscape. Acknowledgments This article draws upon the collective knowledge and experiences of industry leaders and practitioners, including insights from technical blogs, case studies from companies like Amazon, Slack, and Doordash, and contributions from the wider tech community. References https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/reducing-scope-of-impact-with-cell-based-architecture.html https://github.com/wso2/reference-architecture/blob/master/reference-architecture-cell-based.md https://newsletter.systemdesign.one/p/cell-based-architecture https://highscalability.com/cell-architectures/ https://www.youtube.com/watch?v=ReRrhU-yRjg https://slack.engineering/slacks-migration-to-a-cellular-architecture/ https://blog.roblox.com/2023/12/making-robloxs-infrastructure-efficient-resilient/

By Shantanu Kumar CORE

Services, or servers, are software components or processes that execute operations on specified inputs, producing either actions or data depending on their purpose. The party making the request is the client, while the server manages the request process. Typically, communication between client and server occurs over a network, utilizing protocols such as HTTP for REST or gRPC. Services may include a User Interface (UI) or function solely as backend processes. With this background, we can explore the steps and rationale behind developing a scalable service. NOTE: This article does not provide instructions on service or UI development, leaving you the freedom to select the language or tech stack that suits your requirements. Instead, it offers a comprehensive perspective on constructing and expanding a service, reflecting what startups need to do in order to scale a service. Additionally, it's important to recognize that while this approach offers valuable insights into computing concepts, it's not the sole method for designing systems. The Beginning: Version Control Assuming clarity on the presence of a UI and the general purpose of the service, the initial step prior to service development involves implementing a source control/version control system to support the code. This typically entails utilizing tools like Git, Mercurial, or others to back up the code and facilitate collaboration, especially as the number of contributors grows. It's common for startups to begin with Git as their version control system, often leveraging platforms like github.com for hosting Git repositories. An essential element of version control is pull requests, facilitating peer reviews within your team. This process enhances code quality by allowing multiple individuals to review and approve proposed changes before integration. While I won't delve into specifics here, a quick online search will provide ample information on the topic. Developing the Service Once version control is established, the next step involves setting up a repository and initiating service development. This article adopts a language-agnostic approach, as delving into specific languages and optimal tech stacks for every service function would be overly detailed. For conciseness, let's focus on a service that executes functions based on inputs and necessitates backend storage (while remaining neutral on the storage solution, which will be discussed later). As you commence service development, it's crucial to grasp how to run it locally on your laptop or in any developer environment. One should consider this aspect carefully, as local testing plays a pivotal role in efficient development. While crafting the service, ensure that classes, functions, and other components are organized in a modular manner, into separate files as necessary. This organizational approach promotes a structured repository and facilitates comprehensive unit test coverage. Unit tests represent a critical aspect of testing that developers should rigorously prioritize. There should be no compromises in this regard! Countless incidents or production issues could have been averted with the implementation of a few unit tests. Neglecting this practice can potentially incur significant financial costs for a company. I won't delve into the specifics of integrating the gRPC framework, REST packages, or any other communication protocols. You'll have the freedom to explore and implement these as you develop the service. Once the service is executable and tested through unit tests and basic manual testing, the next step is to explore how to make it "deployable." Packaging the Service Ensuring the service is "deployable" implies having a method to run the process in a more manageable manner. Let's delve into this concept further. What exactly does this entail? Now that we have a runnable process, who will initiate it initially? Moreover, where will it be executed? Addressing these questions is crucial, and we'll now proceed to provide answers. In my humble opinion, managing your own compute infrastructure might not be the best approach. There are numerous intricacies involved in ensuring that your service is accessible on the Internet. Opting for a cloud service provider (CSP) is a wiser choice, as they handle much of the complexity behind the scenes. For our purposes, any available cloud service provider will suffice. Once a CSP is selected, the next consideration is how to manage the process. We aim to avoid manual intervention every time the service crashes, especially without notification. The solution lies in orchestrating our process through containerization. This involves creating a container image for our process, essentially a filesystem containing all necessary dependencies at the application layer. A "Dockerfile" is used to specify the steps for including the process and dependencies in the container image. Upon completion of the Dockerfile, the docker build cli can be used to generate an image with tags. This image is then stored locally or pushed to a container registry, serving as a repository for container images that can later be pulled onto a compute instance. With these steps outlined, the next question arises: how does containerization orchestrate our process? This will be addressed in the following section on executing a container. Executing the Container After building a container image, the subsequent step is its execution, which in turn initiates the service we've developed. Various container runtimes, such as containerd, podman, and others, are available to facilitate this process. In this context, we utilize the "docker" cli to manage the container, which interacts with containerd in the background. Running a container is straightforward: "docker run" executes the container and consequently, the developed process. You may observe logs in the terminal (if not run as a daemon) or use "docker logs" to inspect service logs if necessary. Additionally, options like "--restart" can be included in the command to automatically restart the container (i.e., the process) in the event of a crash, allowing for customization as required. At this stage, we have our process encapsulated within a container, ready for execution/orchestration as required. While this setup is suitable for local testing, our next step involves exploring how to deploy this on a basic compute instance within our chosen CSP. Deploying the Container Now that we have a container, it's advisable to publish it to a container registry. Numerous container registries are available, managed by CSPs or docker itself. Once the container is published, it becomes easily accessible from any CSP or platform. We can pull the image and run it on a compute instance, such as a Virtual Machine (VM), allocated within the CSP. Starting with this option is typically the most cost-effective and straightforward. While we briefly touch on other forms of compute infrastructure later in this article, deploying on a VM involves pulling a container image and running it, much like we did in our developer environment. Voila! Our service is deployed. However, ensuring accessibility to the world requires careful consideration. While directly exposing the VM's IP to the external world may seem tempting, it poses security risks. Implementing TLS for security is crucial. Instead, a better approach involves using a reverse proxy to route requests to specific services. This ensures security and facilitates the deployment of multiple services on the same VM. To enable internet access to our service, we require a method for inbound traffic to reach our VM. An effective solution involves installing a reverse proxy like Nginx directly on the VM. This can be achieved by pulling the Nginx container image, typically labeled as "nginx:latest". Before launching the container, it's necessary to configure Nginx settings such as servers, locations, and additional configurations. Security measures like TLS can also be implemented for enhanced protection. Once the Nginx configuration is established, it can be exposed to the container through volumes during container execution. This setup allows the reverse proxy to effectively route incoming requests to the container running on the same VM, using a specified port. One notable advantage is the ability to host multiple services within the VM, with routing efficiently managed by the reverse proxy. To finalize the setup, we must expose the VM's IP address and proxy port to the internet, with TLS encryption supported by the reverse proxy. This configuration adjustment can typically be configured through the CSP's settings. NOTE: The examples of solutions provided below may reference GCP as the CSP. This is solely for illustrative purposes and should not be interpreted as a recommendation. The intention is solely to convey concepts effectively. Consider the scenario where managing a single VM manually becomes laborious and lacks scalability. To address this challenge, CSPs offer solutions akin to managed instance groups, comprising multiple VMs configured identically. These groups often come with features like startup scripts, which execute upon VM initialization. All the configurations discussed earlier can be scripted into these startup scripts, simplifying the process of VM launch and enhancing scalability. This setup proves beneficial when multiple VMs are required to handle requests efficiently. Now, the question arises: when dealing with multiple VMs, how do we decide where to route requests? The solution is to employ a load balancer provided by the CSP. This load balancer selects one VM from the pool to handle each request. Additionally, we can streamline the process by implementing general load balancing. To remove individual reverse proxies, we can utilize multiple instance groups for every service needed, accompanied by load balancers for each. The general load balancer can expose its IP with TLS configuration and route setup, ensuring that only service containers run on the VM. It's essential to ensure that VM IPs and ports are accessible solely by the load balancer in the ingress path, a task achievable through configurations provided by the CSP. At this juncture, we have a load balancer securely managing requests, directing them to the specific container service within a VM from a pool of VMs. This setup itself contributes to scaling our service. To further enhance scalability and eliminate the need for continuous VM operation, we can opt for an autoscaler policy. This policy dynamically scales the VM group up or down based on parameters such as CPU, memory, or others provided by the CSP. Now, let's delve into the concept of Infrastructure as Code (IaC), which holds significant importance in efficiently managing CSP components that promote scale. Essentially, IaC involves managing CSP infrastructure components through configuration files, interpreted by an IaC tool (like Terraform) to manage CSP infrastructure accordingly. For more detailed information, refer to the wiki. Datastore We've previously discussed scaling our service, but it's crucial to remember that there's typically a requirement to maintain a state somewhere. This is where databases or datastores play a pivotal role. From experience, handling this aspect can be quite tricky, and I would once again advise against developing a custom solution. CSP solutions are ideally suited for this purpose. CSPs generally handle the complexity associated with managing databases, addressing concepts such as master-slave architecture, replica management, synchronous-asynchronous replication, backups/restores, consistency, and other intricate aspects more effectively. Managing a database can be challenging due to concerns about data loss arising from improper configurations. Each CSP offers different database offerings, and it's essential to consider the specific use cases the service deals with to choose the appropriate offering. For instance, one may need to decide between using a relational database offering versus a NoSQL offering. This article does not delve into these differences. The database should be accessible from the VM group and serve as a central datastore for all instances where the state is shared. It's worth noting that the database or datastore should only be accessible within the VPC, and ideally, only from the VM group. This is crucial to prevent exposing the ingress IP for the database, ensuring security and data integrity. Queues In service design, we often encounter scenarios where certain tasks need to be performed asynchronously. This means that upon receiving a request, part of the processing can be deferred to a later time without blocking the response to the client. One common approach is to utilize databases as queues, where requests are ordered by some identifier. Alternatively, CSP services such as Amazon SQS or GCP pub/sub can be employed for this purpose. Messages published to the queue can then be retrieved for processing by a separate service that listens to the queue. However, we won't delve into the specifics here. Monitoring In addition to the VM-level monitoring typically provided by the CSP, there may be a need for more granular insights through service-level monitoring. For instance, one might require latency metrics for database requests, metrics based on queue interactions, or metrics for service CPU and memory utilization. These metrics should be collected and forwarded to a monitoring solution such as Datadog, Prometheus, or others. These solutions are typically backed by a time-series database (TSDB), allowing users to gain insights into the system's state over a specific period of time. This monitoring setup also facilitates debugging certain types of issues and can trigger alerts or alarms if configured to do so. Alternatively, you can set up your own Prometheus deployment, as it is an open-source solution. With the aforementioned concepts, it should be feasible to deploy a scalable service. This level of scalability has proven sufficient for numerous startups that I have provided consultation for. Moving forward, we'll explore the utilization of a "container orchestrator" instead of deploying containers in VMs, as described earlier. In this article, we'll use Kubernetes (k8s) as an example to illustrate this transition. Container Orchestration: Enter Kubernetes (K8s) Having implemented the aforementioned design, we can effectively manage numerous requests to our service. Now, our objective is to achieve decoupling to further enhance scalability. This decoupling is crucial because a bug in any service within a VM could lead to the VM crashing, potentially causing the entire ecosystem to fail. Moreover, decoupled services can be scaled independently. For instance, one service may have sufficient scalability and effectively handle requests, while another may struggle with the load. Consider the example of a shopping website where the catalog may receive significantly more visits than the checkout page. Consequently, the scale of read requests may far exceed that of checkouts. In such cases, deploying multiple service containers into Kubernetes (K8s) as distinct services allows for independent scaling. Before delving into specifics, it's worth noting that CSPs offer Kubernetes as a compute platform option, which is essential for scaling to the next level. Kubernetes (K8s) We won't delve into the intricacies of Kubernetes controllers or other aspects in this article. The information provided here will suffice to deploy a service on Kubernetes. Kubernetes (K8s) serves as an abstraction over a cluster of nodes with storage and compute resources. Depending on where the service is scheduled, the node provides the necessary compute and storage capabilities. Having container images is essential for deploying a service on Kubernetes (K8s). Resources in K8s are represented by creating configurations, which can be in YAML or JSON format, and they define specific K8s objects. These objects belong to a particular "namespace" within the K8s cluster. The basic unit of compute within K8s is a "Pod," which can run one or more containers. Therefore, a config for a pod can be created, and the service can then be deployed onto a namespace using the K8s CLI, kubectl. Once the pod is created, your service is essentially running, and you can monitor its state using kubectl with the namespace as a parameter. To deploy multiple pods, a "deployment" is required. Kubernetes (K8s) offers various resources such as deployments, stateful sets, and daemon sets. The K8s documentation provides sufficient explanations for these abstractions, we won't discuss each of them here. A deployment is essentially a resource designed to deploy multiple pods of a similar kind. This is achieved through the "replicas" option in the configuration, and you can also choose an update strategy according to your requirements. Selecting the appropriate update strategy is crucial to ensure there is no downtime during updates. Therefore, in our scenario, we would utilize a deployment for our service that scales to multiple pods. When employing a Deployment to oversee your application, Pods can be dynamically generated and terminated. Consequently, the count and identities of operational and healthy Pods may vary unpredictably. Kubernetes manages the creation and removal of Pods to sustain the desired state of your cluster, treating Pods as transient resources with no assured reliability or durability. Each Pod is assigned its own IP address, typically managed by network plugins in Kubernetes. As a result, the set of Pods linked with a Deployment can fluctuate over time, presenting a challenge for components within the cluster to consistently locate and communicate with specific Pods. This challenge is mitigated by employing a Service resource. After establishing a service object, the subsequent topic of discussion is Ingress. Ingress is responsible for routing to multiple services within the cluster. It facilitates the exposure of HTTP, HTTPS, or even gRPC routes from outside the cluster to services within it. Traffic routing is managed by rules specified on the Ingress resource, which is supported by a load balancer operating in the background. With all these components deployed, our service has attained a commendable level of scalability. It's worth noting that the concepts discussed prior to entering the Kubernetes realm are mirrored here in a way — we have load balancers, containers, and routes, albeit implemented differently. Additionally, there are other objects such as Horizontal Pod Autoscaler (HPA) for scaling pods based on memory/CPU utilization, and storage constructs like Persistent volumes (PV) or Persistent Volume Claims (PVC), which we won't delve into extensively. Feel free to explore these for a deeper understanding. CI/CD Lastly, I'd like to address an important aspect of enhancing developer efficiency: Continuous Integration/Deployment (CI/CD). Continuous Integration (CI) involves running automated tests (such as unit, end-to-end, or integration tests) on any developer pull request or check-in to the version control system, typically before merging. This helps identify regressions and bugs early in the development process. After merging, CI generates images and other artifacts required for service deployment. Tools like Jenkins (Jenkins X), Tekton, Git actions and others facilitate CI processes. Continuous Deployment (CD) automates the deployment process, staging different environments for deployment, such as development, staging, or production. Usually, the development environment is deployed first, followed by running several end-to-end tests to identify any issues. If everything functions correctly, CD proceeds to deploy to other environments. All the aforementioned tools also support CD functionalities. CI/CD tools significantly improve developer efficiency by reducing manual work. They are essential to ensure developers don't spend hours on manual tasks. Additionally, during manual deployments, it's crucial to ensure no one else is deploying to the same environment simultaneously to avoid conflicts, a concern that can be addressed effectively by our CD framework. There are other aspects like dynamic config management and securely storing secrets/passwords and logging system, though we won't delve into details, I would encourage readers to look into the links provided. Thank you for reading!