As the year goes by, I feel it’s getting harder and harder to define who I am.
Author: Jayden
-
Notes on gRPC Best Practices
gRPC has become a standard for service-to-service communication. However, developers also need to pay attention to governance, such as managing proto files, handling cross-project dependencies, generating SDKs for different languages, and version control across production, development, and temporary branches. Without this, projects can quickly become disorganized and increase development time costs.
In the past, we had to maintain different compilers for each language in the proto project, including various Dockerfiles, which was both troublesome and error-prone. In recent years, I found that https://buf.build/ has done a great job. With the tools it provides, we can easily compile versions for multiple languages and push proto files to the service as the single source of truth for downstream services.
References
-
Distributed Design of MQTT Broker
P2P Implementation: https://github.com/forgoes/bifrost
Abstract—The Message Queuing Telemetry Transport (MQTT [1]) protocol is a lightweight, publish-subscribe network protocol designed for constrained devices and low-bandwidth, high-latency, or unreliable networks. An MQTT broker is a central component in the Message Queuing Telemetry Transport (MQTT) protocol, responsible for managing and facilitating communication between clients in a publish-subscribe architecture. However, designing a distributed MQTT broker to handle massive connections involves addressing a wide range of challenges related to scalability, reliability, latency, consistency, security, resource management, and operational complexity. Each of these aspects requires careful planning, robust architectural solutions, and advanced algorithms to ensure the system meets the demands of large-scale IoT applications. The proposed design is structured into distinct layers, incorporating the Raft consensus protocol [2] and consistent hashing, offers a robust solution for distributed MQTT brokers.
Keywords—MQTT, broker, distributed, cluster, cloud, consistent hash, LSM tree
Introduction
As the number of IoT devices continues to grow exponentially, a single-server MQTT broker faces inherent limitations in CPU, memory, and network capacity, making it inadequate for handling massive connections and high message throughput. One approach is a peer-to-peer architecture, within which packets must be routed through multiple nodes before reaching their destination, leading to increased latency and decreased message throughput as the number of nodes grows.
This paper presents an in-depth examination of a proposed design’s architecture, as depicted in Figure 2, comprising three distinct layers. Each layer encapsulates specific functionalities and contributes synergistically towards the overarching objectives of the system.
