Azure 上各種 Message Queue 的比較

 
資料來源整理自 Azure 官網：

• 什麼是服務匯流排傳訊？
• 比較訊息服務
• Azure 佇列和服務匯流排佇列的比較

 

傻瓜快速分類：

• 傳遞的是 Event 類型( 訊息量不大，多為狀態，射後不理)，考慮 Event 類型
  
• 如果是離散型資料，考慮 Event Grid 
• 如果是串流行資料，考慮 Event Hubs
• 傳遞的是 Message 類型 （訊息量較大，帶有資訊，需保證送達），考慮 Message Queue
  
• 訊息很多很大不用保證 FIFO，選擇 Storage Queue
• 訊息不會很大，要保證FIFO
  

此外 Service Bus 才有支援 AMQP，更多資訊請參考： 同樣都是Message Queue 到底AMQP跟JMS 有什麼差別？

 

Update 找到一個更棒的整理：

source: Azure Messaging Service in a picture

 

 

當JPA 遇到 JSON 偷懶是進步的動力

話說這篇壓了好久遲遲不敢放出來...不過想想放出來讓大家笑笑，讓大家少走點冤枉路也好~:P

上圖是一般系統開發常見的資料流程，不管是先設計Db schema 或是先設計 Object Model，到最後你可能都會遇到兩個問題：

1. 怎麼把DB Table轉成Java物件
2. 怎麼把Java物件轉成Json格式傳出去 (常見的就是RESTful API)

Apache Kafka 又一個messaging system !?

圖片來源：網路

這次在ApacheCon2013 上看到  Apache Kafka 的專案介紹，心想又一個messaging system !?

Intra-cluster Replication in Apache Kafka

於是又好奇的去收集了許多資料，首先參考Slideshare的一篇介紹Apache Kafka 的投影片，Kafka號稱最大的特色是同時混和了Offline log以及Realtime Message  兩種功能。

還記得上一篇文章我有提到各種分散式Log-aggregation系統 (如Scribe 和 Flume)，他們的架構都是Push driven architecture，雖然具高效能高擴充性，但是有以下缺點：

• 預期的端點(End points)都是大型叢集(如：Hadoop)
• 端點(End points)不能有太多即時性商業邏輯 (business logic in real-time)

因為他們最主要的工作就是盡量快速的去消化客戶端push 過來的log，而跟傳統的Messaging System(如：RabbitMQ , ActiveMQ)，並不夠Scale ?：
(不太懂這邊的意思，還需要要研究一下...)

• No API for batching, transcational (broker retain consumers stream position)
• No Message persistence means multiple consumers over time are impossible limiting architecture

圖片來源：Apache Kafka

談到MQ大家最有興趣的一點就是效能了，看到投影片的圖...有沒有那麼神！？

圖片來源：Apache Kafka

不過仔細看benchmark 的條件與環境設定，還是發現他為了增加吞吐的效能，對於資料的的完整性和確保性作了取捨：

Kafka producer currently doesn't wait for ack form the broker. Without ack , there is no gurantee that every published message is actually receved by the broker

所以如果是必須保證送達的系統，可能就不能使用Kafka 了~

RabbitMQ vs. Kafka

了解了Kafka特性後，還是不免會想要跟RabbitMQ作一下比較，參考Quora的這篇文章"RabbitMQ vs Kafka which one for durable messaging with good query feature"，針對兩個系統的特色，節錄以下內容：

a) Use Kafka if you have a fire hose of events (100k+/sec) you need delivered in partitioned order 'at least once' with a mix of online and batch consumers, you want to be able to re-read messages, you can deal with current limitations around node-level HA (or can use trunk code), and/or you don't mind supporting incubator-level software yourself via forums/IRC. 

b) Use Rabbit if you have messages (20k+/sec) that need to be routed in complex ways to consumers, you want per-message delivery guarantees, you don't care about ordered delivery, you need HA at the cluster-node level now, and/or you need 24x7 paid support in addition to forums/IRC.

不過作者也回答道，如果要求的是Real time streaming process (filter/query) 的功能的話，這兩個都不適合，反而應該考慮Storm：

Neither offers great "filter/query" capabilities - if you need that, consider using Storm on top of one of these solutions to add computation, filtering, querying, on your streams

透過這些文章與解釋我似乎開始了解 Kafka or Storm 的用途與應用情境，也漸漸了解到為什麼很多公司都是使這兩個系統用來補足Hadoop real-time process 不足的地方，因為實在有太多公司使用這種組合，下面舉例infochimps這間公司如何看待Storm and Kafka：

Why should you care?

With Storm and Kafka, you can conduct stream processing at linear scale, assured that every message gets processed in real-time, reliably. In tandem, Storm and Kafka can handle data velocities of tens of thousands of messages every second.
Stream processing solutions like Storm and Kafka have caught the attention of many enterprises due to their superior approach to ETL (extract, transform, load) and data integration.
Storm and Kafka are also great at in-memory analytics, and real-time decision support. Companies are quickly realizing that batch processing in Hadoop does not support real-time business needs. Real-time streaming analytics is a must-have component in any enterprise Big Data solution or stack, because of how elegantly they handle the “three V’s” — volume, velocity and variety.
Storm and Kafka are the two technologies on the list that we’re most committed to at Infochimps, and it is reasonable to expect that they’ll be a formal part of our platform soon.

ps. 話說趨勢也有自己的MQ叫做TME - TrendMicro Message Exchange，所以我才會覺得怎麼又一個MQ....XD

Reference:
[1] Message Queue Evaluation Notes
[2] Background Jobs in Ruby on Rails
[3] Intra-cluster Replication in Apache Kafka

同樣都是Message Queue 到底AMQP跟JMS 有什麼差別？

Source: U2Canvas 2012

不知道最近寫的blog算不算是策展(Curation)的一種，正因為我遇到一些問題，有一些疑問，所以我就自問自答的整理了網路上收集來的資料(Reference)，用自以為類似寫論文的方式整理這些資料，並且嘗試消化吸收，看看可不可以提出自己的一些見解？ 嗯這都是題外話~XD
今天要自問自答的題目是 -  同樣都是Message Queue到底AMQP跟JMS 有什麼差別？ AMQP 優點在哪?

Message Queue

在介紹JMS和AMQP之前先說一下什麼時候會需要用到Message Queue，就如同下圖所示，如果你今天需要一個Reliable (保證傳達) 且 Async (允許非同步) 的訊息傳達和交換方式，這時候你就需要Message Queue，而首選就是AMQP。此外對於Cloud 和SOA 來說 AMQP也是佔了很重要的角色。

 Source:AMQP and the future of web messaging

JMS的是什麼？

JMS 是Java的標準 API 使用兩種機制來傳遞訊息：

• store-and-forward
• publish/subscribe

但是 JMS 就像許多JSR的標準一樣，只是制定Interface以及行為模式，所以本身並沒有實作，必須倚賴其他的3rd-party實作，此外就如同它的名字，它只能在Java的環境使用，所以不適合跨平台跨系統的整合。而AMQP的出現就是為了要解決這個問題，不但實作了JMS的標準更提供了API for C, C++, Python, C# or any other language on Linux, Solaris, Windows, Z/OS, etc... [1]

AMQP 的緣起

AMQP 全名 Advanced Message Queuing Protocol 是一個由OASIS公開的標準，它誕生是在 2006年 由金融業 JP Morgan 的 John O;Hara  了解決異質平台訊息交換所的問題所提議，目的是要建立一個標準的協定與平台 - 包含資料的結構(binary wire-level protocol) 、如何傳遞，另外也定義了以下的特點 message orientation, queuing, routing (including point-to-point and publish-and-subscribe), reliability and security.[1]

AMQP與JMS主要差異

這一段就直接貼原文了，這種東西真的很難翻譯..@@

AMQP mandates the behavior of the messaging provider and client to the extent that implementations from different vendors are truly interoperable, in the same way as SMTP, HTTP, FTP, etc. have created interoperable systems. Previous attempts to standardize middleware have happened at the API level (e.g. JMS) and this did not create interoperability.[3] Unlike JMS, which merely defines an API, AMQP is a wire-level protocol. A wire-level protocol is a description of the format of the data that is sent across the network as a stream of octets. Consequently any tool that can create and interpret messages that conform to this data format can interoperate with any other compliant tool irrespective of implementation language.

AMQP主要定義了以下主要元件

AMQP (note that Rabbit works with many protocols including STOMP, HTTP, HTTPS, XMPP, and SMTP), here are some basic component descriptions:

• Exchange 
• The entity within the server which receives messages from producer applications and optionally routes these to message queues within the server
• Exchange type 
• The algorithm and implementation of a particular model of exchange. In contrast to the "exchange instance", which is the entity that receives and routes messages within the server
• Message queue 
• A named entity that holds messages and forwards them to consumer applications，Queues are bound to exchanges with patterns
• Binding 
• An entity that creates a relationship between a message queue and an exchange. Message are routing to queues by matching routing keys against binding patterns.
• Routing key 
• A virtual address that an exchange may use to decide how to route a specific message. Message are published with a routing key.

所以主要的元件如下圖所示：

所以AMQP最主要就是多了Exchange的機制，除了 store-and-forward、publish/subscribe外還多了許多的Routing 方法 [5]：

• Direct Exchange

• Fanout Exchange 

• Topic Exchange

• Header Exchange
• System Exchange    

Refernece:
[1] How is this different from the Java Messaging Service (JMS)?
[2] Advanced Message Queuing Protocol
[3] Migrating From JMS to AMQP: RabbitMQ, Spring, Apache Camel, and Apache Qpid
[4] Introduction to amqp messaging with Rabbitmq
[5] Exchanges

如何安裝RabbitMQ Cluster

建立測試環境與測試系統總是最花時間的，趁著今天重新安裝一組RabbitMQ 的測試環境，把所有的步驟和順序都整理起來吧。

安裝RabbitMQ

說實在RabbitMQ安裝起來真的還蠻簡單的，參考RabbitMQ - 安裝指南：

• 要先安裝epel-release的套件。 (會不定期更新，要確認是否有新版)
• 再安裝erlang

# rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-7.noarch.rpm
# yum install erlang
# rpm --import http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
# rpm -Uvh http://www.rabbitmq.com/releases/rabbitmq-server/v2.8.4/rabbitmq-server-2.8.4-1.noarch.rpm


設定Cluster

最簡易的方法就是先啟動，cluster master node產生 /var/lib/rabbitmq/.erlang.cookie 這份文件，然後再把這份文件copy 到 每一個cluster node鄉對應的目錄下面去，接下來就參考RabbitMQ - create cluster這份文件，範例事先建立3台的cluster。
 
1. 啟動每台rabbitMQ
# rabbit1$ rabbitmq-server -detached
# rabbit2$ rabbitmq-server -detached
# rabbit3$ rabbitmq-server -detached

2. 確認每一台rabbitmq 是否是standalone狀態

# rabbit1$ rabbitmqctl cluster_status
Cluster status of node rabbit@rabbit1 ...
[{nodes,[{disc,[rabbit@rabbit1]}]},{running_nodes,[rabbit@rabbit1]}]
...done.
# rabbit2$ rabbitmqctl cluster_status
Cluster status of node rabbit@rabbit2 ...
[{nodes,[{disc,[rabbit@rabbit2]}]},{running_nodes,[rabbit@rabbit2]}]
...done.
# rabbit3$ rabbitmqctl cluster_status
Cluster status of node rabbit@rabbit3 ...
[{nodes,[{disc,[rabbit@rabbit3]}]},{running_nodes,[rabbit@rabbit3]}]
...done.
3. 依序加入cluster

rabbit2$ rabbitmqctl stop_app
Stopping node rabbit@rabbit2 ...done.
rabbit2$ rabbitmqctl reset
Resetting node rabbit@rabbit2 ...done.
rabbit2$ rabbitmqctl cluster rabbit@rabbit1
Clustering node rabbit@rabbit2 with [rabbit@rabbit1] ...done.
rabbit2$ rabbitmqctl start_app
Starting node rabbit@rabbit2 ...done. 

4. 確認cluster status是否正確加入

rabbit1$ rabbitmqctl cluster_status
Cluster status of node rabbit@rabbit1 ...
[{nodes,[{disc,[rabbit@rabbit1]},{ram,[rabbit@rabbit2]}]},
{running_nodes,[rabbit@rabbit2,rabbit@rabbit1]}]
...done.
rabbit2$ rabbitmqctl cluster_status
Cluster status of node rabbit@rabbit2 ...
[{nodes,[{disc,[rabbit@rabbit1]},{ram,[rabbit@rabbit2]}]},
{running_nodes,[rabbit@rabbit1,rabbit@rabbit2]}]
...done.


設定Web Management Plugin

參考 rabbitmq - Management Plugin，重點是要在Cluster node的每一台都要安裝這個plugin
# rabbitmq-plugins enable rabbitmq_management 
安裝好以後，要記得重啟服務
# service rabbitmq-server restart

設定好就可以連到http://server-name:55672/#/ 看看web 介面有沒有起來，預設的帳號密碼都是guest，如果是內部測試那還沒關係，如果是要連到外面，一定記得要改帳號密碼和permission。

設定帳號

最後我們可以設定一個vhost 以及這個vhost的帳號密碼和權限

#rabbitmqctl add_vhost /demo
#rabbitmqctl add_user demo mypass
#rabbitmqctl set_permissions -p /demo demo ".*" ".*" ".*"



Reference:

[1] RabbitMQ 指令集