flume使用场景 flume与kafka的比较

发表于 2019-10-25 11:04:25

马上注册，结交更多数据大咖，获取更多知识干货，轻松玩转大数据

您需要登录才可以下载或查看，没有帐号？立即注册

x

Is Flume a good fit for your problem?

If you need to ingest textual log data into hadoop/HDFS then Flume is the right fit for your problem, full stop. For other use cases, here are some guidelines:

Flume is designed to transport and ingest regularly-generated

event data over relatively stable, potentially complex topologies. The notion of “event data” is very broadly defined. To Flume, an event is just a generic blob of bytes. There are some limitations on how large an event can be - for instance, it cannot be larger than what you can store in memory or on disk on a single machine - but in practice, flume events can be everything from textual log entries to image files. The key property of an event is that they are generated in a continuous, streaming fashion. If your data is not regularly generated (i.e. you are trying to do a single bulk load of data into a Hadoop cluster) then Flume will still work, but it is probably overkill for your situation. Flume likes relatively stable topologies. Your topologies do not need to be immutable, because Flume can deal with changes in topology without losing data and can also tolerate periodic reconfiguration due to fail-over or provisioning. It probably won’t work well if you plant to change topologies every day, because reconfiguration takes some thought and overhead.

上面是flume官网的说明，翻译如下：

Flume是否适合你的问题？

如果你想把可文本化的日志数据提取到HDFS，那么Flume是非常适合的。对于其他场景，有些东西是需要考量的：

Flume被设计用来传输、提取定期生成的数据的，这些数据是传输在相对稳定的、可能是复杂的拓扑结构上的。每个数据就是一个event。“event data”的概念是非常广泛的。对于Flume而言，一个event就是一个blob字节数据。这个event的大小是有限制的，例如，不能大于内存或硬盘或单机可以存储的大小。事实上，flume的event可以是任何东西，从日志文本到图片文件。Event的关键点是不断生成、流式的。如果你的数据不是的定期生成的（比如一次性的向Hadoop集群导入数据），Flume可以工作，但是有点杀鸡用牛刀了。Flume喜欢相对稳定的拓扑结构。你的拓扑结构不必是不可改变的，因为Flume可以在不丢失数据的前提下处理拓扑结构的改变，并且能容忍由于故障转移导致的周期性的重新配置。但如果你每天都要改变拓扑结构，那么Flume将不能很好的工作，因为重新配置会产生开销。

简而言之，有两点：

1、数据。数据是定期生成的。

2、网络拓扑相对稳定。

Kafka、Flume都可以实现数据的传输，但它们的侧重点不同。

Kafka追求的是高吞吐量、高负载（topic下可以有多个partition）

Flume追求的是数据的多样性：数据来源的多样性、数据流向的多样性

如果数据来源很单一、想要高吞吐的话可以使用Kafka

如果数据来源很多、数据流向很多的话可以使用Flume

也可以将Kafka和Flume结合起来使用。

来自为知笔记(Wiz)