Don't miss the upcoming webinar: Building Real-Time Data Pipelines with a 3rd Generation Stream Processing Engine - sign up now!

Batch Connectors

Application Log Sample code on Github Docs

Source: No Sink: Yes Batch

Logs all the data items it receives at the INFO level. The primary purpose is development when running Jet on a local machine.

Avro Sample code on Github Docs

Source: Yes Sink: Yes Batch & Streaming

Jet can read and write Avro-serialized data from self-contained files (Avro Object Container format), HDFS and Kafka. Kafka connector can be configured to use the schema registry.

Custom Connector Sample code on Github Docs

Source: Yes Sink: Yes Batch & Streaming

Jet provides a programming interface allowing you to write your own Connectors for both batch and streaming.

Files (Batch) Sample code on Github Docs

Source: Yes Sink: Yes Batch

Reads all the files in a local directory. The same directory has to be available to all cluster members (using shared network file system for example). The sink writes output to several files in the configured directory to avoid contention when writing by multiple parallel instances.

Hazelcast ICache (Batch) Sample code on Github Docs

Source: Yes Sink: Yes Batch

Fetches entries from a Hazelcast ICache (source). Supports a predicate and projection pushdown. The connector makes use of data locality when reading from an embedded Hazelcast IMDG. Sinks write entries to an ICache using map.put() or using an Entry Processor to update the entries in ICache instead of replacing.

Hazelcast IList Sample code on Github Docs

Source: Yes Sink: Yes Batch

Reads items retrieved from a Hazelcast IList. All elements are read on a single member of a Jet cluster — IList isn’t partitioned. Sink adds items to the IList.

Hazelcast IMap (Batch) Sample code on Github Docs

Source: Yes Sink: Yes Batch

Fetches entries from a Hazelcast IMap (source). Supports a predicate and projection pushdown. The connector makes use of data locality when reading from an embedded Hazelcast IMDG. Sinks write entries to an IMap using map.put() or using an Entry Processor to update the entries in IMap instead of replacing.

HDFS Sample code on Github Docs

Source: Yes Sink: Yes Batch

Reads from and writes to Apache Hadoop HDFS. Reading makes use of a data locality if the Jet and Hadoop clusters are co-located.

JDBC Sample code on Github Docs

Source: Yes Sink: Yes Batch

Reads or writes the data from/to relational database or another source that supports the standard JDBC API. Supports parallel reading for partitioned sources.

Hazelcast Jet

Main Menu