To that database that supports asynchronous requests. Note: Improving throughput by just scaling the MapFunction to a very high parallelism is in some cases possible as well, but usuallyĬomes at a very high resource cost: Having many more parallel MapFunction instances means more tasks, threads, Flink-internal networkĬonnections, network connections to the database, buffers, and general internal bookkeeping overhead.Īs illustrated in the section above, implementing proper asynchronous I/O to a database (or key/value store) requires a client At the very least, the waiting time is amortized over multiple requests. That way, the waiting time can be overlaid with sending other requests and Makes up the vast majority of the function’s time.Īsynchronous interaction with the database means that a single parallel function instance can handle many requests concurrently and Naively accessing data in the external database, for example in a MapFunction, typically means synchronous interaction:Ī request is sent to the database and the MapFunction waits until the response has been received. That communication delay with the external system does not dominate the streaming application’s total work. When interacting with external systems (for example when enriching stream events with data stored in a database), one needs to take care Note: Details about the design and implementation of the asynchronous I/O utility can be found in the proposal and design documentįLIP-12: Asynchronous I/O Design and Implementation.ĭetails about the new retry support can be found in documentįLIP-232: Add Retry Support For Async I/O In DataStream API. This page explains the use of Flink’s API for asynchronous I/O with external data stores.įor users not familiar with asynchronous or event-driven programming, an article about Futures andĮvent-driven programming may be useful preparation. We recommend you use the latest stable version.Īsynchronous I/O for External Data Access This documentation is for an unreleased version of Apache Flink. The need for Asynchronous I/O Operations.Hadoop MapReduce compatibility with Flink.

Conversions between Table and DataStream.Conversions between PyFlink Table and Pandas DataFrame.

Aasync download windows