This example creates a simple sink that assigns records to the default one hour time buckets. withRollingPolicy ( DefaultRollingPolicy. forRowFormat ( new Path ( outputPath ), new SimpleStringEncoder ( "UTF-8" )). val sink : StreamingFileSink = StreamingFileSink. Import. import .fs.Path import .StreamingFileSink import .rollingpolicies.DefaultRollingPolicy val input : DataStream =. bucketCheckInterval (default = 1 min) : Millisecond interval for checking time based rolling policiesīasic usage for writing String elements thus looks like this:.Custom RollingPolicy : Rolling policy to override the DefaultRollingPolicy.In addition to the bucket assigner, the RowFormatBuilder allows the user to specify: Row-encoded formats need to specify an Encoder that is used for serializing individual rows to the OutputStream of the in-progress part files. Please check out the JavaDoc for StreamingFileSink for all the configuration optionsĪnd more documentation about the implementation of the different data formats. Stored and the encoding logic for our data. When creating either a row or a bulk encoded sink we have to specify the base path where the buckets will be Bulk-encoded sink: StreamingFileSink.forBulkFormat(basePath, bulkWriterFactory).Row-encoded sink: StreamingFileSink.forRowFormat(basePath, rowEncoder).These two variants come with their respective builders that can be created with the following static methods:
The StreamingFileSink supports both row-wise and bulk encoding formats, such as Apache Parquet. If checkpointing is disabled, part files will forever stay in the `in-progress` or the `pending` state,Īnd cannot be safely read by downstream systems. IMPORTANT: Checkpointing needs to be enabled when using the StreamingFileSink. The default policy rolls part files based on size, a timeout that specifies the maximum duration for which a file can be open, and a maximum inactivity timeout after which the file is closed. Additional part files will be created according to the configurable Each bucket will contain at least one part file forĮach subtask of the sink that has received data for that bucket. This means that each resultingīucket will contain files with records received during 1 hour intervals from the stream.ĭata within the bucket directories are split into part files. With a default time-based bucketing where we start writing a new bucket every hour. The bucketing behaviour is fully configurable Given that the incoming streams can be unbounded,ĭata in each bucket are organized into part files of finite size. The streaming file sink writes incoming data into buckets. Supported by the Flink FileSystem abstraction. This connector provides a Sink that writes partitioned files to filesystems
We recommend you use the latest stable version. This documentation is for an out-of-date version of Apache Flink.
0 kommentar(er)
