The Amazon S3 synapse enables data synchronization between the files stored in Amazon S3 bucket and Syncari entities. Using the Syncari’s Sync Studio, users can map Amazon S3 files (csv file) to Syncari entities, and write updates back to S3. Data synchronization can be controlled at the entity level, allowing the user to turn off synchronization for a single entity.
Once the entities are enabled in the Sync Studio for synchronization, the initial sync starts. All existing records for the entity are pulled efficiently from Amazon S3 files and synced to Syncari. This is called the inbound pipeline for the entity. Once all existing data is extracted, the sync switches automatically to incremental sync. During incremental sync, the delta changes from Amazon S3 files are synced to Syncari in near real time.
To perform the incremental data load from Amazon S3 files to Syncari, the watermark field needs to be defined for every Amazon S3 files in Syncari.
The Amazon S3 Synapse allows synchronization on the below listed objects:
Syncari synchronizes Amazon S3 files schema and its mappings with Syncari’s schema during every sync cycle. The newly added columns on the Amazon S3 files are detected automatically and available for usage in the pipelines. They are not activated by default. The deleted columns Amazon S3 files are detected automatically. These objects and fields are marked as deleted in Syncari. They no longer participate in synchronization.