End-to-End Integration Headache

These Tools Won’t Simplify End-to-End Integration. Here’s Why.

Share this post

Recently, FiveTran announced a partnership with Census to “simplify end-to-end integration.”  Despite their bold claim to simplify, we believe the new partnership actually complicates data unification initiatives while exploiting affected businesses by ensuring they pay for two tools to achieve an outcome that should have always been possible with one. Let’s dive in.

In this context, “end-to-end integration” means that with subscriptions to both products, a customer could create ELT (extract, load, and transform) workflows with FiveTran to bring operational data into a warehouse (e.g. Snowflake), analyze it, then send actionable insights back to the operational systems and people that need them using reverse ETL workflows created in Census. See the diagram below for more detail.

FiveTran and Census

Before we discuss why this approach will create some significant operational headaches, let’s first discuss a few key reasons why someone might be willing to go down this path at all.

Growing need for data unification  

Integrating data from two or three top systems just won’t cut it today.  To compete, organizations are under tremendous pressure to bring together all their customer data, which is often spread out across tens, if not hundreds, of disparate applications. APIs, modern architectures, and ETL approaches are enabling this by providing real-time, streaming access to the data we need, but not all integrations are created equal.

Cry for help on data models

Every application, it seems, describes your data in a slightly different way, making 100% unification impractical, if not impossible.  A generation from now, I expect they’ll look back at this time and wonder why we made life so hard on ourselves.  Until then, customers are looking to vendors and technologies to help them align and transform disparate, ever-changing schemas and data into something more manageable: a unified data model.

The limitations of a data warehouse as your single source of truth

Data warehouse vendors like Snowflake are leading the market conversation right now because they’ve made it stupid simple to bring massive amounts of data into a central location for analysis. There are two obvious problems with this approach. (1) it creates a non-operational data silo and (2) insights need to be made available on a timely basis to the operational teams that need them to make day-to-day decisions — hence the emergence of Census and “Reverse ETL.”

3 solutions + operational friction vs. 1 solution +  better data

And now, let’s discuss the limitations of delivering on the above with Census, FiveTran and Snowflake.  If a user were to mimic the solution architecture of diagram 1, they’ll run into a few rather predictable headaches:

Operational complexity

When end-system schemas change (e.g. custom field gets added/deleted, API gets updated), how do those changes propagate across all three systems? Do FiveTran and Census update their connectors at the same time?  Will a data model change in FiveTran or SnowFlake break any scripts running in Census? And what happens to data in transit while the connections and data models are out of sync?  Managing this change is likely feasible if your deployments are simple – but I doubt any of these vendors only want you to solve simple problems with their solutions.

Visibility gaps

Data is never perfect, and it seems the one constant with cross-functional data transactions is that something almost always goes wrong.  The main problem with a 3 system configuration is that when something goes wrong, pinpointing WHERE it went wrong becomes extremely difficult.  If you’ve ever experienced “swivel-chair management” you know how hard it is to correlate error messages in one management pane with those of another.  

Cross-system data integrity chaos

Any unification or normalization exercise will almost certainly result in data conflicts and duplicates.  Data warehouses control for this from a reporting and analytics lens, not an operations lens.  Attempts to bring data that’s been standardized for reporting BACK into operational systems with some significant due diligence around HOW that data was used will almost certainly result in data chaos.

Syncari: A better way to modernize your data stack

Syncari uniquely combines the ELT capabilities of FiveTran and the “reverse ETL” capabilities of Census in one complete platform to realistically simplify end-to-end integration.  Having these two technology domains on one complete data automation platform, along with a fully managed data warehouse of your choosing (e.g. SnowFlake or RedShift), gives you unprecedented visibility, governance and control over your data without the operational headache of a multi-vendor approach. 

As you can see in the diagram below, Syncari’s complete data automation platform delivers the basics from Census + FiveTran and so much more:

Syncari Modern Data Stack

Let’s explore the capabilities in a bit more detail:

Automated data operations

  • Self-healing connectivity to extract and load data
  • Resilient, multi-directional sync to all your systems inline
  • Automatic schema change management

Continuous data unification

  • A unified data model that automatically normalizes cross-system data
  • Codeless data transformation and management (or via dbt if that’s your thing)
  • Centralized enrichment, merge and dedupe

Customer 360 + actionable insights everywhere

  • Interactive, cross-system data views
  • Cross-system Data Fitness Index + inline improvement recommendations
  • Extensible to leading Data Lake and Warehouse platforms (e.g. Snowflake, Redshift)
  • Extensible to the BI tool of your choice (e.g. Tableau, Looker)

Why put up another barrier to operational efficiency and increased revenue when Syncari is designed to knock these barriers down? It’s time to truly simplify end-to-end integration. Learn more by visiting our On-Demand Demo Hub.

__

About the author:

Aaron Landgraf is Syncari’s Head of Marketing. He brings 8+ years of integration experience at MuleSoft, watching it grow from scrappy startup to market leader in data integration and API management.

Share this post

Join hundreds of revenue leaders fighting the good fight against bad data.