In essence, it is a service that lets you transfer data between systems, bi-directionally if you require it to. Azure Data factory has a vast number of connectors available, making it really easy to import (and export) data from pretty much any standard data source out there. Some examples include:
Excel (why is this not in a DB?)
Most companies understand the benefits of moving to the cloud and have a roadmap in place to get there. The challenge is that not all systems move over to the cloud at the same time. And as we all know, most applications make use of more than one data source. This creates a chicken and egg type of dilemma, what system moves to the cloud first and how do we keep the dependent systems still stuck on premises functional?
We can pay for a VPN or a similar solution that creates a tunnel between our on-premises network and the system in the cloud, but this can get really expensive. Or we can move all dependent systems in one go, but this is a complex and risky undertaking.
This is where Azure Data Factory saves the day. It surfaces the on-prem data to the cloud using replication with very low cost. Below is a snapshot of high-level cost at time of writing:
Use the following link for the latest pricing:
Now, this approach will not work for all scenarios, but what we've seen is that the majority of related data needed is for lookup data, the type of data that does not change frequently. Think Cost Centers, Departments, Expense Types. If the data captured in cloud needs to be replicated back, it will also not be available immediately, there will be some delay depending on the configuration of the Integration runtime.
Here's a basic diagram of all the parts involved
For the purposes of this post, a basic understanding of the components should suffice. You will need:
A Data Source - Where the data will be pulled from, CSV file, SQL Database, Oracle, etc.
Source Linked Service - Secure connection to the data source
Source Data Set - Describes the data pulled from the source
Activity - The action needed to be performed on the data
Target Data Set - Describes the data to be pushed to the destination
Target Linked Service - Secure connection to where the data will reside
Data Target - Final data source where the data will be available from
This execution is overseen by the Integration pipeline, which in turn is managed by the Data Factory, which can have multiple pipelines.
For more complex environments and scenarios, Linked Services can be shared for both the Source and Target Linked Services.
For our example, we will create a duplicate a table of our Cost Centers data from our on-prem SQL database to our Azure SQL database, where NWC and K2 Forms can pull this data from, using Xtensions and SmartObjects. The assumption here is that the on-premises and Azure SQL databases with the table structures have already been provisioned, so we will just be setting up the Data Factory pieces.
Your data sources are replicated and can be used from your cloud resources.
The easiest way to have these run regularly is to create a Trigger for the pipeline. Simply click the Add Trigger button on top, select the + New in the Choose trigger dropdown list and complete the trigger setup. I configured mine to run everyday at 12:00AM:
Once everything is configured, click the Publish all button on top and after the first sync, your NWC Xtension via an Azure function or K2 SQL Service Instance can now consume the data.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.