/avatars_200px_circle_Marcel_Kintscher.png){kind=avatar}
What's in it for you?
Read until the end and you will know what Fivetran custom connectors are useful for, what skills are required and how much effort it is to set them up. Fivetran is the smartest way to load data into a warehouse. It offers seamless connectivity to applications, databases, files and events. They provide you (and your team) more time for analyzing your data, instead of connecting your data.
When to use Fivetran Connectors?
Fivetran, one of our newest technology partners, is the company that has shifted the paradigm from ETL to ELT completely. The company with over 150 existing connectors that help data analysts to ease and speed up their analytics significantly. If you are an IT Professional or Developer and have used Fivetran before or have looked at the variety of connectors they have, you might be as impressed by their software as we are at Biztory.
Source: https://fivetran.com/docs/functions
However, some of you might have wondered sometimes, what’s beyond the 150+ connectors? Imagine you wanted to connect to Slack for example, a popular communications software. You go to Fivetran’s list of connectors, type in Slack and the result is “Ouups, no connector found”. Well, the usual process would be to go to Fivetran’s request intake page and request it. But there is a beautiful alternative to that. May I present? The Fivetran Custom Connector.
About Fivetran Custom Connectors
In simple terms, a Fivetran Custom Connector is a cloud-hosted function that is being set up not by Fivetran but by the customer instead. The function needs to be set up in one of three cloud platforms that Fivetran integrates with:
- Google Cloud Functions
- Microsoft Azure Functions
- Amazon AWS Lambda
Typically, after being set up, the custom function is then being called by Fivetran via an https-request. Using https, Fivetran can guarantee the continuous protection of your data, which is extremely important for you as you’re responsible for building reliable and secure data solutions. When Fivetran calls the function it expects a response which it then feeds into your destination, just like it does with its native connectors. For simplicity, take a look at the graphic below, which explains the process at a glance.
Source: https://fivetran.com/docs/functions
What Skills Do You Need?
First of all, given you have basic knowledge of programming with environments like Node JS, Python or .NET and working with APIs (Application Programming Interfaces), you have the ability to write a custom connector for Fivetran. You can connect to any data source of your choice, select only the data that is relevant to your case and you will still benefit from all the features of Fivetran. In order to be able to host the function, you will need an account with one of the three cloud services mentioned before, as well as basic knowledge about setting up a cloud function. Finally, to set everything up on Fivetran’s side, you need an account with Fivetran and an understanding of setting up connectors and destinations.
Ready? Alright, Here are the Steps to Setting it up!
Back to the Slack case. You want to feed data from Slack into your data warehouse and do analytics on your internal communications. How you do it varies a slight bit depending on the cloud platform where your function will be hosted. However, in general, these are the steps you need to take:
- Understanding the API of your data source
- Writing the cloud function in one of the common programming languages
- Uploading the function to the cloud platform of your choice
- Setting up the connection to your cloud function in Fivetran
- Initial Synch
Let’s walk through the setup steps.
First of all, you need an understanding of the API you want to use. In your case, you want to browse through the Slack API a bit to familiarize yourself with the authentication methods they use, the endpoints they are offering and the data you get back from it. Next, you need to put your code in writing. Your code should follow a certain format so that it can handle https requests coming from Fivetran and send data back in a format that Fivetran knows what to do with it. There are a few really useful examples of custom functions in Fivetran’s documentation.
It is also possible to let Fivetran store state related information after every synchronisation run. In your Slack case, you could make use of this functionality to request only the newest posts from Slack by including the last synched record id into your next API request. This process is called incremental loading and is extremely powerful, as Fivetran does all deduplication of your destination data for you.
After your code is ready and debugged, you can upload it to one of the three cloud platforms as a cloud function. When doing that, make sure it is set up as a https-function. If you struggle with this bit, make sure to leave drop your email and hit the subscribe button at the end of the page. We’ll make sure to help you move into the right direction.
The next and final step is to set up the connection to your cloud function in Fivetran. For that, you can simply log into your Fivetran account and add a new connection. Depending on which cloud platform hosts your function, you’ll have to look for that cloud service provider in the list of available connectors. You’ll have to provide a few details to Fivetran like the https-trigger-url of your function as well as any secrets that you want Fivetran to send to your function. Those could be access keys that your function needs to connect to the Slack API for example.
Finally, make sure that you provide cloud function invoker access to Fivetran’s service account before you finish creating your connection. All of these steps are well described on the right-hand side of your screen while you create the connection in Fivetran.
Last but not least, you hit ‘Save and Test’ and if that is successful you can begin with your initial load. Just like with any other function you can easily change the synchronisation schedule via the setup tab in your newly set up Fivetran connection.
Easily connect data with Fivetran.
In our partnership with Fivetran we share the same passion: Putting people of our customers at the center and making their lives easier with amazing technology. If you would like to see a real-world example of a Fivetran custom connector in the future, make sure to subscribe to our updates at the bottom of the page and stay tuned for new postings.
We'll gladly show you how Fivetran will save you time and resources, make your data engineer's life easier and finally provide a stable and scalable solution to data pipelining. You can always start a free trial, just get in touch with us and we'll hook you up!
/avatar_120x120_email_marcel_kintscher.png?width=120&name=avatar_120x120_email_marcel_kintscher.png){kind=avatar}
Marcel Kintscher
Analytics Consultant
Biztory
