I’ve had enough! Over and over again I see blogs, articles and podcasts that suggest how companies should process their data to then send it to several business applications. A recent suggestions looked something like this:


A group of logos



*Dataform is a modeling layer that uses SQL.

‍

There are many other ways to do it and certainly more complex ones, but the point is that things don’t have to be this complex. Let’s go over this process, briefly:

  • Create event triggers from BigQuery table updates
  • Create a Google Cloud Function triggered by the event you create
  • Send BigQuery data to Intercom's REST API 

Things get even worse if you want to run ML. Now, you would need another couple of blocks in that process since this is not just about running ML, it is also about scaling the infrastructure.

The process can be as complex as you want, but tools and tech have evolved in a way that things don’t have to be painful. Think what happened before Zapier or Stitch. Developers would have to build the endpoints to be able to connect each application. Zapier came in and changed that process. Now with a simple point and click  you are able to automate workflows.

Same thing happens with data processing and ML. This is how it should work:

  1. From Bigquery connect your data to a tool like www.datagran.io.
  2. Model your data and run your ML.
  3. Drag and drop your favorite Business app and send your model’s output..

Check out a quick demo here 

 

That’s it! You don’t need to worry about devops, APIs, or additional custom development. Need a clearer picture? Imagine your marketing team wants to send the output to several hundred applications, would you develop APIs and read documentation for each one of those apps?

Life can be easier for people, we just need to start embracing solutions that allow us to focus on what’s important, instead of just having to build everything in house. Recently, I asked a well known VC why in the Valley tech companies always want to build their own solutions. His answer was “Random opinion: it can be good marketing for hiring developers. AirBnB made Airflow to say: 1) We are hip and cool 2) We have a lot of autonomy 3) We have a lot of data (aka successful) 4) We give back 5) We are thought leaders and you can be one. But, 6) It didn’t exist before”. More on the tweet here.

Key learning: If you are more interested in team efficiency and effectiveness, stop building everything in house.