The previous blog looked into building maintainable data pipelines for large-scale projects. This blog will look at optimising data pipelines for better performance. Recap In the last blog, we looked at the following: An introduction to Apache Airflow Developing manageable scripts for data pipelines Maintaining pipeline configurations Managing pipeline secrets and…

This article provides easy access to the posts I have written in the past. Blogs in different domains would be segmented into sections. Data Science — General 5 Essential Management Strategies For A Data Science Project In this blog, I will share my experience of how working with multiple clients has helped me understand the core…vachan15.medium.com A Beginner’s Guide To Responsible Machine Learning & A.I. There have been numerous times when an A.I. system built to resolve a problem blew up in the face of humanity & made…vachan15.medium.com

This blog will look into Apache Airflow, an open-source workflow orchestration tool, and uncover best practices around the application to make development easier. Before getting into the practices, let us briefly introduce Airflow. Airflow Simply put, Apache Airflow is an open-source workflow orchestration tool. While working on it, we create Directed…

Quite frequently, as a data scientist, I have been asked a question inspired by Sci-Fi movies, “Can A.I. powered robots take on the world and rule the human race?”. As interesting as this question sounds, we are a long way before artificial intelligence systems could have the “intelligence” or the…

In the present world, where data is one of the most valuable asset, organisations are looking into ways to tap into information previously locked. As a consultant, I had an opportunity to have a closer look into the extent of information locked into formats that are not easily accessible by…

In this blog we are going to have a gentle introduction to building end to end data pipelines using some of the serverless technology. …

In this series of blogs we will look into Operationalising machine learning model using AWS serverless infrastructure. The purpose is to build a machine learning pipeline that trains and tunes the model before creating a model endpoint that can be used to get predictions. As part of this series, the…

Ever since machine learning or artificial intelligence took a leap in popularity, more often than not, we think of M.L. as a piece of software/code that has the capabilities of predicting the future given an input data. For instance, forecasting the price of bitcoin or recommending a user with products…

In this blog we are going to focus on creating a data warehouse on AWS Redshift and establishing a connection to it from our local machine. Note : This blog is going to cover an introductory level information of creating a cluster and getting started with writing basic queries. It…