Photo by Viva Luna Studios on Unsplash

This is a post to link together various past/future ones into a single story.

There will probably be some more related posts over time and I will try to keep this index updated!


Photo by the blowup on Unsplash

“Software does not fail” is a provocative statement and your initial reaction maybe to dismiss it. But if we look deeper, software “failure” is never random compared to possible random failure of hardware components (refer to Dr Nancy Leveson, YouTube, Engineering a Safer World). Yes, there are hard to reproduce issues but software execution is as per the code. The failure is of assumptions, requirements and unhandled conditions that go into writing that code. For example, the recent AWS us-east-1 incident involved no hardware or software failure, rather the complexity of interaction of various components and assumptions made in the…


Photo by Jonathan Borba on Unsplash

See other stories in this series

So far I have looked into understanding time series and then forecasting with Prophet, SARIMA, Greykite and Auto-ML with AWS and GCP. This is now the final part to understand how to evaluate and explain accuracy of forecasts.

As with all predictions, explaining and establishing trust in forecasts is important. Understanding this in detail is beyond my scope but similar to an adjusted R-squared value for regression or AUC for a classification problem, what can we use here?

AWS provides a good page of what metrics they run and also Facebook Prophet lists the…


Photo by Patrick Tomasso on Unsplash

See other stories in this series

Following up from some recent time series forecasting with Prophet, SARIMA and Greykite, this is a fourth related topic and I looked into Amazon Forecast from AWS and the GCP Vertex AI forecasting. Using the same dataset as previously to forecast API request rate for a hypothetical service.

AWS

Using the console the process is as expected: create a dataset, provide the data, train model and run inference.

In creating a dataset it was interesting to note the domains and web traffic sounds like the correct one in my case. Picking that forces certain field…


Photo by George Pagan III on Unsplash

As a machine learning learner one important topic to learn is producing recommendations. The Netflix prize is cited everywhere but unfortunately learning resources are simple examples or very complex research papers. Here is my attempt to curate some resources to understand various types of recommendations and keeping a new learner in mind. I will use some libraries to predict and then evaluate outcomes.

Problem

Generally, the goal of recommendations is to show users some products, news stories, services, or anything new to discover that they may be interested in or find engaging (these are known as items). …


Photo by Agnieszka Ziomek on Unsplash

See other stories in this series

I have been learning about time series and recently looked into Facebook Prophet and SARIMA. There is now a new Greykite library from LinkedIn that seeks to create an abstraction over multiple models including Prophet and introduces their own model called Silverkite.

I will not pretend to explain the model (because I don’t fully understand it) but LinkedIn provides the details and my interest is its application. Using the same example as previously for request rate on a hypothetical service and using Silverkite instead. The forecast fits surprisingly well!


See other stories in this series

Previously I had looked into Facebook Prophet and this is now an attempt to learn moving average (MA) models and ARIMA. Wikipedia pages are not intuitive so this is an attempt to understand it and apply to the same server request rate example discussed previously.

Moving Average

Moving average (also known as rolling average) at a given time in the time series is the mean of the previous N values. The N depends on the time series. …


Photo by Hulki Okan Tabak on Unsplash

See other stories in this series

If you are monitoring your web services with Wavefront, Prometheus, Grafana etc. then you are aware of time series. In this post I am sharing my learnings about time series forecasting. Starting with some basics of time series, then using a library from Facebook called Prophet to forecast and finally some thoughts on how this could work in for real world scenarios.

To start here is a sample hourly request data for a hypothetical service for 4 weeks (28 days * 24 = 672 data points). Each row represents average number of calls per…


Getting started from an excellent intro from Charlie Flanagan and his Machine Learning for Business class here is some experimentation with my own models, Google AutoML and AWS SageMaker.

Problem

Product reviews from a women’s clothing ecommerce store are provided. Based on each review it has been classified as whether the reviewer would recommend that product or not. Along with each review some additional features are also provided. The goal is to create a model that can predict the likelihood of product recommendation given the customer review and some additional features.

My Attempt

Using Colab and following Charlie’s example.

Step1 — Data and…


In a recent presentation about API Gateways, I got a question about service mesh and I did not understand the question or have a good answer. So here is an attempt to understand these concepts and see how they fit together in the context of a typical container based microservice deployment. At the end of the day it is about routing and shaping traffic.

API Gateways have been around for a while and are an essential component of the “Full Lifecycle API Management” products as Gartner puts it. In this full lifecycle, the API Gateway is typically a L7 reverse…

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store