Penn State University

Indian Institute of Technology, Bombay

Data Leadership (H1B for Life)

I manage a team of 4 data engineers and 2 Looker specialists

We built a pipeline from Oracle ERP into BigQuery and modified DBT models supporting reporting and operational workflows for teams like the Supply Chain team 

Organized the team so that each business domain is assigned to a specific engineer to help prioritize work for a small team

Currently planning moving to DBT Cloud from self hosted solution to introduce standard SDLC practices to speed up development process and reduce production bugs

Team is strategically working on moving business logic to the DBT layer to make Looker a thin layer around it

Warby Parker

Manager, Data Engineering

I currently lead a team of 6 data scientists and data engineers. I was previously leading 2 teams of 4 engineers each here before a strategic reorg.

Some projects we have worked on:

Re-architected the existing system for ingestion of merchant product feeds from Postgresql to Snowflake to make it more scalable. Streamlined the process of onboarding merchant feeds on to the platform with the goal of almost fully automating it.

Led the creation of a self service platform powered by APIs for the solutions engineering team to enable them to both onboard merchant product feeds as well as data exports to our clients. The project also improved data validation by using the Great Expectations Python library for merchant feeds.

Evangelized data centralization and enabled replication of all data (transactional and 3rd party) on Snowflake using Fivetran to help democratize data and make data available for analytics and application development needs. Standardized on DBT and Snowflake as the data stack for the company.

Currently leading the team on creating the first layer of analytics (business logic/segmentations/time granularities) on the datawarehouse to create a single source of truth for data used by all systems. Concurrently, the team is working on complementary documentation like data dictionaries.

Also leading an effort to make Looker an effective, efficient, and reliable self service dashboard solution for internal Narrativ analytics.

Howl

Staff Engineer / Manager, Data Engineering & Analytics

I lead a team that focuses on machine learning and data engineering efforts at Fundera. 

Our team created the first generation of machine learning infrastructure for training and deployment of near real-time models in production. We deployed our first prediction service as a microservice which generates both features and predictions in real-time. We also developed related infrastructure for easy offline model development. A longer description can be found here: https://biggishdata.blogspot.com/2019/08/powering-real-time-predictions-at.html

The current prediction service ranks loan products by probability of funding for small business owners that use our website. Currently we are working on building out a richer feature set and introducing non-linearities in an effort to improve the model.

Because underwriting at our lenders' end is a manual process, it often takes multiple days to get a funding decision. Some of our ongoing efforts are related to handling this aspect better. For instance, matching online metrics with offline metrics, and how long we should wait before we can assume a model in production has stabilized.

We work collaboratively with product management to improve data quality, bolster data infrastructure and employ best practices in all aspects of the data pipeline. Our goal is to help Fundera excel at being data driven.

I am a strong advocate for helping engineers learn more machine learning skills. In addition to direct mentoring I also organize a "book club" where some of us learn from pertinent machine learning online courses.

Fundera

Machine Learning & Data Enginering Leadership

A recent collaborative project was a Machine Learning infrastructure project to enable online inference for all recommendation models used at Etsy. The architecture is powered by Finatra and involves services running on Kubernetes. Also working on migrating this framework to Google cloud.

As a core member of the recommendation team, I work on the recommendation framework, feature engineering, and modeling to improve recommendation modules on the website that have resulted in GMV wins worth millions of dollars.

I led a small team in a project that overhauled spelling correction at Etsy. We replaced the older framework that was based on a static map of spelling corrections with one that used a statistical model trained on historical search data. Hidden Markov Models provide the basis for the new service which is described in a blog post (https://codeascraft.com/2017/05/01/modeling-spelling-correction-for-search-at-etsy/) contribution I made to the company's tech blog.

Together with another engineer, I worked on the first project (Context Specific Ranking) of its kind at Etsy that allowed a machine learning model to make inferences in real-time. This was in the context of the Our Picks For You module on the home page of the site. Previously, we would create predictions for all our users in a nightly BigData job. CSR enabled us to limit the nightly job to only calculate the candidate set of listings for each user along with their associated feature vectors. When a user now comes to Etsy, we are able to score all the candidates, rank them, and serve the highest ranked listings on the module in real time. We save on generating recommendations for all our user independent of whether they come to website or not. However, the most exciting development here is that this project enables us to now use real-time context in our machine learning models. This is a big step in overcoming the two-day lag inherent in our logs and, relatedly, in our batch-based process.

Etsy

Senior Software Engineer, Data Science

My responsibilities  while working in the Data Engineering team include writing big data jobs in scalding, and building the library code related to running and testing them. I write and maintain the code that handles the business logic on the ETL side, all in scala. I also write and maintain utility scripts which are designed to make working on hadoop as user-friendly to fellow engineers and analysts as possible. Originally, when I joined Etsy, all of big data was based upon cascading.jruby and Java. I was involved in the transition from cascading.jruby to scalding. Some of my time is also spent doing ad-hoc analysis using sql/scalding, and in training fellow employees to be more effective on the hadoop cluster.

We recently rewrote our entire event-logging and ETL pipeline to incorporate Kafka. I wrote the part that downloads data from Kafka to HDFS, also contributed to code that records event requests from the browser, and, finally, also to the middle layer between the webs and the Kafka cluster which is written in golang.

Some of my work falls under the category of improving data quality. I worked closely with analysts to reconcile data from the new event logging pipeline with the older system it replaced, fixing bugs and tweaking business logic as needed. Also lead an effort to help the internal platform team bring in-house metrics in line with what Appboy was reporting.

Working with another data scientist, I worked on a project (Within Session Personalization) that implemented a model that was trained online in real time. The model existed as a KTable in a Kafka Streams application that updated every time a user viewed a listing on the website. It kept track of the top K listings viewed after a specific listing. When a user viewed a listing, we would query the KTable to recommend the top listings that historically followed a click on that listing.

Senior Software Engineer, Data Engineering

Software Engineer, Data Engineering

My primary responsibilities are all related to web analytics. I perform analytics on data from the company's websites using map-reduce jobs written in Python run over our group's Hadoop cluster and using SQL; I use Mathematica for visualization. I am also responsible for researching and implementing all the advanced statistics and mathematics that our analysis requests often entail. I also develop software tools in Java, Python and Mathematica to improve and expand the group's ability to perform fast and reliable analytics. One major purpose of the software I develop is to make NoSQL data available to Mathematica users. I enjoy Big Data algorithm building that is required for extracting complicated metrics from often incomplete data sets.

Wolfram Research

Web Analyst, Corporate Analysis

Analyst

I'm a graduate student in theoretical physics. I do source-modeling of gravitational-wave sources, mainly those originating from neutron stars. My research involves much coding in Mathematica, and combines research from various sub-fields of physics.

Mohit Nayyar