Dhruvi Pandya，印度马哈拉施特拉邦孟买的开发商
Dhruvi is available for hire
Hire Dhruvi

Dhruvi Pandya

Verified Expert  in Engineering

数据工程师和软件开发人员

Location
Mumbai, Maharashtra, India
Toptal Member Since
September 1, 2023

Dhruvi是一名数据工程专业人士，拥有7年的行业经验，最初是一名后端开发人员. Her expertise lies in building, maintaining, and optimizing data pipelines, mostly with stacks such as Spark, Airflow, Storm, Snowplow, Docker, and Kafka. 她对AWS服务(如EC2、EMR、Elastic Beanstalk、Athena和Redshift)有实践经验. Dhruvi还精通敏捷开发和冲刺计划方法.

Big DataData EngineeringData AnalysisData ModelingData VisualizationData Warehouse DesignData WarehousingGitDatabasesPython 3PythonApache AirflowDockerApache KafkaETLDBT (Data Build Tool) MetabaseProtractorOnline Transaction Processing

Portfolio

Saltside Technologies
ETL, Redshift, Amazon Elastic MapReduce (EMR)， AWS Elastic Beanstalk, Python 3...
AccionLabs
Python, JavaScript, Angular, Node.. js, MongoDB, PostgreSQL, Apache气流...
AccionLabs
Node.js, MongoDB, Express.. js, AngularJS, Karma, Protractor, MySQL, Jenkins...

Experience

Git - 7 yearsPython 3 - 5 yearsApache Airflow - 5 yearsApache Kafka - 4 yearsETL - 4 yearsDocker - 4 yearsBig Data - 4 yearsSpark - 3 years

Availability

Part-time

Preferred Environment

Linux, Visual Studio Code (VS Code)， Git

The most amazing...

...我所做的就是优化在Amazon EMR上运行的Spark作业，并将成本降低了25%.

Work Experience

Senior Data Engineer

2021 - PRESENT
Saltside Technologies
  • 与团队合作，使用Snowplow Analytics等工具将Lambda架构迁移到流架构, Apache Kafka, and Apache Storm.
  • 构建并发布了一个后端服务，该服务公开了从Redis缓存中获取用户(卖家和买家)统计数据的API.
  • 优化了长时间运行且消耗资源的AWS EMR Spark作业，降低了25%的成本.
  • 为AWS Redshift数据仓库平台构建和部署基于sql的elt，并创建聚合表以支持Tableau中的KPI仪表板.
Technologies: ETL, Redshift, Amazon Elastic MapReduce (EMR)， AWS Elastic Beanstalk, Python 3, Apache Airflow, Snowplow Analytics, Data Engineering, Python, Documentation, Data Architecture, Data Warehouse Design, SQL, Database Modeling, Jupyter Notebook, OLAP, Amazon Web Services (AWS), Communication, Real-time Streaming, ELK (Elastic Stack), PyCharm, Debian Linux, Docker Hub, PostgreSQL 9, Scripting, MapReduce, Data Analytics, Data Warehousing, Big Data Architecture, Solution Architecture, Data Analysis, Apache Kafka, Lambda Architecture, Streaming

Senior Developer

2020 - 2021
AccionLabs
  • 帮助将基于sql的管道迁移到Spark, 提高数据仓库中可用数据的质量和速度.
  • 设计端到端管道，将产品推荐系统所需的数据导入数据集市. 使用Apache气流构建数据处理管道，通过AWS Lambda公开推荐数据.
  • 创建和维护Spark流管道，为数据仓库提供动力. 与数据分析师合作，在他们的基础上构建报告.
技术:Python, JavaScript, Angular, Node.. js, MongoDB, PostgreSQL, Apache气流, AWS Lambda, Unit Testing, Data Analysis, Spark, Spark Streaming, ETL, Data Engineering, SQL, PySpark, Jupyter Notebook, Apache Spark, Amazon Web Services (AWS), Communication, Real-time Streaming, Pytest, Debian Linux, Docker Hub, Scripting, Data Analytics, OLTP, PL/SQL, Database Architecture, Data Migration, Big Data Architecture, Solution Architecture

Full-stack Developer

2018 - 2020
AccionLabs
  • 与一个团队合作，为一个主要的云提供商客户开发10多个内部应用程序——每个应用程序在单元和端到端测试用例上都有80%以上的测试覆盖率.
  • 与团队一起构建一个通用后端，该后端可以为在MongoDB中创建新集合和所有必要的CRUD操作端点提供认证和基本端点. 这将后端开发时间减少了近80%.
  • 与现场团队协调跨团队项目和敏捷冲刺计划.
Technologies: Node.js, MongoDB, Express.. js, AngularJS, Karma, Protractor, MySQL, Jenkins, Python, Flask-RESTful, Object-relational Mapping (ORM), Communication, OLTP, Database Architecture

Front-end Developer

2016 - 2018
AccionLabs
  • 与一个团队一起为一家初创公司开发安全监控前端. 这包括通过基于套接字的通信显示来自摄像机的实时馈送，并显示用于入侵者检测的实时边界框.
  • 将一个正在进行的项目中40%的过时组件迁移到Angular 5.
  • 记录了80%正在进行的项目遗留代码.
Technologies: Angular, React, Protractor, Selenium, HTML, CSS, CSS3, HTML5, JavaScript, Communication

CDC Pipeline

http://github.com/dhruvip/kafka-connect-cdc
使用Kafka开发了一个CDC管道. 它使用Kafka堆栈从MySQL关系数据库引入实时数据流, 特别是Kafka Connect和Kafka集群, 并将数据转储到Elasticsearch集群中.

Mock Retail Store Analytics

端到端数据管道，为模拟零售商店的分析仪表板提供支持.

这是我的一个个人项目，我展示了我的技能:
•AWS等云平台上的数据架构
•构建强大的数据管道，并与气流协调, Bash Scripting, Python, and plain SQL
•使用Python、Pandas和jupiter进行数据处理和探索性分析
•使用dbt进行数据建模和创建dim/事实表
•数据可视化和仪表板与Metabase

我收集了以下几点见解:
•表现最佳的营销活动
•表现最佳国家的十大创收产品

Fast Food Chain Location Analysis

基于Jupyter notebook的探索性数据分析，以了解成功的食物链所选择的位置与其附近地标之间的相关性. 这有助于新餐馆老板了解这些蓬勃发展的企业在寻找地点时寻找的是什么.

Languages

Python 3, Python, SQL, c++， JavaScript, HTML, CSS, CSS3, HTML5, Scala

Frameworks

Spark, Apache Spark, Angular, Express.js、AngularJS、Protractor、Selenium、Jinja

Libraries/APIs

Pandas, PySpark, Spark Streaming, Node.js、Matplotlib、Flask-RESTful、React

Tools

Git, Apache Airflow, Docker Compose, Snowplow Analytics, PyCharm, Amazon Elastic MapReduce (EMR), Seaborn, Karma, Jenkins, ELK (Elastic Stack), Pytest, Docker Hub, Plotly

Paradigms

ETL, OLAP, Data Science, Unit Testing, Object-relational Mapping (ORM), MapReduce, Lambda Architecture

Platforms

Docker, Apache Kafka, Linux, Jupyter Notebook, Amazon Web Services (AWS), Debian Linux, AWS Lambda, Visual Studio Code (VS Code), AWS Elastic Beanstalk

Storage

Databases, MySQL, Redshift, Database Modeling, OLTP, Database Architecture, NoSQL, MongoDB, PostgreSQL, Apache Hive, PL/SQL

Other

Big Data, Data Analysis, Data Visualization, Shell Scripting, Data Modeling, Data Engineering, Documentation, Data Warehouse Design, Communication, Real-time Streaming, Scripting, Data Analytics, Data Warehousing, Data Structures, Data Migration, Big Data Architecture, Solution Architecture, Algorithms, APIs, Machine Learning, CDC, Data Architecture, Metabase, Data Build Tool (dbt), Cosmos, PostgreSQL 9, Streaming, Macros

2012 - 2016

计算机工程学士学位

艾哈迈达巴德大学-艾哈迈达巴德，印度

SEPTEMBER 2023 - SEPTEMBER 2025

dbt Fundamentals

dbt

JULY 2019 - PRESENT

IBM数据科学专业证书

IBM | via Coursera

Collaboration That Works

How to Work with Toptal

在数小时内，而不是数周或数月，我们的网络将为您直接匹配全球行业专家.

1

Share your needs

在与Toptal领域专家的电话中讨论您的需求并细化您的范围.
2

Choose your talent

在24小时内获得专业匹配人才的简短列表，以进行审查，面试和选择.
3

Start your risk-free talent trial

与你选择的人才一起工作，试用最多两周. 只有当你决定雇佣他们时才付钱.

Top talent is in high demand.

Start hiring