Spark jobs

Business Process Quality Team Leader

Solutions Architect (Hong-Kong) Singapore Req ID: FEQ427R72 Location: Singapore At Databricks, our core principles are at the heart of everything we do; creating a culture of proactiveness and a customer-centric mindset guides us to create a unified platform that makes data science and analytics accessible to everyone. We aim to inspire our customers to make informed decisions that push their business forward. We provide a user-friendly and intuitive platform that makes it easy to turn insights into action and fosters a culture of creativity, experimentation, and continuous improvement. You will be an essential part of this mission, using your technical expertise to demonstrate how our Data Intelligence Platform can help customers solve their complex data challenges. You'll work with a collaborative, customer-focused team who values innovation and creativity, using your skills to create customized solutions to help our customers achieve their goals and guide their businesses forward. Join us in our quest to change how people work with data and make a better world! The impact you will have: - Form successful relationships with clients in Hong Kong to provide technical and business value in collaboration with an Account Executive and a Senior Solutions Architect. - Gain excitement from clients about Databricks through hands-on evaluation and Apache Spark™ programming, integrating with the wider cloud ecosystem and 3rd party applications. - Contribute to building the Databricks technical community through engagement at workshops, seminars, and meet-ups. - Become a Big Data Analytics advisor on aspects of architecture and design. - Support your

Big Data Laser Analyst Vancouver, Washington, United States of America

SNOW Developer 2 Locations posted: Posted 2 Days Ago

Analytics Engineer Limassol, Lemesos, Cyprus Mayflower is a technology company that alters the entertainment industry to a new level of perception and engagement. We are developing a partner advertising high-load system with a large RPS, convenient and modern interface. We are growing exponentially and successfully implementing new technologies to accelerate this growth and the system as a whole. Now we look for a Analytics Engineer to join our team. Job Responsibilities - Transform business requirements into data models and ETL processes - Develop and maintain data marts (Spark, Iceberg, Clickhouse) using DBT - Write and optimize SQL queries for analytical tasks - Document data models and business logic - Ensure data quality through automated tests - Collaborate with analysts to understand data needs - Optimize ETL pipelines and query performance - Monitor and troubleshoot data pipeline issues - Participate in code reviews and develop team best practices You'll thrive here if you have SQL (advanced level): - Complex analytical queries (window functions, CTEs, subqueries) - Knowledge of OLAP systems specifics - Understanding of execution plans and performance optimization (as a plus) Python (intermediate level): - Data processing (pandas, polars) - Task automation (Airflow) DBT - Building transformation DAGs - Incremental models and snapshots - Model documentation - Data testing experience will be a plus Analytical thinking: - Understanding business processes and their reflection in data - Ability to translate business requirements into technical solutions - Knowledge of metrics and KPIs That can be a plus: - Spark, including PySpark -

Staff Data Engineer Gurugram The Data Engineering team is responsible for building ETL pipelines that populate the Internal Data Platform, which drives analytics that help the company run more efficiently. Our team builds highly performant and scalable processes that extract massive datasets and makes those datasets available for querying in an optimal way. We are looking to speak to candidates who are based in Gurgaon for our hybrid working model. What you'll do - Guide the Data Engineering team on building highly performance ETL pipelines using Spark and other Big Data technologies - Help design the architecture of our Internal Data Platform to support the implementation of a robust medallion architecture - Provide thought leadership on ways to achieve infrastructure cost savings on Cloud hyperscalers - Design and build AI agents that can help automate many of the common development and support tasks that the team performs - Work with Security and Compliance teams to ensure that datasets have appropriate permissions and regulations in place - Work with our Data Platform, and Governance sibling teams to make data scalable, consumable, and discoverable We're looking for someone with - 10+ years experience working on enterprise data lakes/warehouses - 5+ years of Spark and Python experience - 5+ years of direct hands-on experience working with AWS or GCP - Thorough AI knowledge, particularly with codegen tools and agentic frameworks - Hive, Iceberg, Glue, or other technologies that expose big data as tables - Familiarity with different big data file types such as

Growth Intern

Electrical Project Engineer Electrical Project Engineer

Customer Service Specialist (m/f/d) (Bulgaria) - Full-time or Part-time Customer Service Specialist (m/f/d) (Bulgaria) - Full-time or Part-time

Senior Staff Software Engineer, Backend (Data and Storage Services) Remote US

Consultant - Cloud Native Infrastructure & AIOps Consultant - Cloud Native Infrastructure & AIOps

Senior Data Engineer Athens, Attiki, Greece

Partner Solutions Architect Tokyo, Japan FEQ127R214 As a Partner Solutions Architect you will work with Databricks' Consulting and System Integrator (C&SI) partners, teammates, and with the technical and sales team members who work directly with our customers. You will develop 'technical champions' within our top C&SI Partners, providing enablement on technical matters related to the Databricks product. Working with our partners, you will help our customers to achieve tangible data-driven outcomes through the use of our Databricks Data Intelligence Platform, helping data teams complete projects and integrate our platform into their enterprise Ecosystem. As a member of our team, you will exercise and develop expertise in those areas, using open-source projects such as Apache Spark™, MLflow, and Delta Lake; and major public cloud infrastructure and services. the product. You will use this expertise to become a trusted advisor to C&SI partners. You will report to the Senior Director, APJ PS. The impact you will have: - Provide partners with the level of enablement they need to assist their clients in evaluating and adopting Databricks including hands-on Apache Spark™ programming and integration with the wider cloud ecosystem - Engage with the partner technical community by leading workshops, seminars, and meet-ups - You will be a Big Data Analytics expert on aspects of architecture and design and will share this with our partner network - Show expertise by producing creative technical solutions and blog post What we look for: - 5+ years of pre-sales or post-sales experience working with external clients or

UK Infrastructure Development Lead (Renewables and Data Center)

Principal Machine Learning Engineer Hybrid - Denver, Denver, Colorado, United States botta is seeking a Principal Machine Learning Engineer to join our Core Data & Analytics team and contribute to our mission to Make Every Purchase Rewarding. We're looking for someone who has a passion for building scalable machine learning solutions using technologies and tools such as Python, Spark, Databricks, AWS, Sagemaker, etc. As a Principal Machine Learning Engineer, you will act as a technical leader and delivery owner for complex, high-impact ML initiatives spanning foundation models, data systems, and large-scale ML platforms. This role will immediately define ML strategy and drive impact through the support, guidance, and collaboration with multiple teams as part of our analytics organization. You'll advance your technical leadership skills by establishing best practices for the team and company, while mentoring other Machine Learning Engineers and Data Scientists on the team. This position is available in multiple locations: Denver, Colorado - if you are in or willing to relocate to the Denver area, this is a hybrid position requiring 3 days in office (Tuesday, Wednesday, and Thursday). Remote options are available for the following states - AZ, AR, CA, FL, GA, IL, IN, IA, KS, MD, MA, MI, MN, MO, MT, NV, NJ, NY, NC, OH, OK, PA, TN, TX, UT, VA, WA, WI. Not based in one of these areas? We will offer a relocation bonus to help make your move to the Mile High City a smooth one. What you will be doing: -

Senior Software Engineer - Database Engine Internals Mountain View, California Our mission at Databricks is to radically simplify the whole data lifecycle from ingestion to ETL, BI, and all the way up to ML/AI with a unified platform. To achieve this goal, we believe the data warehouse architecture as we know it today will be replaced by a new architectural pattern, Lakehouse ( CIDR 2021 paper ), open platforms that unify data warehousing and advanced analytics. The new architecture will help address several major challenges, including data staleness, reliability, total cost of ownership, data lock-in, and limited use-case support. A critical part of realizing this vision is the next generation (decoupled) query engine and structured storage system that can outperform specialized data warehouses in relational query performance, yet retain the expressiveness and of general purpose systems such as Apache Spark™ to support diverse workloads ranging from ETL to data science. As part of this team, you will be working in one or more of the following areas to design and implement these next gen systems that leapfrog state-of-the-art: - Query compilation and optimization - Distributed query execution and scheduling - Vectorized execution engine - Data security - Resource management - Transaction coordination - Efficient storage structures (encodings, indexes) - Automatic physical data optimization What we look for: - A passion for database systems, storage systems, distributed systems, language design, or performance optimization - Experience working towards a multi-year vision with incremental deliverables - Motivated by delivering customer value and impact

Ink Technician Fremont, California posted: Posted 10 Days Ago

Data Engineer Tampa Slide Insurance - Fun. Innovation Driven. Fueled by Passion, Purpose and Technology. At Slide, you will not only be part of a successful team, but you will also be a part of our Slide Vibe/award winning culture where collaboration and innovation are expected, recognized and awarded! As an Azure Databricks Data Engineer at Slide, you will build and support data pipelines using Azure Data Factory , Databricks , PySpark , and Spark SQL . You'll partner with engineering and business teams to deliver reliable data for reporting, analytics, and operational decision-making. What You'll Do Build Data Pipelines Build and maintain ETL/ELT pipelines using Azure Data Factory and Databricks . Develop PySpark and Spark SQL workloads to ingest, transform, and publish enterprise data. Implement Delta Lake tables using a layered lakehouse architecture. Optimize and Scale Data Solutions Design scalable data models and processing frameworks for batch and incremental workloads. Optimize Spark workloads, SQL queries, and storage solutions for performance and cost efficiency. Implement data quality, monitoring, and observability practices across the platform. Support Data Operations Monitor, troubleshoot, and support production data pipelines. Partner with analytics and business teams to deliver trusted, analytics-ready datasets. Contribute to data engineering standards, governance, and platform improvements. What You Bring Bachelor's degree in Computer Science, Information Systems, Data Engineering, or equivalent experience 3+ years of Data Engineering experience in the Property & Casualty (P&C) insurance industry. Hands-on experience building and supporting ETL/ELT pipelines using Azure Data Factory and Databricks . Strong SQL skills,

Senior Data Engineer (Databricks Focused) 6 Locations posted: Posted 30+ Days Ago

Solutions Architect - Public Sector (SLED) Central - United States FEQ227R359 Mission Solutions Architects at Databricks lead the growth of the Databricks Unified Analytics Platform. As a team, we have expertise in cloud platforms, data engineering, data analytics, and data science and machine learning. As a member of our team, you will exercise and develop expertise in those areas, using open-source projects such as Apache Spark, MLflow, and Delta Lake. This is a customer-facing role, where you will work with customers, your teammates, the product team, and our post-sales teams, to identify use cases for Databricks, develop architectures and solutions using our platform, and guide customers through the implementation, to accomplish value. In that process, you will build relationships with our customers, find and build champions, and become a trusted advisor. You will report to the Field Engineering Manager for the team. The impact you will have: - Partner with the sales team to help customers understand how Databricks can help solve their business problems - Provide technical leadership for customers to evaluate and adopt Databricks - Consult on big data architecture, implement proof of concepts for strategic customer projects, data science and machine learning projects, and validate integrations with cloud services and other 3rd party applications - Build and present references architectures, how-tos, and demo applications for customers - Become an expert in, and promote Databricks inspired open-source projects (Spark, Delta Lake, MLflow, and Koalas) across developer communities through meetups, conferences, and webinars - Travel to customers in your

Sr. Software Development Engineer – Big Data, AWS Elastic MapReduce (EMR), EMR EKS East Palo Alto, California, USA Want to change the world with Big Data and Analytics? Come join us on the Amazon Web services (AWS) Elastic MapReduce (EMR) team! EMR is a massively scaled distributed service that gives users fast and predictable Big Data performance with clusters running Hadoop, Hive, Pig, Impala, Spark, Presto, HBase and more, with the ability to effortlessly scale up and down as needed. We run millions of customer clusters, enabling processing on vast datasets. We are developing innovative new features including the next-generation cluster management system, real-time processing for big data and intuitive ways of interacting with data, and we're looking for top engineers to build them from the ground up. This is a hands-on position where you will be do everything from designing and building extremely stable components and features for the savviest customers in the business to help them get the best results. You will have a chance to work with the open source community and contribute significant portions its software to open source projects possibly including Hadoop, Spark, Pig and Hbase. You need to not only be a top software developer with excellent programming skills, an understanding of big data and parallelization, and a stellar record of delivery but also excel at leadership and customer obsession and have a real passion for massive-scale computing. If you want to truly test your mettle against the hardest challenges in distributed systems to

Engineering Booster Program Melaka (Malaysia)

Databricks Administrator / Platform Engineer Any Location / Remote posted: Posted 21 Days Ago

Specialist Solutions Architect Seoul, South Korea FEQ227R130 As a Specialist Solutions Architect (SSA), you will guide customers in building big data solutions on Databricks that span a large variety of use cases. These are customer-facing roles, working with and supporting the Solution Architects, requiring hands-on production experience with Apache Spark™ and expertise in other data technologies. SSAs help customers through design and successful implementation of essential workloads while aligning their technical roadmap for expanding the usage of the Databricks Lakehouse Platform. As a deep go-to-expert reporting to the Director of Field Engineering, you will continue to strengthen your technical skills through mentorship, learning, and internal training programs and establish yourself in an area of specialty - whether that be performance tuning, machine learning, industry expertise, or more. The impact you will have: - Provide technical leadership to guide strategic customers to successful implementations on big data projects, ranging from architectural design to data engineering to model deployment - Architect production level workloads, including end-to-end pipeline load performance testing and optimisation - Provide technical expertise in an area such as data management, cloud platforms, data science, machine learning, or architecture - Assist Solution Architects with more advanced aspects of the technical sale including custom proof of concept content, estimating workload sizing, and custom architectures - Improve community adoption (through tutorials, training, hackathons, conference presentations) - Contribute to the Databricks Community What we look for: - You will have experience in a customer-facing technical role with expertise in at least one of