Spark jobs

BIE II, ShipTech Analytics, ShipTech Analytics Hyderabad, Telangana, IND Do you want to be in the forefront of engineering big data analytical solutions that takes Amazon's global transportation models to the next generation? Do you have a solid analytical thinking, metrics driven decision making and want to solve problems with solutions that will meet the growing worldwide need? We are looking for top notch Business Intelligence Engineers to be part of ShipTech Analytics team to build data and analytics solutions powering Amazon's global transportation network. We perform data modeling, build real time analytical platforms using big data tools and AWS technologies like Hadoop, Spark, EMR, SNS, SQS, Lambda, Kinesis Firehose, DynamoDB Streams. As a BIE II, you'll work on building analytical solutions that empower operations teams worldwide. You'll develop both near-real-time (NRT) streaming pipelines and batch processing pipelines that deliver critical insights for decision-making. You'll also build metrics for the transportation network, while contributing to innovative AI-powered solutions delivering automated insights across the transportation network. The ideal candidate relishes working with large volumes of data, enjoys the challenge of highly complex technical contexts, and, above all else, is passionate about data and analytics. The candidate is an expert with data modeling, ETL design and business intelligence tools and passionately partners with the business to identify strategic opportunities where improvements in data infrastructure creates out-sized business impact. The candidate is a self-starter, comfortable with ambiguity, able to think big (while paying careful attention to detail), and enjoys working in a fast-paced

Senior Software Developer - Oracle Health, US citizenship required United States, US We're looking for highly skilled AI engineers to design and build high-scale, cloud-based data processing pipelines that ingest, transform, and analyze massive volumes of healthcare data with low latency, powering business insights and analytics across EHR and RCM systems. You will leverage LLMs, AI agents, and modern data platforms to solve problems like clinical decision support, revenue optimization, and workflow automation while using AI-assisted development tools to accelerate delivery. Responsibilities As a member of the software engineering division, you will assist in defining and developing software for tasks associated with the developing, debugging or designing of software applications or operating systems. Provide technical leadership to other software developers. Specify, design and implement modest changes to existing software architecture to meet changing needs. Key Responsibilities Build and enhance data pipelines, ETL workflows, and transformations. Contribute to LLM/agent-based features and analytics use cases. Work with EHR/RCM datasets and support KPI/dashboard development. Learn and apply best practices in cloud, data engineering, and LLMOps. Mandatory Qualifications: BS/MS in in Computer Science or equivalent. 5+ years of relevant software engineering experience. Strong software engineering skills in Python/Java. Strong knowledge of SQL. Deep expertise in data engineering: ETL, data transformation, data modeling (Spark, SQL) Experience building high-scale distributed data systems. Cloud experience (OCI/AWS/Azure). Experience with creating major new functionality in a software system all the way from design, through development and testing to production deployment. Experience with collaborating across multiple functional areas to develop

Lead Data Scientist 3 Locations posted: Posted Yesterday

Principal Member of Technical Staff United States, US We're looking for highly skilled AI engineers to design and build high-scale, cloud-based data processing pipelines that ingest, transform, and analyze massive volumes of healthcare data with low latency, powering business insights and analytics across EHR and RCM systems. You will leverage LLMs, AI agents, and modern data platforms to solve problems like clinical decision support, revenue optimization, and workflow automation while using AI-assisted development tools to accelerate delivery. Responsibilities As a member of the software engineering division, you will assist in defining and developing software for tasks associated with the developing, debugging or designing of software applications or operating systems. Provide technical leadership to other software developers. Specify, design and implement modest changes to existing software architecture to meet changing needs. Key Responsibilities Design and develop scalable data pipelines and AI-driven workflows. Build LLM/agent-based solutions for business use cases (revenue leakage, readmissions, automation). Own end-to-end features from data ingestion through transformation and on to insights. Optimize systems for performance, scale, and low latency. Mentor junior engineers and contribute to design decisions. Mandatory Qualifications: BS/MS in in Computer Science or equivalent. 8+ years of relevant software engineering experience. Strong software engineering skills in Python/Java. Strong knowledge of SQL. Deep expertise in data engineering: ETL, data transformation, data modeling (Spark, SQL) Experience building high-scale distributed data systems. Cloud experience (OCI/AWS/Azure). Experience with creating major new functionality in a software system all the way from design, through development and testing to production deployment. Experience

Snowflake Data Engineer PUNE, Mahārāshtra, India

Senior Applied Scientist, AWS Central Economics and Science San Francisco, California, USA AWS is one of Amazon's largest and fastest growing businesses, serving millions of customers in more than 190 countries. We use cloud computing to reshape the way global enterprises use information technology. We are looking for entrepreneurial, analytical, creative, flexible leaders to help us redefine the information technology industry. If you want to join a fast-paced, innovative team that is making history, this is the place for you. AWS Central Economics & Science (ACES) drives best practices for objectively applying economics and science in decision making across AWS. The team collaborates with AWS science and business teams to identify, frame, and analyze complex and ambiguous problems of the highest priority. Through data-driven insights and modeling, ACES supports strategic decision-making across the AWS global organization, including sales operations and business performance optimization. The ACES Sales Channels team is hiring an Applied Scientist (Senior or below) to advance our mission of providing rigorous, causal-inference-driven recommendations for AWS sales optimization. This role will focus on building ML systems with a causal modeling foundation, designing seller incentive mechanisms, and developing intervention strategies across the entire sales motion. Key job responsibilities • Causal ML System Development: Build and deploy machine learning models that emphasize causal inference, ensuring recommendations are grounded in valid interventions • Incentive Design: Define and model incentives that drive desirable behaviors across AWS sales channels, partner programs, and reseller ecosystems • Stakeholder Collaboration: Work with business stakeholders to understand requirements,

Senior Data Architect, Semantic Layer & Gen AI Readiness Senior Data Architect, Semantic Layer & Gen AI Readiness

Director, Data Engineering — Customer Success Score California - San Francisco; Washington - Seattle About the Role We're building data products that will define Salesforce's next era of agentic intelligence - powering smarter, adaptive, and self-optimizing product experiences at enterprise scale. As Director of Data Engineering, you'll set the strategic direction, architectural vision, and organizational execution for the data systems behind Customer Success Score (CSS), one of Salesforce's most critical product intelligence assets. This is a high-visibility leadership role for a strategic technologist who thrives at the intersection of data, AI, and platform engineering. What You'll Do Platform Strategy & Roadmap Own the roadmap for the CSS data platform, aligning engineering investment with Salesforce's agentic and AI-first product strategy Establish architectural principles and governance standards for telemetry, semantic modeling, and metadata-driven discovery at enterprise scale Drive convergence across product analytics, ML infrastructure, and AI data foundations - breaking down silos and creating shared organizational leverage Represent data engineering at the executive level; shape organizational priorities and secure resources for strategic initiatives Technical Architecture & Excellence Set and uphold the technical bar for distributed data systems - including fault-tolerant batch and streaming architectures (Spark, Trino, Flink, Kafka, dbt, Snowflake) Define engineering standards across software quality, CI/CD, observability, and reliability Guide platform evolution to support autonomous agent reasoning, real-time adaptive decisioning, and AI-native product experiences Champion semantic consistency, metric governance, and trusted signal definition across the organization Evaluate emerging technologies and drive adoption decisions that extend the platform's strategic value Organizational Leadership

Director, Data Architecture 12 Locations posted: Posted 4 Days Ago

​​Senior MLOps Engineer​ 2 Locations posted: Posted 30+ Days Ago

Data Software Engineer Krakow What your day could consist of: - Architect and Build Pipelines by designing, developing, and maintaining high-throughput data pipelines that seamlessly transform data across our ecosystem - Execute Data Modeling and Schema Design , utilizing DDL and DML operations to optimize data structures for both performance and long-term scalability - Champion Engineering Excellence by implementing automated testing, CI/CD pipelines, and Infrastructure as Code (Terraform) to ensure system health and reliability . - Collaborate Cross-Functionally with Data Science and AI teams to engineer feature stores and automated MLOps pipelines, moving cutting-edge research into live production environments - Drive System Monitoring and health checks, ensuring our containerized and orchestrated environments (Kubernetes, Docker, Airflow) run at peak efficiency . - Act as a Technical Advisor to stakeholders, translating complex data architectures into clear, non-technical conversations that guide business decisions Technical environment: - Databases: MySQL, Amazon Aurora, Amazon RDS, PostgreSQL - ata Storage and Analytics: AWS S3, Apache Iceberg, Snowflake, Apache Druid, Looker - Data Processing and Transformation: Apache Spark, DBT - Languages: Python (primary) - Infrastructure as Code: Terraform - Containerization and Orchestration: Kubernetes, Docker, Airflow The Ideal Candidate Will Bring: - 4+ years of professional software engineering experience , with a proven track record of building and maintaining production-grade data systems . - Deep Data Architecture expertise , with the ability to design modern data stack components and schemas that balance immediate performance with future growth . - Proficiency in Infrastructure as Code , specifically using Terraform to

Senior Applied Scientist, Full-funnel Agentic Intelligence and Models Palo Alto, California, USA Amazon Advertising is one of Amazon's fastest growing and most profitable businesses. Amazon's advertising portfolio helps merchants, retail vendors, and brand owners succeed via native advertising, which grows incremental sales of their products sold through Amazon. The primary goals are to help shoppers discover new products they love, be the most efficient way for advertisers to meet their business objectives, and build a sustainable business that continuously innovates on behalf of customers. Our products and solutions are strategically important to enable our Retail and Marketplace businesses to drive long-term growth. We deliver billions of ad impressions and millions of clicks and break fresh ground in product and technical innovations every day! Amazon continues to develop its advertising program. Ads run in our Stores (including Consumer Stores, Books, Amazon Business, Whole Foods Market, and Fresh) and Media and Entertainment publishers (including Fire TV, Fire Tablets, Kindle, Alexa, Twitch, Prime Video, Freevee, Amazon Music, MiniTV, Audible, IMDb, and others). In addition to these first-party (1P) publishers, we also deliver ads on third-party (3P) publishers. We have a number of ad products, including Sponsored Products and Sponsored Brands, display and video products for smaller brands, including Sponsored Display and Sponsored TV. We also operate ad tech products, including Amazon Marketing Cloud (a clean-room for advertisers), Amazon Publisher Cloud (a clean-room for publishers), and Amazon DSP (an enterprise-level buying tool that brings together our ad tech for buying video, audio, and display

Senior Cloud Engineer Warszawa,Mazowieckie,Poland

Data Platform Engineer Hill AFB, UT At Accenture Federal Services, nothing matters more than helping the US federal government make the nation stronger and safer and life better for people. Our 13,000+ people are united in a shared purpose to pursue the limitless potential of technology and ingenuity for clients across defense, national security, public safety, civilian, and military health organizations. Join Accenture Federal Services, a technology company within global Accenture. Recognized as a Glassdoor Top 100 Best Place to Work, we offer a collaborative and caring community where you feel like you belong and are empowered to grow, learn and thrive through hands-on experience, certifications, industry training and more. Join us to drive positive, lasting change that moves missions and the government forward! Accenture Federal Services is seeking a Data Platform Engineer to join our secure cloud platform team supporting government customers at Hill Air Force Base. This role is execution-focused with strong data platform engineering depth - you will design, build, and operate the streaming and batch data pipelines, lakehouse patterns, and platform services that enable rapid, secure delivery across a complex government program. The ideal candidate brings hands-on GitOps and infrastructure-as-code expertise, is comfortable in classified multi-network environments, and embodies extreme ownership. What Success looks like: - Data pipelines are reliable, automated, and instrumented with observability and quality gates. - Platform services (Kafka, Airflow, Spark, EMR/Glue/Redshift) are standardized and version-controlled. - Infrastructure is defined as code and operated through GitOps - no snowflake systems. - Security and

Lean Engineer Reynosa, MX posted: Posted 15 Days Ago

Data Scientist II - AMZ9774210 New York, New York, USA MULTIPLE POSITIONS AVAILABLE Employer: AMAZON DEVELOPMENT CENTER U.S., INC. Offered Position: Data Scientist II Job Location: New York, New York Job Number: AMZ9774210 Position Responsibilities: Design and implement scalable and reliable approaches to support or automate decision making throughout the business. Apply a range of data science techniques and tools combined with subject matter expertise to solve difficult business problems and cases in which the solution approach is unclear. Acquire data by building the necessary SQL / ETL queries. Import processes through various company specific interfaces for accessing Oracle, RedShift, and Spark storage systems. Build relationships with stakeholders and counterparts. Analyze data for trends and input validity by inspecting univariate distributions, exploring bivariate relationships, constructing appropriate transformations, and tracking down the source and meaning of anomalies. Build models using statistical modeling, mathematical modeling, econometric modeling, network modeling, social network modeling, natural language processing, machine learning algorithms, genetic algorithms, and neural networks. Validate models against alternative approaches, expected and observed outcome, and other business defined key performance indicators. Implement models that comply with evaluations of the computational demands, accuracy, and reliability of the relevant ETL processes at various stages of production. Position Requirements: Master's degree or foreign equivalent degree in Statistics, Applied Mathematics, Economics, Engineering, Computer Science, or a related field and one year of experience in the job offered or a related occupation. Employer will accept a Bachelor's degree or foreign equivalent degree in Statistics, Applied Mathematics, Economics, Engineering, Computer

HV Lead Engineer - Renewable Energy

Sr. Applied Scientist, Alexa Excellence AI Ops, Alexa Excellence AI Ops Chennai, Tamil Nadu, IND We are seeking a Senior Applied Scientist to join the Alexa Availability team within Alexa Excellence. This role leads the research and development of machine learning and statistical models that power Alexa's reliability at massive scale - serving hundreds of millions of customers globally. The ideal candidate will tackle complex, ambiguous problems spanning time series multivariate modeling, statistical anomaly detection, LLM-based operational intelligence, and adaptive threshold systems. They will design production-grade ML solutions, establish rigorous evaluation frameworks, and ensure AI systems are grounded, reliable, and free from systematic bias - leveraging techniques such as RAG, confidence scoring, knowledge graph integration, and counterfactual testing. This scientist will partner with engineers, product managers, and operations leaders to translate scientific innovation into production systems that directly impact Alexa's availability worldwide. They will drive the scientific agenda for the team, mentor fellow scientists, and influence the broader Alexa Excellence organization through technical leadership and cross-team collaboration. Key Focus Areas: Anomaly detection and predictive failure modeling Cross-service correlation and LLM-driven operational intelligence Production ML at the intersection of large-scale distributed systems and applied science Model reliability, hallucination mitigation, and grounding for operational AI Key job responsibilities As a Senior Applied Scientist on the Alexa Availability team, you will lead the research and development of machine learning and statistical models that power Alexa's reliability at scale. You will work on some of the most complex and ambiguous problems in the space

Lead, Customer Advocacy Remote US

Senior Machine Learning Platform/Ops Engineer Barcelona, Catalunya, Spain We power people's progress. At Preply, we're all about creating life-changing learning experiences. We help people discover the magic of the perfect tutor, craft a personalised learning journey, and stay motivated to keep growing. Our approach is human-led, tech-enabled - and it's creating real impact. We've just reached unicorn status with a $150M Series D, accelerating our vision to transform education through human-led, AI-enhanced learning. Today, 100,000+ tutors teach 90+ languages to learners in 180 countries - and we're only getting started. As a category-defining company, we're shaping what the future of learning looks like at global scale. Every Preply lesson sparks change, fuels ambition, and drives progress that matters. Joining Preply means helping define the future of education at global scale, and building something that truly matters for millions of people, every day. About the role As Preply scales its AI-powered learning platform, we're looking for an experienced Senior ML Platform/Ops Engineer to help productionize machine learning systems with high reliability, performance, and observability. You'll work at the intersection of ML, data engineering, and cloud infrastructure enabling fast, secure, and reproducible model development from training to deployment. We've reached 90%+ adoption of AI coding tools across engineering, and we're now moving towards more autonomous, AI-Augmented development at scale. At Preply, engineers have direct access to the best tools available, with the freedom to use them fully and experiment as they build. You'll collaborate closely with ML Scientists, Backend Engineers, and Data

AIML - Staff ML Infrastructure Engineer, ML Platform & Technology - Pre-training Infrastructure San Francisco Bay Area, United States of America Apple is where individual imaginations gather together, committing to the values that lead to great work. Every new product we build, service we create, or Apple Store experience we deliver is the result of us making each other's ideas stronger. That happens because every one of us shares a belief that we can make something wonderful and share it with the world, changing lives for the better. It's the diversity of our people and their thinking that inspires the innovation that runs through everything we do. When we bring everybody in, we can do the best work of our lives. Here, you'll do more than join something - you'll add something! As an engineer on ML Compute team, your work will include: - Drive large-scale pre-training initiatives to support cutting-edge foundation models, focusing on resiliency, efficiency, scalability, and resource optimization. - Enhance distributed training techniques for foundation models. - Research and implement new patterns and technologies to improve system performance, maintainability, and design. - Optimize execution and performance of workloads built with JAX, PyTorch, XLA and CUDA on large distributed systems. - Leverage high-performance networking technologies such as NCCL for GPU collectives and TPU interconnect (ICI/Fabric) for large-scale distributed training. - Architect a robust MLOps platform to streamline and automate pretraining operations. - Operationalize large-scale ML workloads on Kubernetes, ensuring distributed trainings are robust, efficient, and fault-tolerant. - Lead

Sr. Applied Scientist, Ads AI Core Infrastructure New York, New York, USA Amazon Advertising is one of Amazon's fastest growing and most profitable businesses, responsible for defining and delivering AI-powered solutions that transform how advertisers make strategic decisions. We deliver billions of ad impressions and process massive volumes of advertiser data every single day. You'll work with us to pioneer breakthrough approaches in how AI agents access and reason over real-time advertiser data at scale. We are using generative AI and agentic systems to help advertising agents provide instant, strategic advice to millions of advertisers. You will need to invent new techniques for agent orchestration, context optimization, and code generation to ensure we're delivering accurate, trustworthy insights with minimal latency and token consumption. You'll create feedback loops to ensure our solutions are constantly evaluating themselves and improving. The Ads Real-Time Data Service team is seeking an exceptional Applied Scientist to research and develop novel approaches for agent-data interaction. The Ads Real-Time Data Service team is solving one of the most critical challenges in advertising AI: instant access to advertiser context. We're building the infrastructure that provides immediate, pre-computed access to advertiser data via Model Context Protocol (MCP) servers-an emerging standard for AI agent-data interaction. We're building summarized data for context using a mix of state of the art techniques like CodeAct and RAG-based embeddings, achieving a fundamental transformation in how AI agents interact with data. This role balances applied research (60%) with productionization (40%), giving you the opportunity to both

Data Scientist II - AMZ24268.1 New York, New York, USA Employer: Metro-Goldwyn-Mayer Studios Inc. Position: Data Scientist II Location: New York, NY Multiple Positions Available: Design and implement scalable and reliable approaches to support or automate decision making throughout the business. Apply a range of data science techniques and tools combined with subject matter expertise to solve difficult business problems and cases in which the solution approach is unclear. Acquire data by building the necessary SQL / ETL queries. Import processes through various company specific interfaces for accessing Oracle, RedShift, and Spark storage systems. Build relationships with stakeholders and counterparts. Analyze data for trends and input validity by inspecting univariate distributions, exploring bivariate relationships, constructing appropriate transformations, and tracking down the source and meaning of anomalies. Build models using statistical modeling, mathematical modeling, econometric modeling, network modeling, social network modeling, natural language processing, machine learning algorithms, genetic algorithms, and neural networks. Validate models against alternative approaches, expected and observed outcome, and other business defined key performance indicators. Implement models that comply with evaluations of the computational demands, accuracy, and reliability of the relevant ETL processes at various stages of production. (40 hours / week, 8:00am-5:00pm, Salary Range $169541 - $207500) Amazon.com is an Equal Opportunity - Affirmative Action Employer - Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation Basic Qualifications: Master's degree or foreign equivalent degree in Statistics, Applied Mathematics, Economics, Engineering, Computer Science, or a related field and one year of experience in the

Data Center Critical Facilities Junior 2 Locations posted: Posted 8 Days Ago