Spark jobs

Senior Applied Scientist, JP Science and Data - AVS ProServe Tokyo, JPN We are seeking a talented, customer-focused Senior Applied Scientist to join the AVS ProServe Science and Data Team. In this role, you will develop and apply machine learning algorithms to solve ambiguous business problems, leading the roadmap to help vendors understand their customers, identify opportunities for growth and define long-term strategy. The ideal candidate brings deep expertise in machine learning, interpretable models, transformers, and experimentation, along with the business acumen to translate problems into scalable science solutions. We're looking for a self-starter with an entrepreneurial spirit who is comfortable with ambiguity, demonstrates strong attention to detail, and thrives in a fast-paced, data-driven environment - with a passion for driving measurable impact. Key job responsibilities • Develop customer understanding models by designing and building orchestrated ML solutions that enable vendors to deeply understand their customers and uncover growth opportunities • Bridge science and business strategy by translating model outputs into actionable insights and recommendations that inform vendor growth strategies, customer acquisition, and long-term planning • Measure and validate business impact by closing the loop between science solutions and business outcomes - establishing measurement frameworks that quantify impact, surface new opportunities, and continuously refine the path to vendor growth • Lead cross-functional collaboration by working with engineers, scientists, consultants, and business leaders to deploy scalable solutions while communicating complex technical concepts clearly to non-technical audiences • Stay at the forefront of innovation by applying state-of-the-art techniques in machine learning, interpretable

Staff Software Engineer, Compute Infrastructure Bellevue, Washington RDQ427R175 At Databricks, we are passionate about helping data teams solve the world's toughest problems - from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to improve their business. Founded by engineers - and customer obsessed - we leap at every opportunity to solve technical challenges, from designing next-gen UI/UX for interfacing with data to scaling our services and infrastructure across millions of virtual machines. And we're only getting started. At Databricks, the Compute Infrastructure organization builds and operates the foundation that runs all Data, AI and stateful workloads across all major clouds. Our platform launches tens of millions of VMs per day, operates thousands of Kubernetes clusters, and must deliver extreme elasticity, reliability and cost efficiency. As a Staff Software Engineer and Tech Lead on the Compute Infra team, you will design and build the systems that power Databricks' compute infrastructure to enable engineers to quickly launch and scale world-class products. The impact you will have: - Develop the compute abstractions that provide powerful capabilities for all Databricks workloads, enabling engineers to build world-class products with high velocity and best-in-class performance - Design the workload orchestration and scheduling systems that orchestrates all types of workloads (serving, batch, stateful, GPU) with high performance and efficiency - Scale the fleet management systems that launch

Databricks Data Engineering Manager Location not specified

Engineering Manager- Data Platform Bangalore, Karnataka About the Role We are looking for a seasoned Engineering Manager to lead our Data Platform. You will own the architecture and evolution of a Petabyte-scale Data Lakehouse and a Self-serve Analytics Platform, enabling real-time decision-making across the organization. In this role, you will drive consistency and quality by defining the right engineering strategies. You will oversee multiple engineering projects, ensure timely execution, collaborate across functions, and mentor engineers to grow into high-performing contributors. About the Role We are looking for a seasoned Engineering Manager to lead our Data Platform. You will own the architecture and evolution of a Petabyte-scale Data Lakehouse and a Self-serve Analytics Platform, enabling real-time decision-making across the organization. In this role, you will drive consistency and quality by defining the right engineering strategies. You will oversee multiple engineering projects, ensure timely execution, collaborate across functions, and mentor engineers to grow into high-performing contributors. What You Will Do - Drive technical roadmap and architecture for driving efficiency across the platform along with automated data governance. - Define and monitor SLOs/SLAs for data availability, latency, and quality across real-time and batch pipelines. - Drive excellence in engineering quality and lead solutioning for complex product problems - Collaborate with Analytics, Data Science and Product teams to drive adoption of the Self-Serve Platform and reduce time to high quality insights. - Champion robust data security standards, including RBAC, PII masking, and encryption, to ensure DPDP compliance in a democratized data environment. - Manage

On-Device ML Infrastructure Engineer (CoreML Runtime), Graphics, Games and Machine Learning Cupertino, United States of America Imagine being at the forefront of an evolution where powerful AI meets the elegance of Apple silicon. The On-Device Machine Learning team transforms groundbreaking research into practical applications, enabling billions of Apple devices to run powerful AI models locally, privately, and efficiently. We stand at the unique intersection of research, software engineering, hardware engineering, and product development, making Apple a top destination for on-device machine learning innovation. Our team builds the essential infrastructure that enables machine learning at scale on Apple devices. This involves onboarding innovative architectures to embedded systems, developing optimization toolkits for model compression and acceleration, building ML compilers and runtimes for efficient execution, and creating comprehensive benchmarking and debugging toolchains. This infrastructure forms the backbone of Apple's machine learning workflows across Camera, Siri, Health, Vision, and other core experiences, contributing to the overall Apple Intelligence ecosystem. If you are passionate about the technical challenges of running sophisticated ML models on resource-constrained devices and eager to directly impact how machine learning operates across the Apple ecosystem, this role presents an incredible opportunity to work on the next generation of intelligent experiences on Apple platforms. We are seeking an ML Infrastructure Engineer with a specific focus on graph compilers and runtimes. If you are a highly motivated software engineer who is creative, versatile, and passionate about machine learning operator primitives, common compiler optimizations, runtimes, and system software engineering in the fast-paced and dynamic

Full Stack Python Engineer Washington, DC At Accenture Federal Services, nothing matters more than helping the US federal government make the nation stronger and safer and life better for people. Our 13,000+ people are united in a shared purpose to pursue the limitless potential of technology and ingenuity for clients across defense, national security, public safety, civilian, and military health organizations. Join Accenture Federal Services, a technology company within global Accenture. Recognized as a Glassdoor Top 100 Best Place to Work, we offer a collaborative and caring community where you feel like you belong and are empowered to grow, learn and thrive through hands-on experience, certifications, industry training and more. Join us to drive positive, lasting change that moves missions and the government forward! The work: Key Responsibilities: Technical Leadership & Code Ownership - Set technical direction for an 8-10 person engineering team. - Serve as primary architect and code-quality gatekeeper (review/merge all GitHub PRs). - Define and enforce standards for architecture, performance, testing, documentation, and CI/CD. - Own the end-to-end codebase for a distributed forecasting platform built on Python and Apache Spark (GCP). - Champion and evolve the internal ML tooling library used for training, evaluation, metrics, and visualization. Forecasting Platform Development - Build and maintain scalable data pipelines and feature engineering workflows for intermittent and lumpy MRO/spare‑parts demand. - Extend and improve metrics instrumentation (sMAPE, MASE, Dollar‑Weighted MAE, bias, diagnostics). - Support multi-model training, evaluation, selection, calibration, and DFU lifecycle management. - Integrate and maintain MLflow for experiment

Principal Software Engineer, Apple Data Platform Cupertino, United States of America The Apple Data Platform team powers analytics and machine learning across Apple. Our mission is to deliver cutting-edge, reliable, and user-friendly infrastructure for data ingestion, storage, processing, and interaction, enabling teams to build data-intensive applications with ease and success. Our team provides a fully managed Lakehouse solution powered by Apache Iceberg and the Iceberg REST Catalog, interactive querying using Trino and Druid, and data processing using Apache Spark & Flink. Our platform supports both analytical and machine learning workloads across Apple, serving key products and organizations including Siri, Search, Media, iCloud, News, Apple Intelligence, and Maps. We invest in open source technologies, and continuously innovate to meet evolving customer needs and market trends. We are seeking Principal-level engineers passionate about building world-class infrastructure products. In this role, you will serve as a technical leader on the Data Platform team, guiding team members in both technical and professional growth. You will collaborate closely with cross-functional teams to lead the planning, execution, and success of technical projects, with a focus on enhancing the experience for Apple's internal customers. The ideal candidate will have outstanding communication skills, proven data infrastructure/platform design and implementation capabilities, strong business acumen, and an innate drive to deliver results. They will be a self-starter, comfortable with ambiguity and will enjoy working in a fast-paced dynamic environment. Build and operate Apple's largest data platform supporting exabyte scale. Scale and operationalize our largest multi-tenant, highly secure, and compliant Data

Corporate Account Executive - UK&I London Join Pigment: The AI Platform Redefining Business Planning Pigment is the AI-powered business planning and performance management platform built for agility and scale. We connect people, data, and processes in one intuitive, feature-rich solution, empowering every team-from Finance to HR-to build, adapt, and align strategic plans in real time. Founded in 2019, Pigment is one of the fastest-growing SaaS companies globally. Industry leaders like Unilever, Snowflake, Siemens, and DPD use Pigment daily to make more informed decisions and confidently navigate any scenario. With a team of 600+ across Paris, London, New York, Toronto, San Francisco and Austin, we've raised nearly $400M from top-tier investors and were named a Visionary in the 2024 Gartner® Magic Quadrant™ for Financial Planning Software. At Pigment, we take smart risks, celebrate bold ideas, and challenge the status quo-all while working as one team. If you're driven by innovation and ready to make an impact at scale, we'd love to hear from you. Join Pigment: The AI Platform Redefining Business Planning Pigment is the AI-powered business planning and performance management platform built for agility and scale. We connect people, data, and processes in one intuitive, feature-rich solution, empowering every team-from Finance to HR-to build, adapt, and align strategic plans in real time. Founded in 2019, Pigment is one of the fastest-growing SaaS companies globally. Industry leaders like Unilever, Snowflake, Siemens, and DPD use Pigment daily to make more informed decisions and confidently navigate any scenario. With a team of 600+

Account Executive - Business & Corporate Development Remote - Houston, Texas posted: Posted 2 Days Ago

AI Architect IN-KA-Bangalore, India (SKAV Seethalakshmi) GESC Position: AI Architect Job Description: A highly skilled and motivated AI Architect for designing, developing, and implementing artificial intelligence (AI) solutions within an organization. The role focuses on leveraging AI & GenAI technologies, machine learning (ML), natural language processing (NLP), computer vision, and predictive analytics to solve complex business problems and drive innovation. The AI Architect bridges the gap between business objectives and technology, ensuring that AI systems align with the organization's strategic goals. What we're looking for: Education : Bachelor's or Master's degree in Computer Science, Artificial Intelligence, Data Science, or a related field. Advanced degree preferred. Experience: Minimum of 5 years of experience in AI projects, data science, or a related field. Skills: Expertise in AI & GenAI technologies, frameworks, and methodologies (e.g., supervised/unsupervised learning, deep learning, NLP, computer vision). Proficiency in programming languages like Python, .Net Strong understanding of cloud technologies (Azure & GCP), APIs, and microservices. Proven experience in managing technical projects, preferably in AI, machine learning, or data-driven technologies. Knowledge of big data processing tools (e.g., Hadoop, Apache Spark). Hands on experience with tools and platforms used in AI development (e.g., Azure AI Foundry, Ollama, DataBricks etc.). Experience in architecting AI Solutions using on premise and Cloud technologies. Experience with agile methodologies and tools (e.g., Jira, Trello) for managing technical projects. Demonstrated ability to manage multiple projects simultaneously and prioritize effectively. Strong problem-solving, analytical, and communication skills. Knowledge of AI ethics and regulatory compliance frameworks. Proficiency with data

Applied AI Engineer - iCloud Data Cupertino, United States of America Would you like to drive the future of Apple's data platform and shape how AI fundamentally transforms the way we build, operate, and scale data at Apple, while having the unique opportunity to impact some of the most far-reaching software applications in the world? The iCloud Data organization within Apple Services enables iCloud users to access all their content across apps (Photos, Mail, Messages, FaceTime, Calendar, Enterprise & Education etc) on every device, all the time, through consistent, scalable, timely, accurate, complete and fully integrated data infrastructure that surfaces relevant information. We are investing deeply in a new generation of AI-native capabilities, agents, intelligent workflows, and self-serve analytics, to accelerate our Data Engineering and Data Science teams and define what an AI-first data organization looks like at Apple scale. If this excites you and you're energized by taking novel AI techniques from research to production on hard, high-leverage, high-scale problems, we'd love to hear from you! We're seeking a top-tier Applied AI Engineer with strong architectural thinking, deep AI/ML knowledge and robust software skills, who has built AI products end-to-end, has sharp intuition for LLMs, agents, retrieval and evaluation, and shares our passion for trustworthy data-driven products at Apple. Build the AI foundation of our data platform, scalable and trustworthy AI products, agents and workflows that power self-serve analytics, experimentation, and data engineering across iCloud, in partnership with Engineering, Data Science, Product, Platform and Research, improving how we build,

AI/ML Data Scientist 2 Locations posted: Posted 8 Days Ago

Lead Power Engineer Sanchong posted: Posted 22 Days Ago

[Growth Engineering] Senior Staff Back-end Engineer I Shanghai, China In Growth Engineering, our goal is to increase the number of transacting Customers while also growing their purchase frequency and lifetime value. We build core platforms and services that power Coupang's growth and marketing efficiency. Various platforms under our portfolio include Search Advertising Platform, Affiliate Platform, Display targeting & re-targeting Platform, CRM, Audience Platform, Push Marketing Platform, and other core marketing services for user profile, landing, tracking & attribution. ML opportunities across these portfolios include targeting, personalization, product & offer recommendations, budget optimization, real-time bidding, creative optimization, fraud detection, and various others. As a Senior Staff Engineer within Coupang's Growth Engineering team, you will be instrumental in architecting and developing a next-generation CRM platform. You will lead initiatives to ensure the platform delivers an end-to-end campaign management including Campaign Plan, Target Customer Selection, Promotion optimization, Product Selection, Creative Management, Landing Page Creation and Campaign Performance Measurement. Design a real-time decision engine to empower downstream services across the Growth Engineering ecosystem. Your expertise will directly contribute to advancing Coupang's data-driven capabilities, aligning technical innovation with strategic business objectives. Key Responsibilities: - Design, develop, and automate large-scale, high-performance streaming data processing systems to drive business growth and enhance product experience - Build scalable infrastructure and pipelines using best-in-class software engineering practices - Ensure the platform supports high availability, reliability, and low-latency. - Take full ownership end to end: from solution investigation/benchmark, development to operational health - Establish and execute short and long-term architectural

Engenheiro de Dados Sênior Remote | São Carlos, SP, Brazil Company Description: A Serasa Experian é a primeira e a maior Datatech do Brasil. Líder em soluções de inteligência para análise de riscos e oportunidades, com foco nas jornadas de crédito, autenticação e prevenção à fraude. Com tecnologia de ponta, inovação e os melhores talentos, transforma a incerteza do risco na melhor decisão, ajudando pessoas a realizarem seus sonhos e empresas de todos os portes e segmentos a prosperarem. Temos 22.000 pessoas operando em 32 países e a cada dia estamos investindo em novas tecnologias, profissionais talentosos e inovação para ajudar todos os clientes a maximizarem cada oportunidade. Com sede corporativa em Dublin, Irlanda, a Experian está listada na Bolsa de Valores de Londres (EXPN) e compõe o índice FTSE 100. Job Description: We are looking for a Data | Sofware Engineer to lead the evolution of a petabyte-scale data platform on AWS. In this role, you will design and build highly scalable, resilient, and high‑performance data pipelines and services, applying advanced software and data engineering practices. You will collaborate closely with product, engineering, and operations teams to enable next‑generation data‑driven marketing solutions, accelerate innovation, and ensure long‑term platform scalability and reliability. Your key responsibilities Design, implement, and evolve a petabyte‑scale AWS data platform , applying advanced engineering, distributed systems principles, and cloud‑native practices to ensure performance, reliability, and scalability. Build and optimize highly scalable data pipelines using Scala, Spark, and cloud‑native services, ensuring efficient, resilient, and production‑ready data processing

Technical Escalation Engineer Toronto Join Pigment: The AI Platform Redefining Business Planning Pigment is the AI-powered business planning and performance management platform built for agility and scale. We connect people, data, and processes in one intuitive, feature-rich solution, empowering every team-from Finance to HR-to build, adapt, and align strategic plans in real time. Founded in 2019, Pigment is one of the fastest-growing SaaS companies globally. Industry leaders like Unilever, Snowflake, Siemens, and DPD use Pigment daily to make more informed decisions and confidently navigate any scenario. With a team of 600+ across Paris, London, New York, Toronto, San Francisco and Austin, we've raised nearly $400M from top-tier investors and were named a Visionary in the 2024 Gartner® Magic Quadrant™ for Financial Planning Software. At Pigment, we take smart risks, celebrate bold ideas, and challenge the status quo-all while working as one team. If you're driven by innovation and ready to make an impact at scale, we'd love to hear from you. The Opportunity We are looking for a Technical Escalation Engineer to work with our Engineering team. You will act as a liaison between the Customer Support Team and the Engineering teams to be the safekeeper of the Pigment platform. Together with your counterparts, you will secure the necessary means so we proactively watch and prevent issues from occurring and ensure that any event happening in production is addressed the way it should. Nothing shall pass! It's a role where you are encouraged to take many initiatives. You will have

Software Development Engineer II, Amazon Security Data Engineering Herndon, Virginia, USA If you thrive in a fast-paced, innovative environment and are interested in pushing the boundaries of what's possible with big data, cloud computing, and security analytics, Amazon Security Platform Engineering is the place for you. We support a suite of services and products to ingest, optimize and query petabyte scale datasets efficiently, and enable critical security use cases such as incident response, security analytics, threat-hunting, and low-latency detection. Successful candidates for this position will possess: Strong software engineering fundamentals, including expertise in object-oriented design, algorithms, and data structures Experience building and operating scalable, fault-tolerant, distributed systems Proficiency in Java, Scala or Python with a solid understanding of multi-threading and memory management Knowledge of big data compute engines such as open table formats like Apache Iceberg, and Apache Spark, Trino, Flink. Excellent verbal and written communication skills Self-motivation and the ability to deliver high-quality results in a fast-paced environment. Key job responsibilities Build and support Amazon Security's Data Lake, consisting of services for data cataloging, optimization, compaction, re-organization, and indexing at petabyte scale. Drive technical vision and contribute to the roadmap for systems that manage petabyte-scale data catalogs supporting Apache Iceberg, AWS Glue, Parquet, and other formats. Design and implement distributed systems for data optimization that balance multiple latency scenarios from minute-level batch detections to month-spanning threat hunting queries. Modernize security intelligence platform's data catalog and governance services to provide customers with rule-based access controls and fine-grained data access. Build

Lead Distributed Data Platform Engineer 9 Locations posted: Posted 30+ Days Ago

Databricks Engineering Consultant Location not specified

Member of Technical Staff (Software Engineer, Data Platform) San Francisco, California, United States, New York City, Palo Alto About the Role The Data Platform team owns the end-to-end data lifecycle at Perplexity, from ingestion through processing, storage, and serving, powering product features, analytics, experimentation, AI workloads, and the company's data lake. The team defines the architecture for batch and streaming systems, the orchestration and observability stack, and a self-serve data platform, while thoughtfully combining platforms such as Databricks and Snowflake with open-source technologies including Spark, Kafka, Flink, Airflow, Dagster, dbt, Iceberg, Delta Lake, and ClickHouse. In this senior/staff role, you will shape architecture, set standards, and drive the long-term technical direction of Perplexity's data ecosystem. Key Responsibilities - Design and operate large-scale batch and streaming data pipelines that directly power Perplexity product features, AI training and evaluation workflows, analytics, and experimentation. - Build event-driven and streaming systems (Kafka, Kinesis, PubSub, or similar) for real-time ingestion, transformation, and delivery, alongside batch frameworks for backfills, aggregations, and offline computation. - Lead the architecture of data orchestration using tools like Airflow or Dagster, owning scheduling, dependency management, retries, SLAs, and end-to-end observability for critical data flows. - Set and enforce guarantees for data correctness, freshness, lineage, and recoverability, designing systems that handle rapid scale growth, partial failures, and evolving schemas without disrupting AI workloads or product experiences. - Build self-serve data platforms that let engineers, data scientists, and analysts safely discover data, define contracts, and create and operate their own pipelines with minimal

Senior Staff Magnetics Engineer Senior Staff Magnetics Engineer

Senior Software Engineer, ML/AI Platform United States Attentive® is the AI marketing platform for 1:1 personalization redefining the way brands and people connect. We're the only marketing platform that combines powerful technology with human expertise to build authentic customer relationships. By unifying SMS, RCS, email, and push notifications, our AI-powered personalization engine delivers bespoke experiences that drive performance, revenue, and loyalty through real-time behavioral insights. Recognized as the #1 provider in SMS Marketing by G2, Attentive partners with more than 8,000 customers across 70+ industries. Leading global brands like Crate and Barrel, Urban Outfitters, and Carter's work with us to enable billions of interactions that power tens of billions in revenue for our customers. With a distributed global workforce and employee hubs in New York City, San Francisco, London, and Sydney, Attentive's team has been consistently recognized for its performance and culture. We're proud to be included in Deloitte's Fast 500 (four years running!), LinkedIn's Top Startups , Forbes' Cloud 100 (five years running!), Inc.'s Best Workplaces , and the Human Rights Campaign Foundation's Corporate Equality Index ! About the Role We're looking for a self-motivated, highly driven Senior Software Engineer to join our Machine Learning Platform (MLPlatform) team. As a team, we enable Attentive's Machine Learning (ML) practice to directly impact Attentive's AI product suite through the tools to train, serve, and deploy ML models with higher velocity and performance, while maintaining reliability. We build and maintain a foundational ML platform that spans the full ML lifecycle for use

Data Engineer - Data Warehouse Architect Reston, VA This position focuses on developing, implementing, and maintaining architecture solutions across a large enterprise data warehouse to support effective and efficient data management and enterprise-wide business intelligence analytics. Responsibilities: Implement and optimize data pipeline architectures for sourcing, ingestion, transformation, and extraction processes, ensuring data integrity and compliance with organizational standards. Develop and maintain scalable database schemas, data models, and data warehouse structures; perform data mapping, schema evolution, and integration between source systems, staging areas, and data marts. Automate data extraction workflows and create comprehensive technical documentation for ETL/ELT procedures; collaborate with cross-functional teams to translate business requirements into technical specifications. Establish and enforce data governance standards, including data quality metrics, validation rules, and best practices for data warehouse design and architecture. Develop, test, and deploy ETL/ELT scripts using SQL, Python, Spark, or other relevant languages; optimize code for performance and scalability. Tune data warehouse systems for query performance and batch processing efficiency; apply indexing, partitioning, and caching strategies. Perform advanced data analysis, validation, and profiling using SQL and scripting languages; develop data models, dashboards, and reports in collaboration with stakeholders. Conduct testing and validation of ETL workflows to ensure data loads meet SLAs and quality standards; document testing protocols and remediation steps. Troubleshoot production issues, perform root cause analysis, and implement corrective actions; validate data accuracy and consistency across systems. Basic Qualifications: Minimum of 3 years of experience in data analysis. Additional Qualifications: Strong analytical and problem-solving skills with attention to detail.

Sales Associate San Jose, California posted: Posted 7 Days Ago