Spark jobs

Senior Data Engineer George Town, Penang, Malaysia Company Description: Job Description: We are looking for a Senior Data Engineer to lead the design and delivery of scalable, high-performance data platforms that power analytics and business intelligence across the organization. This role combines hands-on engineering, architecture leadership, and team mentorship, requiring someone who can translate complex business needs into robust data solutions while driving best practices in modern data engineering. Responsibilities: Design and development of robust, scalable data pipelines and ETL/ELT solutions Architect and optimize data processing frameworks using Apache Spark and Delta Lake Develop and manage data solutions on the Databricks platform Collaborate with business, analytics teams to translate requirements into technical designs Ensure data quality, integrity, performance, and security across data platforms Implement and enforce data engineering best practices, standards, and governance Integrate data from multiple internal and external sources using modern integration tools Drive CI/CD implementation and DevOps practices for data pipelines Monitor, troubleshoot, and optimize data workflows and system performance Mentor junior engineers and provide technical guidance Participate in design reviews and contribute to architectural decisions Qualifications: Strong proficiency in Apache Spark, Delta Lake, and distributed data processing Advanced SQL and Python programming skills Extensive hands on experience with the Databricks platform Solid experience in data pipeline design, ETL/ELT frameworks, and data orchestration Experience with traditional databases such as Oracle and SQL Server Proficiency with source code management and project tracking tools such as GitHub and Jira Additional Information: Renesas is an embedded semiconductor solution provider driven

Member of Technical Staff - ML Research Engineer, Data San Francisco, United States, Boston, Remote About Liquid AI Spun out of MIT CSAIL, we build general-purpose AI systems that run efficiently across deployment targets, from data center accelerators to on-device hardware, ensuring low latency, minimal memory usage, privacy, and reliability. We partner with enterprises across consumer electronics, automotive, life sciences, and financial services. We are scaling rapidly and need exceptional people to help us get there. The Opportunity Our Data team powers Liquid Foundation Models across pre-training, vision, audio, and emerging modalities. Public data sources are plateauing. Model performance increasingly depends on purpose-built datasets. We need ML-minded engineers who can collect, filter, and synthesize high-quality data at scale. We treat data as a research problem, not an infrastructure problem. Our engineers run experiments, design ablations, and measure how data decisions move model quality. We will match you to the team where you can grow the fastest and have the most impact: pre-training, post-training RL, vision-language, audio, or multimodal. While San Francisco and Boston are preferred, we are open to other locations. What We're Looking For We need someone who: - Thinks like a researcher, ships like an engineer: We need people who form hypotheses, run experiments, and measure results. Our engineers understand deep-theoretical research, and our researchers ship production systems. - Learns fast and adapts: We work across modalities that evolve weekly. We need people who pick up new domains quickly and thrive with ambiguity. - Obsesses over data quality:

Sr Applied Scientist III, Supply Chain Optimization Technologies - SCAIL New York, New York, USA Our team is at the forefront of machine learning research. Led by scientists including Dean Foster, the team is dedicated to developing innovative RL algorithms and applying them to complex, real-world challenges in Amazon's global inventory and supply chain network. Our focus is not just on advancing theoretical knowledge but also on implementing these insights to optimize operations and enhance customer satisfaction. We foster a collaborative environment where exploration of new ideas and tackling complex problems is encouraged. The supply chain spans a wide range of operations,managing decisions that impact billions of dollars worth of inventory. For scientists passionate about impactful research in machine learning and AI, our team offers a dynamic and fulfilling environment to make a tangible difference in the field and Amazon's operations. Key job responsibilities - Design, implement, and evaluate innovative models, agents, and software prototypes. - Collaborate with a team of experienced scientists to drive technological advancements. - Develop innovative solutions to complex business problems in collaboration with partner teams. - Contribute to Amazon's global science community through collaboration and publication of ground-breaking research. - Engage in research projects that contribute to the wider scientific community, sharing findings through publications in top-tier journals and conferences. A day in the life A day in the life As a part of our team, you will be working alongside thought leaders like Dean Foster, contributing to academic research and complex, real-world applications. Your

Data Engineer ( Bigdata/Databricks) CHENNAI, Tamil Nādu, India

AI Engineer, Advanced Solutions Lab, Google Cloud Chicago, IL, USA; +2 more The AI Engineer role within the Advanced Solutions Lab (ASL) focuses on delivering and evolving a sophisticated Machine Learning and Generative AI curriculum for global participants. Operating in an immersive environment, you will work directly with customers to apply innovative AI solutions to high-impact business challenges and specific industry use cases. You will lead the daily educational journey for participants while continuously improving the program by integrating internal Google expertise and recommending the best open-source frameworks and models. Beyond these core duties, you will actively participate in the broader Google ML community through research collaboration, engineering projects, and strategic initiatives that shape the future of AI.Google Cloud accelerates every organization's ability to digitally transform its business and industry. We deliver enterprise-grade solutions that leverage Google's cutting-edge technology, and tools that help developers build more sustainably. Customers in more than 200 countries and territories turn to Google Cloud as their trusted partner to enable growth and solve their most critical business problems.Individual pay is determined by factors including job-related skills, experience, and relevant education or training. US: $152000 - $222000 (USD) + 15% bonus target + bonus + equity + benefits Learn more about benefits at Google . Drive the Advanced Solutions Lab by delivering content, identifying machine learning (ML) experts across Google to support specific sessions, and providing ongoing curriculum enhancements. Lead and support customers' machine learning projects from framing to implementation in the Advanced Solutions Lab. Design

Data/Infrastructure Advocate Engineer - EMEA Remote

Cleared Data Engineer Washington, DC At Accenture Federal Services, nothing matters more than helping the US federal government make the nation stronger and safer and life better for people. Our 13,000+ people are united in a shared purpose to pursue the limitless potential of technology and ingenuity for clients across defense, national security, public safety, civilian, and military health organizations. Join Accenture Federal Services, a technology company within global Accenture. Recognized as a Glassdoor Top 100 Best Place to Work, we offer a collaborative and caring community where you feel like you belong and are empowered to grow, learn and thrive through hands-on experience, certifications, industry training and more. Join us to drive positive, lasting change that moves missions and the government forward! The work: - Pipeline Architect : Design, build, and maintain scalable end-to-end data pipelines using Databricks, Spark, and related technologies. - Transformation Titan : Develop efficient data processing and transformation workflows to support analytics and reporting needs. - Integration Hero : Integrate diverse data sources including APIs, databases, and cloud storage into unified datasets. - Collaboration Champion : Work closely with cross-functional teams (data science, analytics, business units) to design and implement data solutions that align with business goals. - Quality Guardian : Implement robust validation, monitoring, and observability processes to ensure data accuracy, completeness, and reliability. - Automation Avenger : Contribute to data governance, security, and automation initiatives within the data ecosystem. - Cloud Commander : Leverage AWS services (e.g., S3, Glue, Lambda, Redshift) to build

Data Scientist Bangalore Area posted: Posted Yesterday

Software Development Engineer, Measurement, Ad Tech, and Data Science (MADS) Santa Monica, California, USA Are you passionate about Big Data and distributed systems? Do you thrive on building state-of-the-art products at petabyte scale? Join a team of world-class engineers and industry-leading experts operating one of the largest big data platforms for advertising at Amazon scale. We are seeking an experienced Software Development Engineer to build the next generation of advertising measurement and reporting systems. You will leverage open-source technologies such as Spark and Java, alongside AWS services including EMR, Kinesis, DynamoDB, and Bedrock to deliver solutions that serve a wide set of users across Amazon's full advertising suite - spanning display, search, native, and video on all devices. The ideal candidate brings a proven track record of delivering highly distributed systems at scale, with deep expertise in design patterns including SDKs, libraries, and versioning strategies. You are operationally excellent, experienced in collaborating across multiple teams, and ready to build the infrastructure that powers advertising insights at Amazon scale. Key job responsibilities - Demonstrate strong engineering fundamentals and apply real-world experience to solve complex technical challenges - Showcase diverse architectural approaches and design patterns in technical designs - Think big and stay informed about industry standards and emerging trends in the Ad Tech space - Communicate effectively across all leadership levels, both verbally and in writing - Lead deep technical discussions with senior engineers, collaborating to solve challenging problems - Quickly grasp new systems and provide valuable technical consultation to stakeholders

Lighting Design Engineer Davidson, North Carolina posted: Posted Yesterday

Staff Data Engineer Bangalore Netradyne harnesses the power of Computer Vision and Edge Computing to revolutionize the modern-day transportation ecosystem. We are a leader in fleet safety solutions. With growth exceeding 4x year over year, our solution is quickly being recognized as a significant disruptive technology. Our team is growing, and we need forward-thinking, uncompromising, competitive team members to continue to facilitate our growth. About Netradyne: Netradyne provides AI-powered technologies for fleet management and safer roads. An award-winning industry leader in fleet safety and video telematics solutions, Netradyne empowers thousands of commercial fleet customers across North America, Europe, and Asia to enhance their driver performance, reduce risk, and optimize operations. Netradyne sets the standard among transportation technology companies for enhancing and sustaining road safety, with an industry-leading 25+ billion miles vision-analyzed for risk and an industry-first driver scoring system that reinforces safe behaviors. Founded in 2015, Netradyne is headquartered in San Diego with offices in San Francisco, Nashville, the UK and Bangalore. For more details visit: www.netradyne.com Job Overview: As a Staff Data Engineer - ML at Netradyne, an award-winning industry leader in fleet safety and video telematics solutions, you will take senior technical ownership of the machine learning and data platforms that underpin our AI-powered fleet analytics. This role is responsible for designing, building, and scaling production-grade ML pipelines, generative AI capabilities, and real-time data streaming systems across cloud and edge environments, delivering actionable insights that make fleet operations safer and more efficient. You will collaborate closely with machine

Staff Data Scientist - Verification & Validation Foster City, CA Zoox is on an ambitious journey to develop a full-stack autonomous vehicle system for cities. We are seeking a Staff Data Scientist to join a verification and validation team that evaluates safety-critical AI systems. You will join a team of software and data engineers that leverage methods including log data analysis, simulation, and closed-course structured testing. You'll work cross-functionally with AI software, System Design and Mission Assurance, Simulation, Sensors, and other teams to develop, execute, and iterate on validation methods and pipelines. These pipelines evaluate safety-critical systems, are highly visible, and are an important critical path element of launching our service. The ideal candidate brings a hybrid of statistical rigor and engineering mindset to drive clarity from ambiguity, establish new processes, and propel the team forward. This is a deeply technical and hands-on role where you will be expected to be a self-sufficient builder and coder, not just a manager of projects. Zoox is on an ambitious journey to develop a full-stack autonomous vehicle system for cities. We are seeking a Staff Data Scientist to join a verification and validation team that evaluates safety-critical AI systems. You will join a team of software and data engineers that leverage methods including log data analysis, simulation, and closed-course structured testing. You'll work cross-functionally with AI software, System Design and Mission Assurance, Simulation, Sensors, and other teams to develop, execute, and iterate on validation methods and pipelines. These pipelines evaluate safety-critical systems, are

Enterprise Account Executive - West US Join Pigment: The AI Platform Redefining Business Planning Pigment is the AI-powered business planning and performance management platform built for agility and scale. We connect people, data, and processes in one intuitive, feature-rich solution, empowering every team-from Finance to HR-to build, adapt, and align strategic plans in real time. Founded in 2019, Pigment is one of the fastest-growing SaaS companies globally. Industry leaders like Unilever, Snowflake, Siemens, and DPD use Pigment daily to make more informed decisions and confidently navigate any scenario. With a team of 600+ across Paris, London, New York, Toronto, San Francisco and Austin, we've raised nearly $400M from top-tier investors and were named a Visionary in the 2024 Gartner® Magic Quadrant™ for Financial Planning Software. At Pigment, we take smart risks, celebrate bold ideas, and challenge the status quo-all while working as one team. If you're driven by innovation and ready to make an impact at scale, we'd love to hear from you. Join Pigment: The AI Platform Redefining Business Planning Pigment is the AI-powered business planning and performance management platform built for agility and scale. We connect people, data, and processes in one intuitive, feature-rich solution, empowering every team-from Finance to HR-to build, adapt, and align strategic plans in real time. Founded in 2019, Pigment is one of the fastest-growing SaaS companies globally. Industry leaders like Unilever, Snowflake, Siemens, and DPD use Pigment daily to make more informed decisions and confidently navigate any scenario. With a team of 600+

Sr. Materials Engineer (Materials Laboratory) Hawthorne, CA SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not. Today SpaceX is actively developing the technologies to make this possible, with the ultimate goal of enabling human life on Mars. SR. MATERIALS ENGINEER (MATERIALS LABORATORY) The Sr. Materials Engineer is a self-starter responsible for ensuring the quality, integrity, and usability of data generated across the Materials Lab while leading the development and qualification of new and improved test processes. You will establish engineering controls and quality systems, train lab personnel on first principles, investigate data anomalies, and drive capability expansion across Thermal, Characterization, Chemistry, Reliability, Mechanical Testing, and R&D segments. This role requires strong project management skills to simultaneously advance multiple initiatives while building a state-of-the-art laboratory that delivers high-quality data quickly to support SpaceX's most critical hardware programs. RESPONSIBILITIES: - Own data quality and integrity across the Materials Lab, including Thermal Analysis (TGA, LFA, DSC, DMA, STA, etc.), Characterization (SEM, EDS, XRD, optical microscopy, FTIR, etc.), Chemistry (ICP, spark OES, PSD, pycnometer testing, etc.), Reliability (corrosion, coatings, etc.), Mechanical Testing (cryo, room temperature, and elevated temperature tensile/compression/fatigue, Charpy, fracture toughness, etc.), and R&D (torch testing, heat treatments, Gleeble, and process development, etc.) - Develop and implement engineering controls, calibration procedures, and quality systems to ensure consistent, accurate, and traceable data output from all lab areas - Serve as a technical trainer and mentor, teaching lab

Electrification Consultant Ismaning (Munich), Bavaria posted: Posted Today

Site Reliability Engineer, AiDP Production Engineering Austin Metro Area, United States of America The Production Engineering team within the AI and Data Platform (AiDP) organization manages a wide array of real-time, near real-time, and batch analytical solutions. These platforms are integral to core business functions across Apple. These include sales, operations, finance, AppleCare, marketing, and services, and are instrumental in driving critical, data-driven decisions. To build these solutions, we leverage a combination of proprietary and leading open-source technologies such as Kafka, Spark, Iceberg, and Airflow. A key part of our mission is to enable AI-centric automations that enhance the overall efficiency and intelligence of the platform. We are looking for passionate engineers who thrive on solving complex infrastructure challenges at scale, both on-premises and in the cloud. If you are dedicated to optimizing scalable, maintainable, and user-friendly systems, you will find compelling opportunities to make a significant impact at AiDP. The Service Reliability Engineer (SRE) role within AiDP Production Engineering is a dynamic position that blends strategic architectural design with hands-on technical execution. As an SRE, you will be responsible for configuring, tuning, and ensuring the resilience of complex, multi-tiered systems to achieve optimal application performance, stability, and availability. Our team manages critical data pipelines and applications across both bare-metal and cloud computing platforms, delivering essential data processing for all of Apple's key business functions. We operate at an immense scale, handling exabytes of data, petabytes of memory, and tens of thousands of jobs to enable predictable and performance data

Senior Software Engineer (Trust Foundations) New York, New York, United States; San Francisco, California, United States; Seattle, Washington, United States 1 in 4 people in the US have a treatable mental health condition, but most providers don't accept insurance, making therapy too expensive for most people. Headway's mission is to fix this by building a new mental healthcare system everyone can access. We started by solving the biggest barrier to care: insurance. The admin work - credentialing, claims, payment reconciliation - is a nightmare. We've automated that. But we're going further. Over 75,000 providers across all 50 states run their practice on our software, serving over 1 million patients. We are building the best tools for therapists to run their entire practice, reimagining the experience of finding a therapist, and investing in the platform foundations to enable this at scale. We aren't just a billing layer; we are becoming the platform where care actually happens. We're a Series D company with $325M+ in funding (a16z, Accel, Spark Capital, etc.), looking for exceptional people to help us achieve this mission. We want your time here to be the most meaningful experience of your career. Join us, and help change mental healthcare for the better. About this role: Building for trust is non‑negotiable in our mission of making mental healthcare more accessible and affordable across the US: patients share sensitive information, payers demand demonstrably strong controls, and providers depend on reliable, secure infrastructure. Trust Foundations is responsible for instilling confidence in our

Kitting Team Lead- Night Shift Newark, Delaware posted: Posted Today

Electrical Engineer III/Senior/Staff (Project Execution-Power Innovation) 2 Locations posted: Posted 30+ Days Ago

Materials Engineer (Materials Laboratory) Hawthorne, CA SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not. Today SpaceX is actively developing the technologies to make this possible, with the ultimate goal of enabling human life on Mars. MATERIALS ENGINEER (MATERIALS LABORATORY) The Materials Engineer is a self-starter responsible for ensuring the quality, integrity, and usability of data generated across the Materials Lab while leading the development and qualification of new and improved test processes. You will establish engineering controls and quality systems, train lab personnel on first principles, investigate data anomalies, and drive capability expansion across Thermal, Characterization, Chemistry, Reliability, Mechanical Testing, and R&D segments. This role requires strong project management skills to simultaneously advance multiple initiatives while building a state-of-the-art laboratory that delivers high-quality data quickly to support SpaceX's most critical hardware programs. RESPONSIBILITIES: - Own data quality and integrity across the Materials Lab, including Thermal Analysis (TGA, LFA, DSC, DMA, STA, etc.), Characterization (SEM, EDS, XRD, optical microscopy, FTIR, etc.), Chemistry (ICP, spark OES, PSD, pycnometer testing, etc.), Reliability (corrosion, coatings, etc.), Mechanical Testing (cryo, room temperature, and elevated temperature tensile/compression/fatigue, Charpy, fracture toughness, etc.), and R&D (torch testing, heat treatments, Gleeble, and process development, etc.) - Develop and implement engineering controls, calibration procedures, and quality systems to ensure consistent, accurate, and traceable data output from all lab areas - Serve as a technical trainer and mentor, teaching lab technicians and engineers

Senior Security Engineer (Product) New York, New York, United States; San Francisco, California, United States; Seattle, Washington, United States 1 in 4 people in the US have a treatable mental health condition, but most providers don't accept insurance, making therapy too expensive for most people. Headway's mission is to fix this by building a new mental healthcare system everyone can access. We started by solving the biggest barrier to care: insurance. The admin work - credentialing, claims, payment reconciliation - is a nightmare. We've automated that. But we're going further. Over 75,000 providers across all 50 states run their practice on our software, serving over 1 million patients. We are building the best tools for therapists to run their entire practice, reimagining the experience of finding a therapist, and investing in the platform foundations to enable this at scale. We aren't just a billing layer; we are becoming the platform where care actually happens. We're a Series D company with $325M+ in funding (a16z, Accel, Spark Capital, etc.), looking for exceptional people to help us achieve this mission. We want your time here to be the most meaningful experience of your career. Join us, and help change mental healthcare for the better. Headway's mission is a big one - to build a new mental health care system everyone can access. We've built technology that helps people find great therapists with the first software-enabled national network of providers accepting insurance. 1 in 4 people in the US have a treatable mental

Sr Staff Engineer, Equipment Controls Fremont, California posted: Posted 30+ Days Ago

Software Development Engineer, Measurement, Ad Tech, and Data Science (MADS), Measurement, Ad Tech, and Data Science (MADS) Toronto, Ontario, CAN Are you a data-driven problem solver passionate about building scalable systems that drive customer trust and business growth? Join our innovative MADS Ads Data systems in Toronto, where we're expanding to meet increasing customer demand. We provide a secure, privacy-safe, and cloud-based environment for advertisers to perform advanced analytics across multiple data sets and generate actionable insights. Our services ingest and process billions of behavioral signals daily, making speed, scale, and accuracy critical to our success. You'll work in exciting computer science domains, managing massive-scale streaming systems, applying machine learning and data science expertise using Big Data (e.g. Spark) technology to drive optimizations and insights. We're seeking experienced, motivated software engineers with a proven track record in data engineering and scalable web services. If you're passionate about developing analytical solutions to solve complex business problems and driving results that impact Amazon's business, this is the right opportunity for you. The ideal candidate is innovative, with excellent problem-solving, analytical, and technical skills. You're comfortable working on Linux systems and have a deep understanding of agile software development. You excel at building exceptional customer relationships, uncovering the true requirements behind feature requests, and simplifying complex problems to tackle challenges. Join our team and be part of a dynamic and collaborative environment where you can make a significant impact on Amazon's advertising solutions. Basic Qualifications: - 3+ years of non-internship professional software development

Project Engineer Project Engineer