Spark jobs

Data Architect Warminster, Wiltshire posted: Posted 2 Days Ago

Sr. Solutions Architect - Public Sector (Hunter) Maryland; Virginia; Washington, D.C. FEQ127R326 Mission Solutions Architects at Databricks lead the growth of the Databricks Unified Analytics Platform. As a team, we have expertise in cloud platforms, data engineering, data analytics, and data science and machine learning. As a member of our team, you will exercise and develop expertise in those areas, using open-source projects such as Apache Spark, MLflow, and Delta Lake. This is a customer-facing role, where you will work with customers, your teammates, the product team, and our post-sales teams, to identify use cases for Databricks, develop architectures and solutions using our platform, and guide customers through the implementation, to accomplish value. In that process, you will build relationships with our customers, find and build champions, and become a trusted advisor. You will report to the Field Engineering Manager for the team. The impact you will have: - Partner with the sales team to help customers understand how Databricks can help solve their business problems - Provide technical leadership for customers to evaluate and adopt Databricks - Consult on big data architecture, implement proof of concepts for strategic customer projects, data science and machine learning projects, and validate integrations with cloud services and other 3rd party applications - Build and present references architectures, how-tos, and demo applications for customers - Become an expert in, and promote Databricks inspired open-source projects (Spark, Delta Lake, MLflow, and Koalas) across developer communities through meetups, conferences, and webinars - Travel to customers in

Engineering Manager - Lakebase Storage London, United Kingdom P-1534 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to improve their business. As an Engineering Manager, you will work with your team to build mission-critical Lakebase services on the Databricks Platform at scale. Lakebase enables teams to deliver applications faster with an optimized database developer experience, integration with the lakehouse, and simplified operations. It provides a managed Postgres transactional layer for application state and workloads that combine operational data, analytics, and AI. You'll join as one of the founding engineering leaders to build out both the Lakebase product and our London presence. Key responsibilities include: - Drive continuous delivery within a team of experts in storage technology, distributed systems and Rust. - Manage the development and rollout of storage services that host millions of customer databases across dozens of regions - Partner with peer engineering teams across Databricks to co-evolve Lakebase services with our global infrastructure. - Lead operational excellence in 24/7 operation of our system The impact you will have: - Hire great engineers to build an outstanding team. - Support engineers in their career development by providing clear feedback and develop engineering leaders. - Ensure high technical standards by instituting processes (architecture reviews, testing) and culture (engineering excellence). - Work with engineering and product leadership to build a long-term

Databricks Data Engineer Location not specified

Sr. Solutions Architect - Public Sector (Federal Civilian) Maryland; Virginia; Washington, D.C. FEQ127R5 Mission Solutions Architects at Databricks lead the growth of the Databricks Unified Analytics Platform. As a team, we have expertise in cloud platforms, data engineering, data analytics, and data science and machine learning. As a member of our team, you will exercise and develop expertise in those areas, using open-source projects such as Apache Spark, MLflow, and Delta Lake. This is a customer-facing role, where you will work with customers, your teammates, the product team, and our post-sales teams, to identify use cases for Databricks, develop architectures and solutions using our platform, and guide customers through the implementation, to accomplish value. In that process, you will build relationships with our customers, find and build champions, and become a trusted advisor. You will report to the Field Engineering Manager for the team. The impact you will have: - Partner with the sales team to help customers understand how Databricks can help solve their business problems - Provide technical leadership for customers to evaluate and adopt Databricks - Consult on big data architecture, implement proof of concepts for strategic customer projects, data science and machine learning projects, and validate integrations with cloud services and other 3rd party applications - Build and present references architectures, how-tos, and demo applications for customers - Become an expert in, and promote Databricks inspired open-source projects (Spark, Delta Lake, MLflow, and Koalas) across developer communities through meetups, conferences, and webinars - Travel to customers

Production I Job Posting Country United States | Job Posting State/Province NV | Job Posting City Sparks Job Details Division: Location: Job Posting City Sparks, Job Posting State/Province NV, Job Posting Country United States Other Available Locations: Welcome to Nucor, where our unique culture sets us apart. Joining Nucor means becoming part of North America's largest steel and steel products manufacturer, offering unmatched career opportunities and stability. We are known for our innovative sustainability practices and are the continent's largest recycler. Our vibrant culture, built on personal connections and teamwork, empowers every team member with the freedom to innovate and contribute to our collective success. Experience a workplace where safety, efficiency, and meaningful personal connections define every day, enriching both our teammates and the communities we serve. Nucor supports its teammates with top-tier benefits like Medical, Dental, Vision, and Disability Insurance, 401K and Roth accounts, Profit sharing, Pay-for-performance bonuses, Paid parental leave, Tuition reimbursement, Stock program, Scholarships and more. Basic Job Functions: Properly follow company and OSHA safety procedures. Attend safety meetings and participate in safety program. General labor duties that may include but not limited to sweeping, shoveling and general clean-up of any area and/or equipment throughout the yard. Keep grounds clear of debris while maintaining safety standards. Assisting the Kiosk Operator with weighing, sorting, and moving material including general duties involving phy

Senior Data Center Critical Facilities Engineer Ashburn posted: Posted 22 Days Ago

Sr. Solutions Architect - Public Sector (SLED) Illinois; Michigan; Minnesota FEQ427R139 Mission Solutions Architects at Databricks lead the growth of the Databricks Unified Analytics Platform. As a team, we have expertise in cloud platforms, data engineering, data analytics, and data science and machine learning. As a member of our team, you will exercise and develop expertise in those areas, using open-source projects such as Apache Spark, MLflow, and Delta Lake. This is a customer-facing role, where you will work with customers, your teammates, the product team, and our post-sales teams, to identify use cases for Databricks, develop architectures and solutions using our platform, and guide customers through the implementation, to accomplish value. In that process, you will build relationships with our customers, find and build champions, and become a trusted advisor. You will report to the Field Engineering Manager for the team. The impact you will have: - Partner with the sales team to help customers understand how Databricks can help solve their business problems - Provide technical leadership for customers to evaluate and adopt Databricks - Consult on big data architecture, implement proof of concepts for strategic customer projects, data science and machine learning projects, and validate integrations with cloud services and other 3rd party applications - Build and present references architectures, how-tos, and demo applications for customers - Become an expert in, and promote Databricks inspired open-source projects (Spark, Delta Lake, MLflow, and Koalas) across developer communities through meetups, conferences, and webinars - Travel to customers in your

Cloud ML DevRel Engineer - EMEA remote

Senior Energy Transition Champion Field Office - Ireland posted: Posted 30+ Days Ago

Senior Data Engineer, AI Systems Movable Ink - Toronto Movable Ink scales content personalization for marketers through data-activated content generation and AI decisioning. The world's most innovative brands rely on Movable Ink to maximize revenue, simplify workflow and boost marketing agility. Headquartered in New York City with close to 600 employees, Movable Ink serves its global client base with operations throughout North America, Central America, Europe, Australia, and Japan. The AI Systems team owns the core recommendations engine and ML platform that powers billions of AI-driven marketing decisions daily across some of the world's largest consumer brands. As a Senior Data Engineer, you will own the Spark-based data pipelines and data infrastructure at the heart of this system - building, scaling, and optimizing the data layer that feeds our production ML models. You will work alongside ML engineers and scientists in a collaborative environment, contributing data pipelines and products to power our core recommender systems and our DaVinci Personalization product. This is an opportunity to work end-to-end on large-scale data systems that touch millions of customers, on a team working at the intersection of data engineering and machine learning. This role will be reporting to the Director of Engineering (AI/ML). Responsibilities: - Build, maintain, and optimize production data pipelines that power AI-driven personalization at scale across content selection, send-time optimization, subject line personalization, and frequency capping - Own and scale Spark-based batch pipelines, including cluster configuration, tuning, and performance optimization across GCP Dataproc - Build and maintain our ML Data

Senior Data Scientist Los Angeles, California, United States Realize your potential by joining the leading performance-driven advertising company and a proud Great Place to Work® Certified employer in the US As a Data Scientist on Taboola's R&D Team in Downtown Los Angeles, utilizing some of the latest technology and tools, you will work on the recommendation and ranking algorithms that drive Realize - Taboola's performance advertising platform serving 600M+ daily active users - and building agentic AI systems alongside it. The models you build directly influence campaign performance and revenue outcomes for advertisers worldwide. To thrive in this role, you'll need: - Master's degree or Ph.D, ideally in Computer Science, Electrical Engineering, Physics, Computational Economics, or similar - 5+ years of hands-on experience in Python and Java - Proficiency with ML frameworks (e.g., PyTorch, Tensorflow, Spark MLlib) - SQL proficiency for data exploration and analysis - System-architecture mindset - ability to design scalable, reliable, high-performance systems and reason about how a model fits into the larger data and serving infrastructure. - Experience taking end-to-end ownership of ML systems: design, build, ship, measure, and maintain - using AI-assisted “vibe coding” tools (e.g., Cursor, Claude, GitHub Copilot). - Experience building automated evaluation harnesses and test frameworks for ML systems - defining benchmarks, baselines, and guardrails that let a team ship confidently and repeatedly - Ability to communicate technical findings clearly, and to collaborate effectively across teams (including early morning or evening syncs with our Tel Aviv team) Bonus points if you have:

Staff Data Engineer - Emerald New York At H1, we believe access to the best healthcare information is a basic human right. Our mission is to provide a platform that can optimally inform every doctor interaction globally. This promotes health equity and builds needed trust in healthcare systems. To accomplish this our teams harness the power of data and AI-technology to unlock groundbreaking medical insights and convert those insights into action that result in optimal patient outcomes and accelerates an equitable and inclusive drug development lifecycle. Visit h1.co to learn more about us. Data Engineering is responsible for the development and delivery of our most important asset - our data. Looking across thousands of data sources from across the globe, the data engineering team is responsible for making sense out of that data to create the world's most extensive and comprehensive knowledge base of healthcare stakeholders and the ecosystem they influence. It is our job to ensure that only accurate, normalized data flows through to our customers, and at a velocity that keeps up with the changes in the real world. As we rapidly expand the markets we serve and the breadth and depth of data we want to collect for our customers, the team must grow and scale to meet that demand. WHAT YOU'LL DO AT H1 As a Staff Data Engineer on the Emerald team, you will play a critical role in shaping the architecture, scalability, and technical direction of H1's healthcare entity resolution platform. EMERALD is responsible

Power Generation Data Center Lead Multiple Locations

Senior Data Engineer - Data Ingestion and Enrichment team Barcelona, Catalunya, Spain We power people's progress. At Preply, we're all about creating life-changing learning experiences. We help people discover the magic of the perfect tutor, craft a personalised learning journey, and stay motivated to keep growing. Our approach is human-led, tech-enabled - and it's creating real impact. We've just reached unicorn status with a $150M Series D, accelerating our vision to transform education through human-led, AI-enhanced learning. Today, 100,000+ tutors teach 90+ languages to learners in 180 countries - and we're only getting started. As a category-defining company, we're shaping what the future of learning looks like at global scale. Every Preply lesson sparks change, fuels ambition, and drives progress that matters. Joining Preply means helping define the future of education at global scale, and building something that truly matters for millions of people, every day. Meet the team! At Preply, the Data ingestion and enrichment team provides a single, trusted, and scalable data foundation. The team ensures that all analytics, machine learning, and product features are built on unified, governed, and production-grade data assets in Preply's Lake House, including the extraction, normalization, and generation of structured data from Preply's unstructured assets, forming a durable data moat for AI-driven products. As a Senior Data Engineer in the Data Ingestion and Enrichment team , you will design and own the data layer that powers both Preply's analytics, machine learning, and product. You will work closely with ML Platform, Applied/Data Scientists, Analytics

Software Dev Engineer, Cradle (Accuro) Detroit, Michigan, USA Build exabyte-scale data processing engines with leading-edge technology. If you want to work with Spark, Java, Scala, Python, Flink, Beam, and AWS services (EMR, EKS, Kinesis, DynamoDB, SQS) - processing and transforming data across Amazon's data lakes - this is the role for you. About the Team The BDT team builds the platform that connects millions of businesses to hundreds of millions of customers across Amazon's global marketplaces. We enable native SQL, machine learning, and functional transformations using Apache Spark - executing batch, ML, and streaming workloads over schema'd data in S3, and writing curated datasets to front-end caches (DynamoDB, Redis, ElasticSearch). Our platform lets customers transition seamlessly between Streaming, Batch, Cache, and Analytics to meet demand at Amazon scale. We are a team that relentlessly innovates, keeps customers at the center, and delivers results with velocity while continuously raising the bar. What We're Looking For • Background in distributed systems development • Solid technical ability and passion for building at scale • Good communication skills and a bias for action Motivation to deliver results in a fast-paced, high-impact environment Key job responsibilities • Design & Delivery - Lead the design and delivery of major features and re-architecture of significant technology components • Quality & Operational Excellence - Ensure software maintains a high bar for quality, security, architecture, and operational excellence • Problem Solving - Identify root causes of widespread issues - including those limiting innovation and delivery velocity - across systems

Data Platform Lead Remote (US) | Remote Description Jump is transforming the live sports experience with the only end-to-end fan engagement platform built specifically for sports teams and venues. By focusing on aligned incentives between teams and fans, our platform unifies ticketing, merchandise, and game day operations - creating a smoother, more fan-friendly experience. Founded in 2021 by e-commerce innovator Marc Lore, MLB legend Alex Rodriguez, and entrepreneur Jordy Leiser, we've raised $60 million from top investors including Alexis Ohanian's Seven Seven Six and Forerunner Ventures. Our platform powers teams across the NBA, WNBA, and NWSL helping them boost ticket sales and deliver innovative fan experiences. We're a remote-first team driven by core values - begin and build with trust, play like the underdog, win as a team, and do your thing. If you're collaborative, adaptable, and eager to shape the future of live sports, Jump is the place for you. The Role Jump's data platform is at an inflection point. What started as an internal capability now powers the analytics and insights our clients rely on every day. We're turning it into a full product: scalable, multi-tenant data infrastructure, an expanding library of integrations pulling from a growing set of first- and third-party data sources, and an AI intelligence layer that transforms raw data into actionable decisions. As Data Platform Lead, you'll own the vision and execution of that platform. You'll lead and grow our data engineering team, partner closely with our AI engineer to shape how agents and

DNASURE_ExxonMobil - Lead Azure Data Engineer - TL Mexico Lead Azure Data Engineer (Ingeniero líder de Datos Azure) Location: Mexico (Mexico City, Guadalajara, Monterrey) - Hybrid as per Infosys Mexico policy (CDT/EDT time) Role Overview As a Lead Azure Data Engineer , you will drive the architecture, design, and optimization of scalable data platforms and pipelines on Azure, while providing technical leadership and guidance to engineering teams. You will apply advanced expertise in SQL, Python, and Spark to build efficient ETL/ELT solutions, lead data modeling and performance tuning efforts, and ensure seamless integration with diverse data sources and BI tools. Additionally, you will establish and enforce data governance, security, and compliance standards, collaborate with stakeholders to translate business needs into robust solutions, and mentor team members to ensure high-quality delivery and alignment with enterprise data strategy. Key Responsibilities - Monitor and respond to alert emails regarding execution failures across all team tools-such as Databricks, Azure Data Factory, or data quality gap reports within our processes. - Proactively respond to user questions on team Zoom channels, capture issues in our processes, and implement necessary fixes. - Work extensively with Azure Cloud Services, Azure Data Factory, Databricks, Python, and Spark, snowflake including handling user requested break-ins such as issue investigation, missing data analysis and remediation of jobs or pipelines that did not run according to their pre-configured schedules. Basic Qualifications - Bachelor's degree or foreign equivalent in Computer Science or related field. - 8+ years of IT experience , including pipeline design,

Senior Data Engineer (Databricks) 2 Locations posted: Posted 9 Days Ago

Inventory Team Lead (Weekend Shift 6am- 6pm) Newark, Delaware posted: Posted 4 Days Ago

Senior II Data Engineer - Ingestion and Enrichment team London, Greater London, United Kingdom We power people's progress. At Preply, we're all about creating life-changing learning experiences. We help people discover the magic of the perfect tutor, craft a personalised learning journey, and stay motivated to keep growing. Our approach is human-led, tech-enabled - and it's creating real impact. We've just reached unicorn status with a $150M Series D, accelerating our vision to transform education through human-led, AI-enhanced learning. Today, 100,000+ tutors teach 90+ languages to learners in 180 countries - and we're only getting started. As a category-defining company, we're shaping what the future of learning looks like at global scale. Every Preply lesson sparks change, fuels ambition, and drives progress that matters. Joining Preply means helping define the future of education at global scale, and building something that truly matters for millions of people, every day. Meet the team! At Preply, the Data ingestion and enrichment team provides a single, trusted, and scalable data foundation. The team ensures that all analytics, machine learning, and product features are built on unified, governed, and production-grade data assets in Preply's Lake House, including the extraction, normalization, and generation of structured data from Preply's unstructured assets, forming a durable data moat for AI-driven products. As a Senior II Data Engineer in the Data Ingestion and Enrichment team , you will own and drive technical vision for the data layer that powers both Preply's analytics, machine learning, and product. You will work closely

Senior/Software Engineer - Data Analytics Pune, IN About the Role PubMatic is seeking Data Analytics-focused Senior / Software Engineers with expertise in building and optimizing AI agents, including strong skills in Hadoop, Spark, Scala, Kafka, Spark Streaming, and cloud-based solutions, with proficiency in programming languages such as Scala and Python. You will be responsible for developing advanced AI agents that enhance our data analytics capabilities, enabling our platform to handle complex information retrieval, contextual understanding, and adaptive interactions, ultimately improving our data-driven insights and analytical workflows. What You'll Do: Build, design, and implement our highly scalable, fault-tolerant big data platform to process terabytes of data and provide customers with in-depth analytics. Develop backend services using Java, REST APIs, JDBC, and AWS. Build and maintain Big Data pipelines using technologies like Spark, Hadoop, Kafka, and Snowflake. Architect and implement real-time data processing workflows and automation frameworks. Design and develop GenAI-powered agents for analytics, operations, and data enrichment use cases using frameworks like LangChain, LlamaIndex, or custom orchestration systems. Integrate LLMs (e.g., OpenAI, Claude, Mistral) into existing services for query understanding, summarization, and decision support. Manage end-to-end GenAI workflows including prompt engineering, fine-tuning, vector embeddings, and retrieval-augmented generation (RAG). Working closely with cross-functional teams on improving the availability and scalability of large data platforms and the functionality of PubMatic software. Participate in Agile/Scrum processes such as sprint planning, sprint retrospective, backlog grooming, user story management, and work item prioritization. Frequently discuss with product managers about the software features to include in the

Acceleration Center - Advisory National - Data Engineer - Senior Associate Ciudad de México, Mexico Join our team as a Senior Associate, Data Engineer and help design, build, and optimize scalable data solutions using PySpark, Python, SQL, and Databricks. Leverage advanced analytics, CI/CD, and ML frameworks to drive business insights and innovation. Grow your expertise in a dynamic, client-focused environment at the forefront of data engineering excellence.

Summer Camp Counselor - Downers Grove Downers Grove, Illinois, United States of America As a Camp Counselor at Bright Horizons, you'll help create fun, meaningful summer experiences children will remember long after camp ends. From leading games to sparking creativity and curiosity, y...