Pyspark Engineer
Job Summary
Job Description
- Proficiency in PySpark
- Strong SQL knowledge
- Data Warehousing Concepts
- Cloud Platform experience (AWS, GCP, Azure)
- Big Data Technologies (Hadoop, Spark)
- Data Modeling experience
- Strong Python skills
- Airflow or other orchestration tools
- Apache Kafka knowledge
- Data visualization tools (Tableau, Power BI)
- Machine learning familiarity
- Agile methodology experience
- Data governance and compliance knowledge
12 skills required
Job Details
Project description
We are looking for skilled PySpark Engineers to join our team, working on a high-impact data engineering project. The project involves processing large datasets, optimizing ETL pipelines, and building scalable solutions to manage complex data workflows. The ideal candidate will collaborate closely with data scientists, data analysts, and software engineers to drive robust, data-driven insights for business decisions.
Responsibilities
Design, develop, and maintain ETL pipelines using PySpark, optimizing for performance and scalability.
Work with large volumes of structured and unstructured data, transforming data to meet business needs.
Integrate data from multiple sources into the data platform, ensuring data integrity and quality.
Collaborate with cross-functional teams to understand data requirements and translate them into efficient data workflows.
Implement best practices for data governance, monitoring, and data security.
Debug and troubleshoot issues across ETL pipelines and data workflows.
Continuously improve performance, scalability, and reliability of existing data pipelines.
Provide documentation and training for data workflows and processes.
Skills
Must have
Proficiency in PySpark: In-depth experience with PySpark for data processing and transformation tasks.
SQL Knowledge: Strong command of SQL for querying and processing data.
Data Warehousing Concepts: Familiarity with data warehousing, data lakes, and data integration principles.
Cloud Platforms: Experience with cloud environments like AWS, GCP, or Azure for data storage and processing.
Big Data Technologies: Hands-on experience with Hadoop and Spark ecosystem (Spark SQL, Spark Streaming).
Data Modeling: Experience in designing and implementing efficient data models.
Python Programming: Strong Python skills, particularly in data manipulation and analysis.
Nice to have
Experience with Airflow or Other Orchestration Tools: Knowledge of workflow orchestration tools for scheduling and monitoring data pipelines.
Knowledge of Apache Kafka: Understanding of Kafka for real-time data streaming and integration.
Familiarity with Data Visualization Tools: Knowledge of visualization tools like Tableau, Power BI, or similar.
Machine Learning Exposure: Familiarity with machine learning concepts, particularly with integrating ML models in data workflows.
Agile Methodology: Experience working in Agile/Scrum environments.
Data Governance and Compliance Knowledge: Understanding of data governance frameworks and compliance standards, such as GDPR.
Other
Languages
English: C1 Advanced
Seniority
Senior
Similar Jobs
Glean
Get notifed when new similar jobs are uploaded
Similar Skill Jobs
Visa
Warner Bros Discovery
PublicisGroupe
Riot Games
PublicisGroupe
Get notifed when new similar jobs are uploaded
Jobs in Sydney, New South Wales, Australia
Canva
Easygo
The Walt Disney Company
The Walt Disney Company
Canva
Flying Bark Productions
Canva
Get notifed when new similar jobs are uploaded
Data Analyst Jobs
PwC
PlayStation Global
Get notifed when new similar jobs are uploaded
About The Company
Luxoft, a DXC Technology Company (NYSE: DXC), is a digital strategy and software engineering firm providing bespoke technology solutions that drive business change for customers the world over. Acquired by U.S. company DXC Technology in 2019, Luxoft is a global operation in 44 cities and 21 countries with an international, agile workforce of nearly 18,000 people. It combines a unique blend of engineering excellence and deep industry expertise, helping over 425 global clients innovate in the areas of automotive, financial services, travel and hospitality, healthcare, life sciences, media and telecommunications.
DXC Technology is a leading Fortune 500 IT services company which helps global companies run their mission critical systems. Together, DXC and Luxoft offer a differentiated customer-value proposition for digital transformation by combining Luxoft’s front-end digital capabilities with DXC’s expertise in IT modernization and integration. Follow our profile for regular updates and insights into technology and business needs.
Get notified when new jobs are added by Luxoft
