Senior PySpark Data Engineer
Luxoft
- Rad od kuće
We are hiring for projects in the Middle East. The team comprises front-end and back-end developers, data analysts and data scientists, architects, analysts, and project managers. You will work with modern technologies and be an integral part of continuing to evolve our high-performing platforms. If you are serious about being a part of this fast-paced and exciting journey, this project should be top of the list!
This role will be primarily responsible for closely collaborating with a business SME to implement data validation rules.
Responsibilities
Collaborating with a business SME to implement data validation rules.
Understand business requirements and actively provide input from a data perspective.
Understand the underlying data and flow of data.
Understand overall platform architecture and capabilities and how to load data effectively.
Recognize and adapt to the changes in processes.
Skills
Must have
7 years + of relevant experience.
Proficient hands-on experience with PySpark-based data platforms:
Databricks, Cloudera, Apache Spark, etc. (3y+).
Proficiency in Python and PySpark (5y+).
Advanced knowledge in SQL (5y+).
Should be able to analyse and understand complex data.
Knowledge of Azure data/delta lake, Azure DevOps, CI/CD, and Git is essential.
Should clearly understand the data platform lifecycle and contribute to preparing Design documents, Unit Test plans, and Code review reports.
Knowledge of data Lake house and business intelligence architecture.
Experience working in an Agile environment (Scrum, Agile) is a plus.
Interpersonal/communication skills (both oral/written) with the ability to communicate at various levels with clarity & precision.
Bachelor's degree in IT/Computer Science or related field
Nice to have
Knowledge of the financial markets, portfolio theory, and risk management is a plus
Other
Languages
English: C1 Advanced
Seniority
Senior
- ...What you will do Build, test, document, and deploy reliable data pipelines, working with diverse data systems such as Snowflake, ClickHouse... ...SaaS). Provide a robust dbt runtime environment for Analytics Engineers, offering guidance and support in building resilient...
- ...company founded by former Google Cloud leaders, architects, and engineers. We are seeking candidates with significant experience in Google... ...for customers transitioning to Google and seeking assistance with data and application modernization. We embed within customer teams to...
- ...for virtual testing of ADAS systems using data-driven development approach. Tools to be... ...German car maker, we are looking for talented engineers and developers. Join our enthusiastic... ...Seniority Regular...Rad od kuće
- ...systems: you'll develop systems capable of handling hundreds of thousands of events per second and manage databases with terabytes of data. You'll build and maintain our data warehouse: you'll create ETLs that integrate data from various sources, optimizing it for analysis...Rad od kuće
- ...are not met. ~ Develop segmentation techniques to partition data (at the level of users, countries, etc.) into optimal groups.... ...decisions to various stakeholders (CMO, user acquisition team, data engineering team, business analytics team). ~ Write optimized queries...Rad od kuće
- ...Familiarity with the financial markets, related data and applications, providing a contextual... ...Proficiency in utilizing PySpark for distributed data processing and analytics... ...1 Advanced Seniority Senior...Rad od kuće
- ...traffic performance; Testing marketing team hypotheses through data analysis; Providing recommendations to the team from an... ...libraries). Preferred qualifications: ~ Bachelor’s degree in engineering, mathematics, statistics, data science, or a related field....Rad od kuće
- На які завдання (обов'язки)? ~ Зв'язати SQL бази данних; автоматизація роботи з використанням API; Проектування та розробка пайплайнів для збору, обробки та збергіання даних; Робота з командою аналітики, для ідентифікації бізнес-ідей і потреб; Створення масштабованих...Rad od kuće
- ...stakeholders: Partner with cross-functional teams to deeply understand their analytics needs, translating business requirements into robust data foundations and BI solutions that enable reliable, actionable insights Automating reports and alerts: Set up automated reporting and...Rad od kuće
- Python Engineer (Data Processing) Description EPAM is a leading global provider of digital platform... ...We are looking for a Mid-level, Senior and Lead Python Engineers to strengthen... ...Experience with Numpy, Pandas, Polars, PySpark Experience in one or more Cloud Technologies...Rad od kuće
- ...About the Role The Engineering Manager position at Super Dispatch involves overseeing a team... ...such as Growth, Operations, Engineering, Data Science, Product, Design, Sales, and... ...have a strong technical background as a Senior Engineer for at least 3 years on the web...Rad od kuće
- ...passionate team of cross-functional engineers who love challenges and want... ...team is new and looking for Senior engineers who will help us... ...backend services, integrate with data warehouse and other services... ...Requirements You're proficient at PySpark and Python (python, databricks...Rad od kuće
- ...for landlords), and Mynd (property management platform for both Retail and Institutional investors). About the team The Data Engineering team is the core of Data, which everything else relies upon. The team is responsible for the development and management of the...
- ...tools, timezones: GMT +/- 3 hours, full time or part time (at least 32h/week)) Join Checkly , a Gartner Cool Vendor , as a Senior Backend Engineer and be at the forefront of the Monitoring as Code (MaC) movement. We're a remote-first startup on a mission to enable...
- ...5+ years of working experience in DevOps/Cloud Engineer roles At least 3 years of production... ...Other Languages English: B2 Upper Intermediate Seniority Senior...Rad od kuće
- ...serving as the largest provider of COVID-19 solutions. Our Data Science, Digital and IT careers are a unique opportunity to... ...#TechWithPurpose Role Description: We are looking for a dedicated Senior Test Automation Engineer to join our dynamic team within PPD, the clinical research...
- ...Job Description We are currently looking for a skilled & passionate Senior Frontend Engineer (m/f/d)to become part of our Purchasing Interface Team at ABOUT YOU! Our Purchasing Interface Team is the main system for the operational processes of our buying, procurement...
- ..., Turkey, Latvia and many others. We are looking for a Senior Test Automation Engineer . Main Requirements: Experience in testing web services... ...knowledge of relational databases (indexes, transactions, data normalization); Familiarity with other types of...Rad od kuće
- ...strategies. What kind of professional are we looking for? ~ QA Engineer experience of 5 years or more; Attention to detail and... ...development process. Nice-to-Have: Practical experience with Unreal Engine/Unity/other game engines; Knowledge of performance testing;...Rad od kuće
- ...convert more shoppers into customers, driving GMV. ~ About The SRE Team We are seeking a highly skilled and experienced Senior Site Reliability Engineer (SRE) to join our team. As an SRE at Gorgias, you will play a crucial role in ensuring the reliability, scalability, and...Rad od kuće
- ...This is a general track for Senior Engineer or above in any team at Canonical. After the first round of interviews we will establish which teams... ...experience REST and gRPC API experience SQL and NoSQL data store expertise Experience with public clouds or OpenStack...
- ...Thailand, Indonesia, Hong Kong, Japan, Australia, Poland, Israel, Turkey, Latvia and many others. We are looking for a Senior Manual QA Engineer . Technology Stack: - Jira+Confluence; - Testrail; - MS SQL Server; - Fiddler/Charles; - Devtools. Your main...Rad od kuće
- Над чем работать? Жанры: Приключения, Казуальные Платформы: Mobile, iOS, Android На какие задачи (обязанности)? Разработка и ведение тестовой документации: чек-листов, тест-кейсов и тест-планов; Обеспечение качества продукта путем ручного тестирования продукта;...Rad od kuće
- We are looking for an experienced QA Engineer to join our team. Your role will be key in... ...strong background in test automation and data quality, who is also a fast learner. Familiarity... ...1 Advanced Seniority Senior...Rad od kuće
- ...problems and appreciate recognition for excellent work. As a core engineer you will step into a greenfield opportunity where your efforts... ...will be working on cutting edge tech including NLP, firmographic data, entity resolution and much more. In this role you will work to...
- What are you working on? The name of the project(s): Dancing Drums Genres: Casino Platforms: iOS, Android For which tasks (responsibilities)? ~ Design, write, and execute test cases and test suits New feature, regression, exploratory testing Configure environments...Rad od kuće
- ...version, enhancing overall gameplay experience. What kind of professional are we looking for? ~ Extensive experience with Unreal Engine 5, including a strong understanding of Blueprints, C++, and rendering techniques. Expertise in UI/UX design and implementation using...Rad od kuće
- ...For which tasks (responsibilities)? Designing and creating realistic visual effects (VFX) for game projects on the Unreal Engine platform. Collaborating with artists, designers, and programmers to integrate effects into the game. Optimizing VFX to ensure high performance...Rad od kuće
- ...game development. What kind of professional are we looking for? ~3+ years of experience in game development, preferably with Unreal Engine 4/5. ~ Strong C++ and blueprinting skills in Unreal Engine 4/5 ~ Strong problem-solving skills and attention to detail. ~ Ability...Rad od kuće
- ...player behavior. Develop reports and dashboards to communicate data insights to stakeholders. Collaborate with cross-functional... ...written and verbal. ~ Ability to collaborate effectively with engineers, product managers, analysts, and marketing to deliver unambiguous...Rad od kuće
Želite li da dobijate više obaveštenja o oglasima za posao?
Prijavite se i dobijte obaveštenja o poslovima sličnim Senior PySpark Data Engineer. Budite prvi kandidat koji će se prijaviti!
Bićete spremni za konkurisanje za 5 minuta. Napravite profesionalnu radnu biografiju