Data Flow Engineer (Warsaw, 40% remote/60% onsite) – EU Public Institutions

Data Flow Engineer (Warsaw, 40% remote/60% onsite) – Frontex

Profile: Data Flow Engineer (Data Flow / Data integration / NiFi / Big Data Streaming & Flow Management).

Place of performance: Frontex Headquarters in Warsaw (Poland), 40% remote/60% onsite.

Duration of the mission: 48 months.

Security Clearance: Restricted / EU Restricted.

Minimum level of education: Level 6.

Minimum English language skills: B2.

Minimum IT relevant experience: 8 years (6 years in relevant roles).

Award Criteria: 50% Price / 50% Quality.

Minimum required scoring for interview: 60%.

Rate: The rate offered depends on the candidate’s level, in accordance with the European Commission’s public grading system. Further details are available upon discussion, for the following estimations:

· Estimated NWH: 230days x4 years.

· Estimated EWH: 23days x4 years.

· Estimated OnC: 3000h x4 years.

Required technical certificates:

At least 1 certification among:

· Cloudera Certified Developer for Apache NiFi or an equivalent certification.

· Cloudera Data Flow (CFM) related certification or an equivalent certification.

· Each equivalent certification must be recognized internationally (subject to acceptance as a valid credential by the Contracting Authority).

Knowledge and Skills

· 01] Expert knowledge in defining, designing, implementing and maintaining complex data flows in Apache NiFi (Cloudera DataFlow).

· [02] Adanced Python programming skills (data processing, NiFi custom logic, flow automation, integrations).

· [03] Advanced in building integrations based on REST API (endpoints' calling, OAuth/JWT authentication, rate limiting, error recovery).

· [04] Hands-on experience in building CDC-based data flows (Change Data Capture) - using native NiFI processes/connectors as well as with SQL Builder.

· [05] Good knowledge of Apache Iceberg (tables, schema evolution, partitioning).

· [06] Knowledge on data governance and catalog in CDP: Apache Atlas (metadata, lineage, tagging) and Apache Ranger (security policies, authorization).

· [07] Experience with Apache Kafka as message broker (topics, producers/consumers, schema registry, NiFi integration) and Apache Avro as serialization standard (schema evolution, compatibility etc.).

Specific Expertise

· [01] Hands-on experience in daily working with Apache NiFi, preferably in CDP environment - design, deployment, monitoring, troubleshooting of advanced flows - min. 2-3 years of hands-on.

· [02] Documented experience in realisation of at least one big integration project using NiFi as central tool (API calling, db integrations, transformations, data routing and delivery).

· [03] Practical knowledge and experience with Apache Iceberg, preferably in CDP environment - table creation/management, NiFi/Spark/Flink integration etc.

· [04] Experience in implementation of CDC pipelines to/from relational databases.

· [05] Practical knowledge of configuring and managing governance/lineage in Apache Atlas + Ranger in the context of NiFi flows (tagging, policies, audit).

· [06] Experience in working with Apache Kafka in CDP ecosystem (integration NiFi -> Kafka -> downstream consumers, schema management with Avro).

Tasks and Responsibilities

· Design, implementation, testing and maintenance of complex data flows in Cloudera DataFlow (Apache NiFi) - ingest, transform, enrich, route, egress.

· Building and optimisation of CDC-based pipelines (real-time / near-real-time) with usage of NiFi, Kafka, Debezium / SQL CDC connectors.

· Integration with external systems via REST API, JDBC, Kafka and other protocols.

· Managing data schemas (Avro), metadata and lineage in Apache Atlas.

· Configuration of security and governance (Ranger policies) for data flows.

· Monitoring, alerting and troubleshooting performance / reliability of data pipelines.

· Collaboration with data engineers, architects and business stakeholders on defining requirements and architecture of data flows.

· Creation and maintenance of SOPs and documentation for data flows, and runbooks.

· Participation in CDP / NiFi / Kafka upgrades and migrations.

Travel: By default, travelling in the interest of service is not foreseen for this position/profile.

· Nevertheless, Frontex may exceptionally request to carry out some services at other locations than Frontex Headquarters or other Contracting Authority’s premises.