Senior Developer

The Senior Developer - Detection Automation will partner with high-powered security and detection research teams to create automated detection pipelines for our suite of cybersecurity products and services. They will partner with researchers as a data engineer, familiar with SQL, Spark, and either/Databricks or Athena/Iceberg/Trino to write and optimise ETL/ELT jobs to ingest raw data from a variety of sources, refine that data, and optimise/tune those datasets into production. They should be comfortable with relational and non-relational databases like SQL and PostgreSQL, big data systems such as Spark, and cloud deployments of software and ETL jobs in AWS. Workflow orchestrators such as Airflow will also be used. Knowledge of cybersecurity and detection engineering is a bonus.
Responsibilities:
Execute on R& D of distributed, highly scalable, and fault-tolerant microservices.
Use test-driven development techniques to develop beautiful, efficient, and secure code.
Create and scale high-performance services that bring new capabilities to Arctic Wolf's data science organisations.
Identify problems proactively and propose novel solutions to solve them.
Continuously learn and expand your technical horizons.
Requirements:
Will collaborate closely with our data science and detection research teams across different cybersecurity domains to define research detection infrastructure requirements and build critical data services.
Has proficiency in big data technologies such as Apache Spark, Databricks, Kafka, SQL, and Terraform.
Has experience interacting with and author workflows, such as prompts or tools, for LLMs, in AWS Bedrock.
Has extensive experience with data pipeline tools (Flink, Spark or Ray) and orchestration tools such as Airflow, Dagster or Step Functions.
Has knowledge of Data Lake technologies, data storage formats (Parquet, ORC, Avro), and query engines (Athena, Presto, Dremio) and associated concepts for building optimised solutions at scale.
Maintains an expert level in one of the following programming languages or similar- Python, Java, Go, Scala.
Is an expert in implementing data streaming and event-based data solutions (Kafka, Kinesis, SQS/SNS or the like).
Has experience deploying software with CI / CD tools, including Jenkins, Harness, Terraform, etc.
Has hands-on experience implementing data pipeline infrastructure for data ingestion and transformation near real-time availability of data for applications and ETL pipelines.

Experience
9-13 yrs