Data Engineer

Data Engineer
Location: San Jose, CA (Remote)
About LatentView
LatentView Analytics is a global data analytics and AI solutions company that partners with Fortune 500 and high-growth clients to turn data into competitive advantage. Our teams combine deep technical expertise with business context to ship measurable outcomes — not just models and dashboards — across industries.
Job Role :
We operate a large-scale security data platform on a modern lakehouse that normalizes vulnerability findings from multiple scanners into a common, standardized schema and drives downstream ticketing. We are onboarding two new scanner sources - one cloud-security-posture scanner and one web-application (DAST) scanner - and need an engineer to build and productionize those integrations end to end.
Responsibilities :
Design and build incremental ingestion pipelines (watermarking, deduplication) for the two new sources against upstream data tables.
Map each source's native findings into the platform's common security schema, including severity/risk normalization and first-seen/last-seen handling.
Solve asset/target correlation for each source - associating findings with the right infrastructure or application entities and their ownership.
Integrate findings into downstream ticketing and exception-handling workflows, including which findings are ticketable and how ownership is attributed.
Merge new sources into a central large-scale table safely (schema evolution, unique-key handling, collision checks) and validate propagation into reporting/history views.
Reconcile output against each source's console (counts, coverage, dedup), and produce documentation and operational runbooks.
Work as an embedded member of the existing security data engineering team (not an isolated solo contributor) - collaborating day-to-day with the platform/security engineering team and other team members on shared schema conventions, merge patterns, and table ownership so new-source integration work lands safely alongside everyone else's changes.
Required skills :
Databricks, AWS, Python, SQL, Pyspark, Databricks, AWS, Python, SQL, Pyspark.
Base pay range: USD 144,000– USD 166,500 annually. This range reflects base salary only — it excludes Incentive and benefits — and is a good-faith estimate for this level and location that may vary based on experience, skills, and where the role is performed.
Communication: This role requires fluent spoken and written English to communicate directly with client stakeholders — a job-related requirement of the client-facing work itself, not of any candidate's background or origin.

Work arrangement
Yes