Sr Associate Software Engineer (Databricks Developer)

Job Description
Amgen is looking for highly motivated expert Senior Data Engineer who can own the design & development of complex data pipelines, solutions and frameworks. The ideal candidate will be responsible for designing, develop, and optimize data pipelines, data integration frameworks, and metadata-driven architectures that enable seamless data access and analytics. This role prefers deep expertise in data processing, distributed computing, data modeling, and governance frameworks to support self-service analytics, AI-driven insights, and enterprise-wide data management.

Roles & Responsibilities:

• Design, develop, and maintain scalable data pipelines using AWS (Redshift, S3, Glue, Lambda) and Databricks (Spark, Delta Lake) to support enterprise analytics and reporting
• Architect and implement robust data models (dimensional and normalized) to enable high-performance querying and optimized reporting in Redshift
• Build and optimize batch and real-time data processing frameworks, leveraging Spark Structured Streaming and cloud-native services
• Lead data ingestion, transformation, and orchestration workflows ensuring data quality, reliability, and performance at scale
• Perform advanced data analytics and root cause analysis to troubleshoot data discrepancies, performance issues, and pipeline failures
• Ensure data security, compliance, and role-based access control (RBAC) across data environments
• Optimize query performance, indexing strategies, partitioning, and caching for large-scale data sets.
• Develop and drive continuous improvements in CI/CD pipelines for automated data pipeline deployments, automated testing, version control, and monitoring for data platforms in a cloud-native environment
• Collaborate with cross-functional teams, including data architects, business analysts, and DevOps teams, to align data engineering strategies with enterprise goals.
• Stay up to date with emerging data technologies and best practices, ensuring continuous improvement
• Partner with BI and reporting teams to support report development, dashboard optimization, and data validation for business stakeholders
• Implement data governance best practices including data lineage, auditing, access controls, and performance tuning
• Mentor junior engineers and contribute to architectural decisions, code reviews, and engineering standards

Basic Qualifications and Experience: 

• Any degree with 5 - 9 years of experience in Computer Science, IT or related field

Functional Skills:
Must-Have Skills:

• Strong solution design and problem solving skills
• Hands-on experience in data engineering technologies such as Databricks, PySpark, SparkSQL Apache Spark, AWS, Python, SQL, Redshift, and Scaled Agile methodologies
• Proficiency in workflow orchestration, performance tuning on big data processing.
• Strong understanding of AWS services
• Experience with Data Fabric, Data Mesh, or similar enterprise-wide data architectures.
• Ability to quickly learn, adapt and apply new technologies
• Strong problem-solving and analytical skills
• Excellent communication and teamwork skills
• Experience with Scaled Agile Framework (SAFe), Agile delivery practices, and DevOps practices.

Good-to-Have Skills:

• Good to have deep expertise in Biotech & Pharma industries
• Experience in writing APIs to make the data available to the consumers
• Experienced with SQL/NOSQL database, vector database for large language models
• Experienced with software engineering best-practices, including but not limited to version control (Git, Subversion, etc.), CI/CD (Jenkins, Maven etc.), automated unit testing, and Dev Ops

Education and Professional Certifications

• 5 to 9 years of Computer Science, IT or related field experience
• AWS Certified Data Engineer preferred
• Databricks Certified preferred
• Scaled Agile SAFe certification preferred

Soft Skills:

• Excellent analytical and troubleshooting skills.
• Strong verbal and written communication skills
• Ability to work effectively with global, virtual teams
• High degree of initiative and self-motivation
• Ability to manage multiple priorities successfully
• Team-oriented, with a focus on achieving team goals
• Ability to learn quickly, be organized and detail oriented
• Strong presentation and public speaking skills