Senior Software Engineer (Databricks)

Job Description: Data Engineering: Strong foundation in data engineering principles, ETL/ELT processes, and data pipeline design patterns PySpark: Proven hands-on experience developing data pipelines using PySpark, including DataFrames API, Spark SQL, and performance optimization Databricks Platform: Practical experience with Databricks workspace, cluster management, notebooks, and job orchestration Workspace AI Agent: Knowledge of Databricks Workspace AI Agent capabilities and integration Data Modelling: Experience implementing data models including dimensional modelling, data vault, or Lakehouse architectures Delta Lake: Understanding of Delta Lake features including ACID transactions, schema evolution, and optimization techniques Python: Strong Python programming skills for data processing and automation SQL proficiency for data querying and transformation Experience with cloud platforms (Azure, AWS, or GCP) Knowledge of streaming data processing (Structured Streaming) Familiarity w...