Lead Java Developer

Role: Lead Engineer – Java Location: Chicago, IL Contract Mandatory Skills – Java, Spark, Data and Cloud (AWS / Azure / GCP) GCP Preferred Job Description8–12 years of experience in production-grade software engineering and data engineering, with a strong foundation in Java-based application development.Demonstrated progression from hands-on Java development roles into data engineering and platform-level responsibilities.Extensive experience designing, building, and operating Spark-based batch data processing systems using Java in cloud or distributed environments.Proven experience working on shared data platforms that support multiple downstream analytics use cases, reporting systems, and business functions.Strong exposure to enterprise data processing workloads, including large-scale structured and semi-structured data handling with performance and reliability considerations.Key Expertise1. Technical SkillsDeep hands-on experience with Java as the primary programming language, including building scalable and maintainable applications for data processing and backend systems.Strong working knowledge of Apache Spark using the Java API, with the ability to design and implement robust batch processing pipelines.Experience working with cloud-based data platforms (GCP preferred), including services such as BigQuery and Cloud Storage, or equivalent services in other cloud environments.Strong understanding of data storage formats and access patterns, including Parquet, Avro, and JSON, with a focus on optimizing data layout for analytical workloads.Experience implementing CI/CD practices for data engineering solutions, including source control strategies, automated deployments, and environment promotion across development, testing, and production.Solid understanding of data security fundamentals, including secure data access patterns, credential management, and compliance-aware data handling.2. Architecture & DesignOwnership of solution and platform-level architecture for batch data processing systems built on Java and Spark.Strong foundation in data modeling principles, including normalization, denormalization, and analytics-oriented schema design based on consumption patterns.Proven experience designing and enforcing layered data architectures, including clear separation of raw, processed, and curated data layers.Ability to define and document architecture standards, design guidelines, and reusable frameworks for ingestion, transformation, and consumption layers.Experience reviewing technical designs across teams to ensure alignment with scalability, performance, and maintainability requirements.Strong understanding of integration patterns across upstream source systems and downstream consumers such as BI tools and reporting platforms.3. Big Data & AnalyticsDeep understanding of OLTP and OLAP concepts, and the implications of analytical workloads on storage layout, compute sizing, and query performance.Proven experience designing and optimizing ETL / ELT frameworks capable of handling large volumes of structured and semi-structured data with predictable performance and reliability.Strong expertise in Spark performance tuning techniques, including partitioning strategies, join optimizations, caching decisions, and query execution analysis.Experience supporting enterprise analytics use cases by delivering high-quality, well-modeled datasets suitable for consumption by BI and reporting tools.Ability to diagnose and resolve complex data issues related to:LatencyData correctnessSchema driftPipeline failures in production environments4. GenAI Adoption & AutomationPractical experience evaluating and adopting AI-assisted development tools to improve developer productivity, code quality, and delivery velocity within data engineering teams.Understanding of how AI-driven techniques can be applied to data engineering use cases, such as anomaly detection, data quality monitoring, and operational insights.Ability to assess emerging GenAI capabilities pragmatically and integrate them into the platform in a controlled, value-driven manner without compromising stability or governance.5. Observability & Performance Optimization (Good to Have)Experience defining observability practices for data platforms, including monitoring of pipeline health, job execution metrics, and operational alerts.Strong hands-on ability to troubleshoot distributed Spark workloads, identify performance bottlenecks, and drive corrective optimizations.Exposure to data lineage, metadata management, or operational dashboards to improve platform transparency and operational maturity.ResponsibilitiesOwn and evolve the solution architecture for Java and Spark-based batch data platforms supporting multiple enterprise use cases.Act as a technical authority for data engineering design decisions, ensuring consistency, scalability, and long-term maintainability of the platform.Guide Technical Leads and Senior Engineers on architecture, design patterns, and implementation best practices through design reviews and hands-on collaboration.Ensure platform implementations meet defined non-functional requirements, including performance, reliability, security, and cost efficiency.Collaborate closely with enterprise architecture, cloud, and security teams to align platform design with organizational standards and constraints.Support delivery planning, technical estimation, and risk assessment for complex data engineering initiatives.Continuously assess platform gaps and drive improvements in architecture, tooling, and engineering practices.Skills & CompetenciesStrong architectural judgment with the ability to balance immediate delivery needs against long-term platform sustainability.Excellent communication skills to articulate complex technical concepts to both engineering teams and senior stakeholders.Ability to operate effectively in ambiguous environments and make well-reasoned technical decisions.Proven capability to mentor and elevate the technical maturity of data engineering teams.