Senior Software Engineer in Data
Senior Software Engineer In Data
Team: CoreTech
Location: Remote
Summary:
Kamu is developing a novel data technology that, similarly to the invention of an SQL database four decades ago, will write a new chapter in humanity's transition towards data economy. This is your opportunity to join an ambitious early-stage startup that has already secured funding and reached product commercialization, and work on an industry-redefining product from a place of relative financial stability.
About Us:
Kamu is the first of a kind decentralized data lakehouse that handles large volumes of structured and highly dynamic data (Industrial IoT, DePIN, scientific data). Think of it as Databricks / Snowflake of Web3, where data can be efficiently shared between individuals and companies while protecting privacy and personal data ownership. Kamu's verifiable stream processing network allows people to build pipelines that continuously clean and integrate data from millions of sources globally, while ensuring that data is 100% auditable and trustworthy, no matter how many hands it went through. Think of it as GitHub for data pipelines, where people can collaborate on data processing on the similar scale as on software. Kamu blurs the boundary between on- and off-chain data by providing the cheapest way to query massive volumes of data from smart contracts and query data from blockchains side-by-side with industry and government data. Kamu brings superior automation, accountability, and infinite composability to the data flows that underpin our society. Kamu is backed by multiple investors and companies including Protocol Labs (the creators of IPFS and Filecoin) and Dell Technologies. We are a: Distributed, multinational company with presence in Canada, Ukraine, and SpainDeeply technical team, with decades of experience in big data, distributed software, and a few PhDs in AI/ML and computer science Team that takes pride in delivering quality products and efficient workflows Believers in core values of Web3, with a very pragmatic view of achieving them Open Source enthusiasts and constantly share progress with the community through publications and conferences.
About You:
You have sharded databases, tuned replication, and dove into intricacies of transaction isolation levels. You know your way around OLAP cubes and transitioned companies from map-reduce to data lakehouses and fabrics. You have developed countless data APIs, optimized join performance in your favorite dataframe library, and wrote a few of analytics engines of your own.
You have a burning passion for data ... yet also a lingering sense that something is missing. You can't help but wonder:
- Why data flows in every company resemble a Rube Goldberg machine
- Why there are hundreds of different analytical databases, while the vast majority of organizations in the world still cannot afford anything beyond Excel?
- Why two decades into Big Data age everyone still struggles to keep even small data up-to-date and of good quality?
- Why despite the mantra of "breaking down the silos" enterprise data produced nothing but silos?
Why reuse and collaboration in data does not exist, and all attempts to create cross-company data repositories turns into "data graveyards"?You, just like us, feel that the world of data is ready for a major innovation that will shake up the status quo. So instead of continuing the "rat race" towards bigger and more performant data that benefits only big tech companies - you want to apply your skills to something that can make a real difference in the world and democratize the data globally.
If this is true - you should talk to us!
Responsibilities:
As a Data Engineer in Kamu you will be working on the core technologies that serve our network and the platform:
- A stream-oriented data format for structured dynamic data that can work with conventional (S3, HDFS) and decentralized (IPFS, Arweave) storage
- A metadata format that serves as a passport of data and describes every event that influenced it
- A protocol for 100% verifiable, reproducible, and auditable multi-party data processing
- A fleet of plug-in data processing engines
And an infrastructure that turns this technology into a novel decentralized and near real-time data lakehouse!
Core technology stack:
Rust
ParquetApache Arrow
Streaming (temporal) SQLApache Spark, Flink, DataFusionIPLD, IPFS, Filecoin
Kubernetes, AWSEthereum blockchain
Your work will include:
Evolving the core data formats and protocols
Improving the existing data engines and integrating new ones
Building an efficient distributed processing infrastructure for running data pipelines and API queries
Designing data access APIs for ingress and egress of data
Building a federated data sharing and compute network
Integrating Kamu with 3rd-party data providers and consumers
Integrating Kamu with blockchain decoding/indexing technologies
Research and implementation of features like:
Privacy-preserving compute
Fine-grain provenanceAI/ML integration with Kamu data pipelines
Communicating your progress to users and the community
Contributing to the product documentation and automated testing
Requirements: BSc in CS or equivalent experience 6+ years of industry experience
Required skills:
- High proficiency in Rust, Java, or ScalaStrong knowledge of SQL and database internals
- Modern data lakehouse architecture and horizontal scaling
- Data science toolkits (Pandas, R)
- Data integration systems and patterns
- Software quality (test pyramid, CI/CD)
Bonus skills:
- Structured data formats (Parquet, Arrow)
- Stateful stream processing fundamentalsCDC, Event sourcing
- Docker, AWS, Kubernetes
- Data visualization (PowerBI, Tableau, Jupyter)
- Development methodologies (Agile, Scrum)
- Open source collaboration
- Blockchain indexing and analytics (Dune, TrueBlocks)
- Decentralized storage (IPFS)
- Good written English skills, ability to write clear documentation
What We Offer:
- Remote work with flexible hours
- Competitive salary, equity$1,500 home office equipment stipend21 days of paid vacation per year
- Conference travel and education budget
Location:
We are a fully remote team, but have a strong preference for candidates residing between UTC-08:00 (Pacific Coast) and UTC+02:00 (Eastern Europe) time zones for maximal overlap between the team members.