Long Story Short: Siddhi - A view of a contributor

Photo by VanveenJF on Unsplash

I have just completed 7 years at WSO2 on 1st January 2020. It was a great journey with a lot of memories to share. All these years, I have worked directly or indirectly related to Siddhi, been a contributor to Siddhi and Siddhi became part of my day to day life. I wanted to write up something around it and thought this would be a good time to share some words about Siddhi and its journey.

Siddhi in the Sinhala language means incidents (things that happen) and in Tamil language, it means knowledge. For whom, not aware of Siddhi, it is a complex event processing and stream processing engine. Siddhi was started as a final year project of the CSE department at the University of Moratuwa (UOM)with the support of WSO2 in 2011. It was co-founded by a few university students from UOM namely Suhothayan Sriskandarajah, Isuru Udana Loku Narangoda, Kasun Gajasinghe and Subash Chaturanga with the guidance of Srinath Perera of WSO2.

As a university final year project, Siddhi was a success. Due to the prominence of the project, it became one of the main projects at WSO2 in 2012. Siddhi was used as one of the core engines with Esper and Drools in the WSO2 Complex Event Processor in 2012. I joined WSO2 on 1st January 2013 to Complex Event Processor Team (which develops and maintains Siddhi) with Suho (Suho was the one and only team member of the CEP team by that time). I still remember Sanjiva asked me to write a Fix adapter for Siddhi as my first task. I have joined WSO2 straight after my graduation; I never knew anything about CEP or Siddhi before joining the WSO2. I only got to know about Siddhi on my 2nd day at WSO2 (There was no preference option given to new joiners to select the team at that time). We have gone through ups and downs in this 7–8 years journey with Siddhi.

Until 2019, Siddhi was a supporting project/product to many other WSO2 products. It is used as a core component within many WSO2 products to cater to the use cases such as real-time analytics, alert generation, event-driven integration, throttling, etc.. In the following sections, I would like to highlight key milestones of Siddhi project and how it evolved.

If you checked the below figure, you will understand the WSO2 Analytics story over the years,

History of WSO2 Analytics Products

In the year 2013 to 2018, Siddhi capabilities are shipped through WSO2 Complex Event Processor, WSO2 Data Analytics Server, and WSO2 Stream Processor to serve the real-time analytics and streaming data integration requirements. In this period, the Siddhi engine has improved with better design, architecture, and implementation to provide high performance. Siddhi 1.x is released in 2012 and now, the Siddhi release version is 5.x.x. If you check the architecture diagrams of Siddhi 1.x and 5.x then you could understand the changes and improvements.

Over the period, we have revisited ourselves, taken feedback from the users and inspired by other vendors in the market to improve the Siddhi engine.

Role of Siddhi at WSO2

Who adopted Siddhi

There is a lot of adoption happened around Siddhi even though Siddhi was not positioned as a first-class product. See the below figure to find the users of Siddhi.

Users of Siddhi
  • Siddhi used as the core component of Uber’s fraud detection system.
  • Siddhi is part of Apache Eagle, a distributed real-time monitoring and alerting engine. This was initiated by eBay and Paypal and currently used in house.

Other than above, it has been used by many undergraduate, postgraduate and doctorate studies as a complex event processing engine as well.

Where are we heading now (From 2019 onwards)?

In early 2019, it was decided to position Siddhi as an independent OSS project at WSO2. As a step, five engineers (Suhothayan Sriskandarajah, Chiran Fernando, Niveathika Rajendran, Buddhi Kothalawala and myself) started working on the Siddhi OSS project. In the process, Siddhi is moved to separate GitHub organization as siddhi-io for better adoption and management and also Siddhi team built a Siddhi GitHub site to spread the news about Siddhi. The primary goal of the siddhi-io is to build a Cloud Native Stream Processor to overcome the complication of building and running streaming applications in Kubernetes and etc…

You could refer to my previous article to get some understanding on Cloud Native Stream Processing and what Siddhi Cloud Native Stream Processor does,

The siddhi-io project consists of nearly 30 committers but the core project team who is working on the project in full time is small in size. Since it is a small team all the engineers in the team get a chance to work on different areas of the project; this includes project development, testing, automation, documentation, evangelism efforts, and community support.

Importantly, we always engage with community users and contributors through siddhi-dev google mail group and Siddhi Slack channel to provide better help and promote community collaboration towards the project. For example, the idea for the Siddhi project logo is decided by the votes of the community members through the siddhi-dev mail thread.

Siddhi-io Logo Proposals

Check out the road map of Siddhi project and provide your suggestions through siddhi-dev google mail group or Siddhi Slack channel.

Key Milestones of Siddhi

  • In 2012, Siddhi 1.x won the bronze award in the Tertiary category at the National Best Quality Software Awards 2011.
  • Siddhi is the only generic complex event processing engine which able to become a finalist in the Distributed Event-based Systems (DEBS) conference grand challenge from 2014 to 2017 consequently (years which Siddhi has participated).
  • In 2015, Siddhi has utilized by Uber to build the scalable event processing solution for fraud detection.
  • In 2015, Experian utilized Siddhi CEP engine to uncover credit intelligence use cases.
  • Siddhi Won the TFL’s data in motion hackathon in 2016 to find solutions to some of London’s key transport challenges through the innovation and creativity of developers working with our open data.
  • In February 2019, Siddhi medium tech blog is introduced to provide official blogs and articles of siddhi-io.
  • In June 2019, Hao Chen became the first external committer to siddhi-io project based on his contribution towards Siddhi and integration of Siddhi to Apache Bahir and Apache Eagle.
  • In July 2019, Siddhi listed in the CNCF landscape as the streaming and messaging technology.
  • In July 2019, Siddhi 5.1.x release was done with a lot of refactoring for better performance and architecture to support building cloud native streaming applications.
  • In August 2019, Siddhi Slack channel is initiated to provide better support for community users and contributors.
  • In September 2019, Siddhi community call is introduced and it is scheduled to run weekly on Tuesday at 7.30 AM PST.
  • In December 2019, Siddhi earned the CII Best Practices Badge from Linux Foundation Core Infrastructure Initiative.
  • In 2019, Siddhi GitHub repo got nearly 400 new Stargazers (which is the highest for Siddhi) due to organic traffic and content based evangelism effort.

Request for OSS Users

It is always challenging to keep an open source project active. Every user must understand the ethics of an OSS user. As an OSS user, it is important that he/she contribute back to the project. While it’s easy to just use without giving back but it is your responsibility to ensure that these projects continue to improve and stay healthy. This does not mean that you should definitely pay something as such. You could contribute back to the project in the following ways,

  • Give a star to the GitHub repository if you felt that the project is beneficial for you.
  • Send a PR if you made an improvement for documentation/code or developed a feature or fixed a bug.
  • Raise a GitHub issue if you found any bug while you are using the project/product.
  • Respond to the queries raised by other community users/contributors.
  • Spread the news about the OSS project (Siddhi).
  • Share your use case as a reference (or provide a testimonial).

In the real open source, you have to give back as equal to what you take.

Thanks for reading the article.

Senior Tech Lead, Speaker @ WSO2. Closely works on Stream Processing and Integration