Wormholes combines several open source components as building blocks.
- PostgreSQL is a relational database that stores links and users data.
- TimescaleDB is a time-series database that is used to store historical click events for links.
- Redis is an in-memory cache that we use as MRU Cache when storing or retrieving links from PostgreSQL.
It is responsible for generating collision-free IDs. It contains following components —
- Bloom - It's a thread-safe wrapper around bloom. The bloom-filter is populated with all existing IDs on startup.
- Bucket - It's a thread-safe wrapper around a slice of given capacity. It is responsible for storing generated IDs. A bucket can have two states — empty and full.
- Store - It's an in-memory store containing a slice of buckets and a channel. When a bucket it popped, the channel receives index of that bucket.
- Factory - It's responsible for generating IDs. when store channel receives an index for empty Bucket, the factory runs a goroutine to populate that bucket.
In distributed mode, Generator runs with a gRPC server and exposes a
GetBucket method to retrieve one bucket at a time.
Creator is a REST server using Fiber that is responsible for creating, updating, deleting and reading links data. Upon startup and before all of the IDs in a Bucket are used, the creator instance retrieves a Bucket full of IDs from Generator.
Instead of adding generated links immediately to database, It uses a queue to batch-process several links at a time.
Redirector is a REST server that powers link redirection. It also collects all the necessary metrics for each click and sends those to TimescaleDB. This is done with a Pipe containing multiple streams.
Analyzer is a REST server that provides all the stats related to links by running analytical queries against TimescaleDB.