Header 1

ioan

ioan biticu's website

A whole lot of websockets

A whole load of websockets.

I've built a system that relies on 5 WebSocket servers. The API I am connecting to is a WebSocket server. I ingest the data coming from it on my own three WebSocket servers (each for a different type of data) and allow users of my websites to connect to a replicated WebSocket server for data. The browsers receive the data in less than 0.5 seconds, and yes, there are queues thrown in the mix for reliability: one (redis) queue for each of the connections to the websocket server feeding me data, and then two other (rabbitmq) queues between my 3 websocket servers and the final websocket server users connect to.

The end result? Two websites and a Discord bot. They allow users to be notified when players come online in the game PlanetSide 2, as well as add extra sounds on top of in-game events, such as killing an enemy or being revived. There is no problem if it takes more than a second for the notifications of players coming online, but there is a problem if you are on a killing spree and it takes more than 1 second to hear it - it should be almost instant.

There is another project in the works besides the websites and the Discord bot, and it's a cheating detector. The obvious cheaters in the game can make people rage quit, and it often takes some time for admins to do something about it. By gathering 100M+ game events every month, I will be able to come up with patterns and an app that quickly finds out who the obvious cheater is in the game, currently within 15-30 minutes of feeding it data.

Links to the projects:

A bit more about the project

PlanetSide 2 is an MMOFPS game, and it involves infantry, armoured tanks, and air battles. Although it's downloadable through Steam, it has its own independent friends list. Therefore, it's not easy to know when someone came online. I often play with around 4 people in small vehicles where someone has to drive and the other has to gun. I need to know when a gunner is available, and when I know they are I can join the game. Another use case is for large outfits. It is useful to know when the leaders are online and for players to join the game. Many players want to play with leadership and objectives in mind. And this is where the PlanetSide 2 tracker comes in.

There are other kill event streams that allow you to add custom sounds for game events; however, there is none that I could find that would work for Linux. Initially, I created my own local one in C# because I was inspired by a streamer - this was around 2 years ago. Since then, I decided to work on the cheating detection system. Since I had to save all the data anyway, it didn't take me long to add a website that would also stream the events in real time for what came out as PS2 immersion.

The architecture

I want to show the architecture behind the system.

For the architecture, I've decided to go all in and make it fault-tolerant and scalable. A total of three machines are used in a Docker Swarm. One is the master, and it hosts Traefik for load balancing requests. All machines host Redis instances for locking purposes. All three of the machines host MySQL (1 master, 2 slaves), and there are two machines hosting RabbitMQ (master / slave). The architecture chosen is an event-driven architecture. There are also various containers running for health checks, logging, graphs, and alerting (I get a phone call within 5 minutes if something goes wrong anywhere). The only container that is not replicated at the moment is the bot, but this will also be taken care of in a future version - it's just not in that many servers yet so anything like sharding is not worth it.

Logos

The logo you might not recognise in the image is Dozzle, a Docker container logs viewer. Although I use Loki and Promtail for log history, Dozzle has been great for looking at the most recent logs. Give it a try!

The servers' OS is Flatcar - it's an OS optimised for hosting containers. One thing that I didn't believe I would get used to is not having a package manager. I've just barely managed to install Tailscale because it doesn't need a package manager. Another feature it has is a toolbox - just inputting the command "toolbox" runs Fedora in a container, and you're free to install apps like htop in it.

Note that even though the architecture diagrams below contain a database, it is only accessible with the API containers, and the other containers use service clients to connect to that. The API doesn't connect to the database directly either; there is a load balancer called ProxySQL that routes writes to the master and reads to slaves.

Let's start with what actually gets the data into the system. The containers in the middle live on my home machine because my server IP hasn't been whitelisted yet. They connect to Census, the API for PlanetSide2, via WebSocket for Player activity (online/offline), XP (headshots, revives), and deaths (kills). The container in the last row in the list is making HTTP requests for other data: mapping player IDs from events to player names and getting friends lists. They also connect to the containers on the right via dedicated WebSocket clients (one path for each). Now that they are finally on my server, the events get published onto the queues.

Ps2 ingestors

Below are the listeners that publish events on queues. They arrive in dispatchers, which save the data in the database, and then get routed to the queues that the final Websocket server listens to. This is the websocket server that finally publishes the events to the users listening for them.

Internal listeners and dispatchers - Page 1 (5).png

Traefik binds both host names to the same WebSocket server. The web servers using NextJS are publishing static pages - they don't interact with anything downstream. That work is left for the WebSocket server. All the data needed is stored in localStorage, which is sent when connecting to the websocket server, and the custom songs for the game are stored in IndexedDB because it can store bigger sounds. This might've been overkill because the sounds you want to play when getting a kill are not that big anyway.

Internal listeners and dispatchers

Next, let's have a look at the bot.

Internal listeners and dispatchers

The bot also connects to the WebSocket server and the database. It only sends a message to the websocket server when it needs to check a player's name, and it may have to be fetched from the API (going all the way to my home machine).

The bot also allows users to get DMs when someone in their friends list is online.

Internal listeners and dispatchers

This is working as a cron job, working every hour - it queries the friends list for the Discord users in the database. Since it only gets their player IDs, it also makes queries for the player names where they are missing. This is done because they might not have logged in for a long time, and this system hasn't been online for that long. Next time they are online, they already exist in the system.

Some containers run in ACTIVE-ACTIVE state, e.g., the dispatchers, meaning they pull events from the queues. Some containers, e.g., the listeners, are in ACTIVE-PASSIVE state, meaning only one of them is accepting WebSocket connections. Traefik beautifully takes care of routing only to the active one. The containers running on my home machine are also replicated on a remote Raspberry Pi. This will only get elected if the primary stops receiving messages for some time, or it gets disconnected (for example, my internet is down).

Below are some images from Grafana Grafana

Grafana

Grafana

Conclusion

This is ps2tracker and ps2immersion. I didn't anticipate they would be the projects they are today, and I would learn so much by working on this. After 30-90 days of gathering data, I will be able to work on the cheater detection system. Until then, I will be working on refining the existing system. For example, I have a system discovery that automatically creates queues for the two WS instances, and I want to get rid of it.

For more info about the websites, as well as trying them yourself, please follow the links: