First up, this article will be a little more detailed and technical than our usual posts by Damian.

We thought it might be time to detail some of the more nitty gritty changes and upgrades that Launtel is making in the spirit of transparency that Launtel is so famous for.
For those that want the short of it but can’t be bothered with my ramblings:
TL;DR: Our portal is getting faster, bigger, stronger, more lean and more mean!

Launtel is always trying to improve our reliability and speed for public facing applications and websites.
To this end some time ago we decided that running our residential portal which is a Perl Dancer2 application using Apache’s mod_cgi in FCGI mode just wasn’t going to cut it.

For those that aren’t familiar with FCGI (Fast Common Gateway Interface), it is a method for a web server to talk to a server side processing engine and then send the results back to the web server. Essentially all web servers do is send content to the browser of the person viewing the website, they generally aren’t able to execute any arbitrary code on the server. That task is left up to an application server.
This is a little different on the ubiquitous Apache2 web server.
Apache2 is a web server AND also an application server, it can send content to the browser and process server side scripts, in our case Perl scripts.
Apache2 achieves this via plugins called mods, specifically mod_cgi.
Whilst all this sounds great for a small site and makes set up and getting started with applications really easy, it does not scale past a few thousand users at all.

There’s a Starman waiting in the sky…

Because we run our portal on Perl Dancer2 it made sense to use a Perl-centric application server so that we could decouple the server side application processing for the content serving web server.
This is where Starman comes in, a Perl based and Perl-centric application server that we can proxy requests to for all server side functionality.
This change to the way our portal systems worked resulted in a visible increase to page response time and also a marked decrease in server load. Apache2 will spawn a new worker process each time it needs to process more requests than one process can handle, as you can imagine this can bring a server to it’s knees.
You may have even noticed a speed increase to page load time and general responsiveness if you are a frequent user of the portal.

So Starman works well and scales nicely, however does Apache2?

The answer is no, past a certain number of concurrent client connections Apache2 is woeful at handling these requests and quickly outstrips the server’s available resources.
The issue stems from the fact that Apache2 is from an era before web applications where even really a thing, the web was simple HTML pages and a bit of Javascript delivered to a client’s browser and then forgotten about. These serving of pages and then getting out of the way is what Apache2 is good at.

How do we progress?

Obviously something needs to be done if Launtel’s client base is to grow, our portal will need to be able to handle tens of thousands of requests over the course of a day and sometimes thousands at one time.
The solution is to replace literally every aspect of our web stack and deliver a much more powerful and performant product that will scale for years to come.

It comes in the form of two pieces of software: NGINX and uWSGI.

NGINX is a high performance and lean web server and reverse proxy. That’s all it is, it’s not an application server and it cannot be made into one.
We already run this web server elsewhere but the server that the portal runs on is serving many other internal and external applications and until recently, ripping out Apache2 seemed like a daunting task.

That is until we discovered uWSGI.
WSGI stands for Web Server Gateway Interface and as the name suggests is for a web server like NGINX (Or even Apache2 if you really feel like using it) to talk to an application server. Remember CGI that we spoke about before? Think of WSGI as SUPER CGI.
It is essentially a common set of APIs and models for a WSGI compatible web server/gateway and application server to talk to each other. So any server of any kind that can talk WSGI can talk to any other server that also supports WSGI.

Think of WSGI as SUPER CGI.

uWSGI is a multi language implementation of WSGI that will allows us to have every application on Athena (Our portal server) to utilise this one common application server. uWSGI will handle the scaling up or down of processes and threads and can also be “hot reloaded” in the case of changes to the code base, resulting in smaller downtime windows from minute long tear down and stand up cycles with Apache2 to just seconds with NGINX+uWSGI.

Hopefully this has been at least somewhat informative and interesting, if we get enough positive feedback about this we may consider doing more technical articles in the future.

Adam Heathcote
Developer Operations
Launtel