This content is password protected. To view it please enter your password below:

Password:

First up, this article will be a little more detailed and technical than our usual posts by Damian.

We thought it might be time to detail some of the more nitty gritty changes and upgrades that Launtel is making in the spirit of transparency that Launtel is so famous for.
For those that want the short of it but can’t be bothered with my ramblings:
TL;DR: Our portal is getting faster, bigger, stronger, more lean and more mean!

Launtel is always trying to improve our reliability and speed for public facing applications and websites.
To this end some time ago we decided that running our residential portal which is a Perl Dancer2 application using Apache’s mod_cgi in FCGI mode just wasn’t going to cut it.

For those that aren’t familiar with FCGI (Fast Common Gateway Interface), it is a method for a web server to talk to a server side processing engine and then send the results back to the web server. Essentially all web servers do is send content to the browser of the person viewing the website, they generally aren’t able to execute any arbitrary code on the server. That task is left up to an application server.
This is a little different on the ubiquitous Apache2 web server.
Apache2 is a web server AND also an application server, it can send content to the browser and process server side scripts, in our case Perl scripts.
Apache2 achieves this via plugins called mods, specifically mod_cgi.
Whilst all this sounds great for a small site and makes set up and getting started with applications really easy, it does not scale past a few thousand users at all.

There’s a Starman waiting in the sky…

 

Because we run our portal on Perl Dancer2 it made sense to use a Perl-centric application server so that we could decouple the server side application processing for the content serving web server.
This is where Starman comes in, a Perl based and Perl-centric application server that we can proxy requests to for all server side functionality.
This change to the way our portal systems worked resulted in a visible increase to page response time and also a marked decrease in server load. Apache2 will spawn a new worker process each time it needs to process more requests than one process can handle, as you can imagine this can bring a server to it’s knees.
You may have even noticed a speed increase to page load time and general responsiveness if you are a frequent user of the portal.

So Starman works well and scales nicely, however does Apache2?

The answer is no, past a certain number of concurrent client connections Apache2 is woeful at handling these requests and quickly outstrips the server’s available resources.
The issue stems from the fact that Apache2 is from an era before web applications where even really a thing, the web was simple HTML pages and a bit of Javascript delivered to a client’s browser and then forgotten about. These serving of pages and then getting out of the way is what Apache2 is good at.

How do we progress?

 

Obviously something needs to be done if Launtel’s client base is to grow, our portal will need to be able to handle tens of thousands of requests over the course of a day and sometimes thousands at one time.
The solution is to replace literally every aspect of our web stack and deliver a much more powerful and performant product that will scale for years to come.

It comes in the form of two pieces of software: NGINX and uWSGI.

NGINX is a high performance and lean web server and reverse proxy. That’s all it is, it’s not an application server and it cannot be made into one.
We already run this web server elsewhere but the server that the portal runs on is serving many other internal and external applications and until recently, ripping out Apache2 seemed like a daunting task.

That is until we discovered uWSGI.
WSGI stands for Web Server Gateway Interface and as the name suggests is for a web server like NGINX (Or even Apache2 if you really feel like using it) to talk to an application server. Remember CGI that we spoke about before? Think of WSGI as SUPER CGI.
It is essentially a common set of APIs and models for a WSGI compatible web server/gateway and application server to talk to each other. So any server of any kind that can talk WSGI can talk to any other server that also supports WSGI.

 

Think of WSGI as SUPER CGI.

 

uWSGI is a multi language implementation of WSGI that will allows us to have every application on Athena (Our portal server) to utilise this one common application server. uWSGI will handle the scaling up or down of processes and threads and can also be “hot reloaded” in the case of changes to the code base, resulting in smaller downtime windows from minute long tear down and stand up cycles with Apache2 to just seconds with NGINX+uWSGI.

Hopefully this has been at least somewhat informative and interesting, if we get enough positive feedback about this we may consider doing more technical articles in the future.

Adam Heathcote
Developer Operations
Launtel

A little over 2 years ago nbn started selling a completely new fibre based product called “Enterprise Ethernet”. Without going into all the technical details, this is a fibre based telecommunications product capable of speeds of up to 1Gbps in both direction (aka “symmetrical”). From the engineering standpoint there is no question that this is a good product for business customers who need it.

The key word here is “need”. In my opinion the product is being heavily oversold to companies who really don’t need it. The original nbn was to be almost all fibre – FTTP. Had that FTTP rollout continued I don’t believe that Enterprise Ethernet would be a product today. Unfortunately, Enterprise Ethernet’s main purpose is to make up for the deficiencies in the network that occurred due to nbn’s switch to the MTM (aka using all the existing copper, HFC etc that was already in the ground).  

Alarm bells started ringing for me when nbn announced some “Business Fibre Zones” around the country. These were areas where nbn was offering Enterprise Ethernet with no upfront install costs. An industrial area north of Launceston, called Bell Bay, was not on this list. Various state and local political figures wanted to know why it had been excluded! 

It had probably been left off the list because it was one of the first areas rolled out with FTTP. They already had fibre! Yet the assumption was that they needed to have Enterprise Ethernet to attract the “world class” companies to the area. It was this point that I realised people had been drinking way too much of the nbn Kool Aid.

nbn had created a new “business nbn” team who job it was to push Enterprise Ethernet and other “business grade” products. Their first job was to label the existing nbn network a “residential grade” network. There was some truth to this based on the technologies that had widely been deployed. However somehow FTTP, which is most definitely a business grade product once extra Service Level Agreements (SLAs) are added, was tarred with the same brush. Launtel had for years been selling business products over nbn’s FTTP technology, including the first in the country to offer 1000/400 (1000Mb down, 400Mb up) back in May 2017. At one point we were heavily criticised by a member of the business team for doing this, I’m guessing because they saw it as a threat to their model.

So what is the difference between FTTP and Enterprise Ethernet? For most businesses, frankly not a lot. There is however a massive cost difference. A typical Enterprise Ethernet product will be 5 to 10 times the price of the equivalent regular FTTP product (aka NEBS) and you will be required to sign a multi-year contract to get it.  

OK so what if you don’t have FTTP, but one of the other poorer technologies: FTTN, FTTB, FTTC & HFC. These technologies are not only slower, but more importantly for businesses, much less reliable. Modern businesses need the internet as much as they need power, any failures cost them dearly as staff sit around unable to do their work. Those businesses have two choices: 1) sign up to a multi year Enterprise Ethernet deal from one of nbn’s retailers (of which Launtel are one) or 2) organise and pay for a technology upgrade to FTTP.

The second option, the technology upgrade, has traditionally been a very long winded and sometimes expensive exercise. nbn have made a few attempts at making this easier include a program where at selected addresses they would offer fixed price upgrades. However more recently nbn started offering free online quotes for anyone, put your address in and it would email you back a price. Unsurprisingly the website became overloaded and they shut it down again, but we expect that to be back online by the end of January. 

The advantage of the tech upgrade process is that it is a single upfront price and then that’s it. Enterprise Ethernet on the other hand is significantly higher prices not only for the entire contract, but also once the contract is renewed. You are also stuck with that retailer for the duration of the contract, regardless of the level of service they provide. However given that one off cost upfront can be a significant drain on a businesses cash flow we are currently investigating whether we can finance the technology upgrades. What surprises me is that nbn don’t do this – most likely because it would compete with their much more profitable Enterprise Ethernet.

So if everyone who needs it had FTTP, would there actually be a market for Enterprise Ethernet? Yes but it would be a very small – very specialised businesses who need the very low jitter, very low contention properties of Enterprise Ethernet, such as an options trader. If you are an accounting firm, a legal firm, a manufacturing business or even a designer or a school, you almost certainly don’t need it. All the modern applications we use today, including video conferencing is designed to run over pretty average connections. The reason is obvious, if they needed an expensive connection to work they would rarely be used (think how often on one of your Microsoft Teams conferences, there is someone working at home).  

Enterprise Ethernet is a very attractive proposition for both nbn and the retailers. The margins are generally very high, and customers are signed up to multi year contracts. It is easy to sell the “sizzle” of an “Enterprise” connection to a medium or large company who is used to paying these sorts of figures for their existing connections from the likes of Telstra, Optus etc – “Now you don’t want that residential grade FTTP do you, Mr Customer?”. Unfortunately these high margins have attracted some of the less ethical sales people that typically live in the telco world. Those who have no real idea of what the product is and why it is useful and more importantly when it is overkill.

What I find particularly frustrating is that nbn have created a completely new product, with completely different service qualification systems, provisioning systems, and different billing. They need not have done this, they could have easily created the product out of the existing NEBS FTTP product. However yet again, nbn have created yet more complexity for the RSPs to deal with.  

So if you are approached by a retailer offering you Enterprise Ethernet, ask yourself carefully if you really need this and whether a simple technology upgrade coupled with good quality retailer who provisions lots of bandwidth (during the work day, when most networks are less busy anyway) wouldn’t do the job just as well. If you already have FTTP, you almost certainly don’t need it. 

Damian Ivereigh
Launtel

In the past seven days we have seen two far reaching and unexpected announcements from NBN that have set in motion the next several years of upgrades to nbn V1.0 that was completed earlier this year.

As you may know, the original plan for the nbn was a universal FTTP model for the city and suburb areas of Australia with Fixed Wireless and Satellite covering the regional and remote communities.

With the election of the Tony Abbott Liberal Governments in 2013, this plan was altered to what is known as a Multi Technology Mix (MTM) network where other technologies, FTTN, HFC and later FTTC, were utilised to speed up and reduce the cost the rollout of the nbn by utilising some of the older copper infrastructure that had been deployed previously. It is arguable whether they actually saw a speedup or reduced their costs, but we will now never know for sure.

Launceston was lucky to be in the position where most of the deployment for FTTP had already commenced and we now enjoy 100% FTTP connectivity across the city and suburbs (the largest pure fibre footprint in the country) whereas other commercial areas in the state (notably Burnie, Devonport, wide areas of Hobart and other commercial zones such as the airport precincts) ended up with the FTTN network, a network that suffers from variable speed, poor reliability and quality and a hard limit of 100Mbps.

The twin announcements from nbn last week covered two main points;

• Enterprise Ethernet: That specially designated ‘Business Zones’ around the country would be earmarked to receive a lower priced premium business grade service of the nbn’s Enterprise Ethernet network if they choose. I have previously written about Enterprise Ethernet and debated the value that this service actually provides over a well provisioned regular nbn service on FTTP. However there is no doubt it is worthwhile improvement over the other technologies. These zones will see reduced pricing for new orders placed in the next 2 years and will be a quick option for businesses requiring more out of their connection than their existing FTTN/HFC/FTTC infrastructure can deliver. The process is to essentially pick a symmetrical speed tier and sign a 1 to 3 year contract. Nbn will then deliver dedicated fibre to your building within the following 3 month period.
  
  
• NEBS (aka “regular” NBN): The announcement about the upgrade of approximately half of the existing FTTN footprint to FTTP ‘enabled’ areas. This program will have a longer deployment timeframe and will involve fibre being run down the street into the existing FTTN areas of both business and residential areas. Any client that then wishes to order a speed tier from their RSP that their current technology cannot deliver will activate a works order for the nbn to visit the site and install the last mile of fibre directly into the premises. The exact mechanics of this ordering and delivery system are still to be announced.

Impact for our region

As I have mentioned in many previous blog posts, the future for any industrially advanced economy such as Australia is a strong and widespread fibre network. It’s the only technology that can deliver the large (and growing at 40% pa) amounts of traffic in an efficient, predictable and reliable manner. Since the completion of the Launceston portion of the nbn, all on FTTP, and activation of gigabit speeds across the city in 2017, Launtel have seen many local and relocated businesses from the mainland take advantage of the new possibilities and grow their businesses, without a limited internet connection getting in the way. From this point of view, Launceston doesn’t have much to gain from these recent announcements given the existing FTTP.

If we zoom out a bit though, these announcements will have a dramatic effect on the people and businesses in all the surrounding areas such as Burnie & Devonport, regional industrial hubs such as the Launceston Airport & Legana and all the surrounding small towns, not to forget the suburban areas where many small businesses start from the proverbial garage.

 

These areas are now on the cusp of having nbn services that are on a par to the services in Launceston and therefore being able to grow their businesses both locally and internationally just as we have seen businesses do in Launceston.

This will be a boon to all of these areas and will in turn have a positive economic effect on the whole region including Launceston. 

 

As an aside, there was also a further announcement made by nbn. This involved the provision of an additional $300M for regional co investment opportunities between local and state government and nbn. Launtel has recently been advocating for the State Government to step in and provide an upgrade path for the commercial areas in Tasmania that missed out on the FTTP rollout in nbn V1.0. This would have been a $50M investment in the digital infrastructure of the State and would have seen most business addresses in the state being upgraded to FTTP. The intention is to inject a post COVID stimulus and create an ongoing commercial return in more employment in digital businesses. Although the announcements made by nbn cover off a lot of these ideas, there will still be work to do. There are vast swathes of Tasmania that are covered by oversubscribed Fixed Wireless towers with no fibre backhaul and there will still be commercial and suburban areas that miss out on FTTP. 

A $50M co investment in the nbn by the state government is still a great idea, would expedite last week’s announcements and would enable a wider footprint to be covered.

While it is tempting to dwell on why we took this circuitous route and why we didn’t just do a full FTTP rollout from the beginning, at some point we have to put all this behind us and just acknowledge that we got there in the end. 

Damian Ivereigh
Launtel
28 Sept 2020

We very much welcome the announcement by NBN that they are reducing the monthly costs and mostly removing the up front fees associated with Enterprise Ethernet (EE). We expect that we will be able to offer this service in many more places and to smaller businesses than we do currently. More than likely this is NBN reacting to price reductions by other big fibre providers who they are in direct competition with and also making life harder for the fibre overbuilders. NBN are a business (though government owned) and they have a responsibility to maximise their revenue opportunity.

However we are very disappointed that this has not been accompanied by an equivalent investment in the regular NBN infrastructure (aka “NEBS”). The most obvious one being starting the upgrade to FTTP. NBN originally had a scheme called “NBN Select” where they were offering a fixed price (and relatively cheap – around $3600) upgrade to FTTP in various business areas. They were upgrading the in-street infrastructure and then inviting people to upgrade their individual premises. Once upgraded they are free to use whichever provider they like. This scheme seems to have mostly fallen away as the NBN Business team appears to now regard NEBS with a certain amount of disdain, claiming it is a low grade, residential, product.

While there is no question that Enterprise Ethernet is a good product, it is still expensive and comes with long contracts. It’s value is really only apparent when compared to non FTTP (HFC, FTTN, FTTC etc) technologies. While there are some use cases for EE where FTTP is already available, they are extremely rare in our experience. Most businesses only buy EE because the technology they have been given is not up to scratch to running a business. This is around performance, but much more importantly around reliability and certainty – the reliability and certainty that only fibre can provide.  

In some ways the fact the EE even exists is a testament to how poor the decision was to move away from the original FTTP rollout back in 2014. The original “business grade” product was Traffic Class 2, that ran over FTTP. Sales of this product were fairly poor because people realised that the standard, and much cheaper, Traffic Class 4 FTTP product was good enough for all but the large businesses. Most business applications (including conferencing systems: Teams, Webex, Zoom etc) do not need high quality, low jitter connections that EE or Traffic Class 2 provide – if they did no-one would use them. 

We have built our business on no contracts – because we prefer to keep our clients through good service rather than locking them in. EE goes against this idea due to the need to take on a long term contract from NBN. This reduces competition overall as customers are locked in to their provider for up to 3 years. There is no opportunity to change provider during that time. 

In many way NBN’s EE is a product of the classical old style telco thinking. We believe we are part of the new style, flexible telco model. A model that allows people to change speeds when they need to, change provider when they need to with a minimum of fuss (within 20 minutes). This is the same model being made available by cloud companies where you can buy computing power by the hour – think Amazon Web Services (AWS). 

If there is one thing Covid-19 has taught us is that it is not enough to have reliable internet in the office, we also need it at home as part of our desire to do more working from home than ever before.

We look forward to NBN making equivalent announcements in the much more flexible NEBS space. These are the investments that are needed to start us down the road of the inevitable FTTP upgrade for Australia.

Damian Ivereigh
Launtel
22nd Sept 2020

Today, for about 4 hours, we saw a major disruption to our voice (telephone call) network. The problem was in the systems of our upstream provider, Vocus. While we don’t have any detailed information of what failed at Vocus, it is already clear that there is a fundamental design flaw in the telephone network of Australia.

During the outage we were able to send outbound calls via an alternate carrier, such that our clients were able to make phone calls without interruption. However we were unable to do this for inbound calls. Unfortunately inbound calls are the calls that most businesses rely on for sales and so this was particularly disruptive. Why were we unable to use an alternate carrier? 

The problem comes down to the mess that our major carriers are in around managing numbers. The basic system is that each carrier announces ahead of time to all the other carriers what numbers it will accept calls for. The process for moving a number between one carrier and another is a ridiculously arduous process called “number porting”, something that often takes days or weeks. So whilst we can send an outbound call via any carrier we like (and just say it comes from xyz number), we have to allocate that same to number to a particular carrier for inbound calls beforehand during the number port process.

You would think there would be some sophisticated protocol (like “SS7”) to do this, however my understanding is that it basically comes down to ad hoc spreadsheets and the file transfer of these between the carriers.

If one of the carriers has a failure there is no method or process for us to quickly tell an alternate carrier to tell all the other carriers to send calls to the alternate carrier instead (and then onto us).

There are only a small number of large carriers that are part of this club that are allowed to send and receive calls to each other. While there are plenty of smaller players, they all have to connect via one of the club members. As you can imagine competition and innovation are rare commodities in this environment.

A good comparison to the issue can be made in the data world with a protocol called BGP. While BGP has its issues, one of the great things about it is that all players (even the smaller ones like us) are allowed to “announce” our addresses (the equivalent of telephone numbers) to the world. We are also allowed to announce via multiple routes, so if one upstream carrier fails another can quickly (in a minute or two) take over, resulting in minimal disruption. Could this technically be done for phone calls, indeed it could – most modern signalling protocols (e.g. SIP that is used by your VoIP phone) can easily handle this.

So unfortunately there really was nothing that we, Launtel, could do about the Vocus outage today. We could choose a different carrier, but in reality all of them will have outages and this is the first system outage they have had in a few years. All providers have to do the same as we do and get their calls through a large provider who is a member of the club.

The one thing we cannot do and the only way to get a fully reliable system, is to use multiple carriers for inbound calls.

Damian Ivereigh
Launtel
15 June 2020