The NBN network has now been installed to over 12 million premises around Australia that cover private homes, apartment complexes, retail shops, offices and factories. Whilst getting a new internet connection via the Launtel user portal is now a routine process of entering your address and picking a daily spend based on the NBN speed tier you are after, getting a connection for an outdoor event such as a music festival, boat show or arts & craft fair can still require a bit of extra work due to the remote nature of these events and that there may not be a NBN connection point in the immediate vicinity to connect to.

In these occasions, the event organisers need to look to adjacent properties and work with the property owners to get an NBN connection active. They then need to access the property so that the required cabling, switches and routers can be installed to get the NBN connection from the NBN connection point back to the event location where it can then be connected to a WiFi system for use by the patrons and stall holders.

Organising these fast broadband connections to remote or difficult locations has been a challenge that Launtel has willingly accepted on a number of occasions.  One event that stands above the rest was providing fast broadband Internet and Wi-Fi infrastructure for Party in the Paddock’s farewell event in February 2020 – https://www.partyinthepaddockfestival.com.au/ – 

 

Launtel partnered with our strategic network infrastructure providers Jettech to organise delivery and routing for a high speed (TC4 1,000/400 Mbps) Connection from a NBN FTTP NTD in Launceston city (LST POI) to a paddock in the middle of nowhere (Watery Plains Rd, White Hills TAS 7258) that even lacked a reliable 4G signal.

 

Route –        Launtel NBN Service -> NBN FTTP NTD -> Jettech P2P Link Invermay – Mt Barrow – Watery Plains Mobile Van receiver -> Launtel 4011 MikroTik Router -> field Ubiquity WiFi Access Points -> Festival Stalls & Patrons

 

Since the Party in the Paddock festival incorporated a campsite for the thousands of festival attendees, along with ticket booths, vehicle inspection station and a sound level monitoring station, we also had to deploy several long life battery packs to provide power to the equipment for the entire four days of the event.

During the four day event operations, we had a peak of over 1,700 simultaneous clients. Launtel also had seven staff members providing around the clock support for the 10,000 festival goers and the hundreds of festival staff and stall holders who relied on the network for their computer and POS systems.

Planning and deployment of a bespoke infrastructure build such as this requires weeks of planning prior to the start of the event, installation over the preceding days in conjunction with the other festival infrastructure and then a tear down in the following days after the festival. 

The Launtel/Jettech links provided a 150mbs symmetrical internet connection via microwave infrastructure from Launceston to the event site at White Hills, Tasmania.

Onsite infrastructure included 30 high capacity Ubiquity access points to provide sufficient Wi-Fi coverage for the event via a dozen POE Switches and over 2km of Cat6 Ethernet Cable. We also have a battery pack deployed at every nerve centre where equipment would converge. Due to the remoteness of the festival, primary power relied on generators so the battery packs were a must.

A total of 815 Gb of data used by vendors and  festival attendees over the 3-day event.

 

NBN networks are generally on hand for festivals and large events within city and metropolitan areas. The difficulty with utilising this NBN connection was the 25 kms of wireless backhaul that was required to link the festival to the NBN.

This content is password protected. To view it please enter your password below:

Password:

First up, this article will be a little more detailed and technical than our usual posts by Damian.

We thought it might be time to detail some of the more nitty gritty changes and upgrades that Launtel is making in the spirit of transparency that Launtel is so famous for.
For those that want the short of it but can’t be bothered with my ramblings:
TL;DR: Our portal is getting faster, bigger, stronger, more lean and more mean!

Launtel is always trying to improve our reliability and speed for public facing applications and websites.
To this end some time ago we decided that running our residential portal which is a Perl Dancer2 application using Apache’s mod_cgi in FCGI mode just wasn’t going to cut it.

For those that aren’t familiar with FCGI (Fast Common Gateway Interface), it is a method for a web server to talk to a server side processing engine and then send the results back to the web server. Essentially all web servers do is send content to the browser of the person viewing the website, they generally aren’t able to execute any arbitrary code on the server. That task is left up to an application server.
This is a little different on the ubiquitous Apache2 web server.
Apache2 is a web server AND also an application server, it can send content to the browser and process server side scripts, in our case Perl scripts.
Apache2 achieves this via plugins called mods, specifically mod_cgi.
Whilst all this sounds great for a small site and makes set up and getting started with applications really easy, it does not scale past a few thousand users at all.

There’s a Starman waiting in the sky…

 

Because we run our portal on Perl Dancer2 it made sense to use a Perl-centric application server so that we could decouple the server side application processing for the content serving web server.
This is where Starman comes in, a Perl based and Perl-centric application server that we can proxy requests to for all server side functionality.
This change to the way our portal systems worked resulted in a visible increase to page response time and also a marked decrease in server load. Apache2 will spawn a new worker process each time it needs to process more requests than one process can handle, as you can imagine this can bring a server to it’s knees.
You may have even noticed a speed increase to page load time and general responsiveness if you are a frequent user of the portal.

So Starman works well and scales nicely, however does Apache2?

The answer is no, past a certain number of concurrent client connections Apache2 is woeful at handling these requests and quickly outstrips the server’s available resources.
The issue stems from the fact that Apache2 is from an era before web applications where even really a thing, the web was simple HTML pages and a bit of Javascript delivered to a client’s browser and then forgotten about. These serving of pages and then getting out of the way is what Apache2 is good at.

How do we progress?

 

Obviously something needs to be done if Launtel’s client base is to grow, our portal will need to be able to handle tens of thousands of requests over the course of a day and sometimes thousands at one time.
The solution is to replace literally every aspect of our web stack and deliver a much more powerful and performant product that will scale for years to come.

It comes in the form of two pieces of software: NGINX and uWSGI.

NGINX is a high performance and lean web server and reverse proxy. That’s all it is, it’s not an application server and it cannot be made into one.
We already run this web server elsewhere but the server that the portal runs on is serving many other internal and external applications and until recently, ripping out Apache2 seemed like a daunting task.

That is until we discovered uWSGI.
WSGI stands for Web Server Gateway Interface and as the name suggests is for a web server like NGINX (Or even Apache2 if you really feel like using it) to talk to an application server. Remember CGI that we spoke about before? Think of WSGI as SUPER CGI.
It is essentially a common set of APIs and models for a WSGI compatible web server/gateway and application server to talk to each other. So any server of any kind that can talk WSGI can talk to any other server that also supports WSGI.

 

Think of WSGI as SUPER CGI.

 

uWSGI is a multi language implementation of WSGI that will allows us to have every application on Athena (Our portal server) to utilise this one common application server. uWSGI will handle the scaling up or down of processes and threads and can also be “hot reloaded” in the case of changes to the code base, resulting in smaller downtime windows from minute long tear down and stand up cycles with Apache2 to just seconds with NGINX+uWSGI.

Hopefully this has been at least somewhat informative and interesting, if we get enough positive feedback about this we may consider doing more technical articles in the future.

Adam Heathcote
Developer Operations
Launtel

A little over 2 years ago nbn started selling a completely new fibre based product called “Enterprise Ethernet”. Without going into all the technical details, this is a fibre based telecommunications product capable of speeds of up to 1Gbps in both direction (aka “symmetrical”). From the engineering standpoint there is no question that this is a good product for business customers who need it.

The key word here is “need”. In my opinion the product is being heavily oversold to companies who really don’t need it. The original nbn was to be almost all fibre – FTTP. Had that FTTP rollout continued I don’t believe that Enterprise Ethernet would be a product today. Unfortunately, Enterprise Ethernet’s main purpose is to make up for the deficiencies in the network that occurred due to nbn’s switch to the MTM (aka using all the existing copper, HFC etc that was already in the ground).  

Alarm bells started ringing for me when nbn announced some “Business Fibre Zones” around the country. These were areas where nbn was offering Enterprise Ethernet with no upfront install costs. An industrial area north of Launceston, called Bell Bay, was not on this list. Various state and local political figures wanted to know why it had been excluded! 

It had probably been left off the list because it was one of the first areas rolled out with FTTP. They already had fibre! Yet the assumption was that they needed to have Enterprise Ethernet to attract the “world class” companies to the area. It was this point that I realised people had been drinking way too much of the nbn Kool Aid.

nbn had created a new “business nbn” team who job it was to push Enterprise Ethernet and other “business grade” products. Their first job was to label the existing nbn network a “residential grade” network. There was some truth to this based on the technologies that had widely been deployed. However somehow FTTP, which is most definitely a business grade product once extra Service Level Agreements (SLAs) are added, was tarred with the same brush. Launtel had for years been selling business products over nbn’s FTTP technology, including the first in the country to offer 1000/400 (1000Mb down, 400Mb up) back in May 2017. At one point we were heavily criticised by a member of the business team for doing this, I’m guessing because they saw it as a threat to their model.

So what is the difference between FTTP and Enterprise Ethernet? For most businesses, frankly not a lot. There is however a massive cost difference. A typical Enterprise Ethernet product will be 5 to 10 times the price of the equivalent regular FTTP product (aka NEBS) and you will be required to sign a multi-year contract to get it.  

OK so what if you don’t have FTTP, but one of the other poorer technologies: FTTN, FTTB, FTTC & HFC. These technologies are not only slower, but more importantly for businesses, much less reliable. Modern businesses need the internet as much as they need power, any failures cost them dearly as staff sit around unable to do their work. Those businesses have two choices: 1) sign up to a multi year Enterprise Ethernet deal from one of nbn’s retailers (of which Launtel are one) or 2) organise and pay for a technology upgrade to FTTP.

The second option, the technology upgrade, has traditionally been a very long winded and sometimes expensive exercise. nbn have made a few attempts at making this easier include a program where at selected addresses they would offer fixed price upgrades. However more recently nbn started offering free online quotes for anyone, put your address in and it would email you back a price. Unsurprisingly the website became overloaded and they shut it down again, but we expect that to be back online by the end of January. 

The advantage of the tech upgrade process is that it is a single upfront price and then that’s it. Enterprise Ethernet on the other hand is significantly higher prices not only for the entire contract, but also once the contract is renewed. You are also stuck with that retailer for the duration of the contract, regardless of the level of service they provide. However given that one off cost upfront can be a significant drain on a businesses cash flow we are currently investigating whether we can finance the technology upgrades. What surprises me is that nbn don’t do this – most likely because it would compete with their much more profitable Enterprise Ethernet.

So if everyone who needs it had FTTP, would there actually be a market for Enterprise Ethernet? Yes but it would be a very small – very specialised businesses who need the very low jitter, very low contention properties of Enterprise Ethernet, such as an options trader. If you are an accounting firm, a legal firm, a manufacturing business or even a designer or a school, you almost certainly don’t need it. All the modern applications we use today, including video conferencing is designed to run over pretty average connections. The reason is obvious, if they needed an expensive connection to work they would rarely be used (think how often on one of your Microsoft Teams conferences, there is someone working at home).  

Enterprise Ethernet is a very attractive proposition for both nbn and the retailers. The margins are generally very high, and customers are signed up to multi year contracts. It is easy to sell the “sizzle” of an “Enterprise” connection to a medium or large company who is used to paying these sorts of figures for their existing connections from the likes of Telstra, Optus etc – “Now you don’t want that residential grade FTTP do you, Mr Customer?”. Unfortunately these high margins have attracted some of the less ethical sales people that typically live in the telco world. Those who have no real idea of what the product is and why it is useful and more importantly when it is overkill.

What I find particularly frustrating is that nbn have created a completely new product, with completely different service qualification systems, provisioning systems, and different billing. They need not have done this, they could have easily created the product out of the existing NEBS FTTP product. However yet again, nbn have created yet more complexity for the RSPs to deal with.  

So if you are approached by a retailer offering you Enterprise Ethernet, ask yourself carefully if you really need this and whether a simple technology upgrade coupled with good quality retailer who provisions lots of bandwidth (during the work day, when most networks are less busy anyway) wouldn’t do the job just as well. If you already have FTTP, you almost certainly don’t need it. 

Damian Ivereigh
Launtel

Those of you who know me, know that I am pretty mild mannered and don’t get excited about a lot, in other words your typical engineer. However over the last few months I have been getting increasingly frustrated with NBN’s HFC product. This morning I was sent this picture…

 

HFC has got to be the worst product NBN have – yes worse than Fixed Wireless – which has recently seen much of their congestion issues fixed. It creates so much work for us an RSP, let alone the poor service delivered to our customers. Here are some of the issues we encounter on a day-to-day basis:

1. The service drops out a lot, for some customers multiple times a day. The NTD lights start flashing and your service is down until it can resync with the HFC Node.
2. The NTD needs to be plugged in and connected before we can provision a service – this requires multiple contacts with the client to double check, get feedback on the lights etc. Even then we are seeing many provisioning attempts fail because apparently the signal levels are too low. This then requires an NBN technician to go out and install a new NTD. This can be on an NTD that had an active service just days previously.
3. They did not bolt the NTD to the wall, like they did with FTTP and Fixed Wireless so it is common for tenants to take the old NTD with them when they leave a property. This NTD is useless to them because it won’t work at another address – so ends up in the trash. We then have to get an NBN technician out to install a new one.
4. There is currently a shortage of NBN NTD’s due to an upstream manufacturing issue (Covid related), so NBN are “pacing out” the appointments in an attempt to manage their dwindling stock. The net result is appointments for a new or a replacement NTD is typically 2 to 3 months out. In fact it has become so bad that on many occasions there are no appointments available at all to be booked. We then have to raise a ticket with NBN to get that fixed (2 to 3 day’s delay).
5. NBN’s systems often show multiple NTD’s listed as being on site, however only one is really there. There is no way to line up which NTD is the real one – they don’t list any identifying feature in the NBN provisioning portal that we can tie up with what the customer can see on the box itself (e.g. the MAC address). So we spend hours trying one NTD, failing and then having to try the other.
6. We often see packet loss on a service that appears to get worse and worse until eventually it stops – we suspect NBN resets something in the node to fix it. Gives a whole new meaning to standard IT trick of rebooting your equipment to fix things.
7. The HFC Nodes appear to have no battery backup, unlike FTTN. So when the power fails the entire area is out. We have seen issues caused by faulty electrics (e.g. an RCD failure) that have taken 24 hours to get an electrician out to fix.
8. The upload speed on HFC is woeful. This is not surprising given that it was designed for cable TV (i.e. broadcast signals moving down to the subscriber). Upload was added as an afterthought, easily gets congested and often struggles to even maintain the 40Mb upload that comes with the 100/40 speed. If you want to know why NBN is pushing the 100/20 speed (“Home Fast”), it is because of HFC.
9. Because it is a broadcast medium – all the subscribers in an area are essentially connected together on a single coax cable – any issues with any one of the subscribers equipment (e.g. a faulty NTD putting noise on the cable) and it affects the entire segment, creating packet loss and dropouts. To make matters worse tracking down that faulty NTD is extremely hard to do remotely, requiring technicians to measure what is going on on the cable.
10. Despite there being two physical ethernet ports on the NTD, NBN will only allow a single service at a time. Now this is not unique to HFC, it is the same with the other MTM technologies. This creates issues because we are forced to do a service transfer (unlike FTTP & FW where we just activate the next port). This is OK, except when our customer has misunderstood the process and there is already a disconnection in process from his previous provider. The transfer fails, the NBN order bombs out and we then have to wait several hours for the port to become free before we can place a new order. All the time, the customer is down.

I cannot believe that NBN are sinking more money (upgrading to DOCSIS 3.1, rolling out new coax – yes new leadins – to subscribers who did not have the original cable TV) into this network. However in some ways they are stuck. The problem with infrastructure is that you can’t change course on a dime – it is extremely expensive to change technology in an area. FTTN now has an upgrade to FTTP plan in the works. There is no equivalent for HFC, so we are likely to be stuck with it for a while. The basic problem is that most people (including politicians and regulators) consider the speed of a service, not it’s reliability or cost of maintenance. For speed HFC actually looks pretty good.

For those people who originally had the Optus cable, which NBN bought and decided was in too poor a state to rescue, so dumped it, you guys dodged a bullet. Most people were put onto FTTC instead. While this is not as good as FTTP, it is way more reliable and predictable than HFC. NBN are now working on increasing the speeds for FTTC. If only Telstra had neglected their HFC network in the same way Optus did.

While NBN’s change to MTM, from the original FTTP rollout for political reasons (why were politicians were allowed to make engineering decisions?), was expected to save NBN money (it actually didn’t), all this has done is to dump a whole bunch of extra costs onto the RSPs and the users who have to deal with the mess. HFC is the standout mistake in MTM, yes worse than FTTN.

I would call on NBN to dump HFC, but that is not going to happen – we are all too deeply dug into the financial hole. My only suggestion is that if you are looking to rent or buy a property, lookup the technology type on our website, if it’s HFC, just move on. You may get lucky with HFC and it will be OK, but do you really want to take that risk?

Damian Ivereigh
Launtel

Those who have known us for a while will know that one of the unique benefits of our service is our daily billing method and the ability to “pause” our service at any time. This allows people who don’t need our internet connection all the time, such as for a holiday house or because they only have very sporadic internet needs (e.g. kids at weekends) to turn our service on for a day at a time. We do this by actually disconnecting the service while it is paused, but saving all the details so that it can be reconnected as it was before when required.

This is really only possible financially if there is no connection charge – a charge levied by the wholesalers every time a connection is established (or the retail service provider is changed). The practice dates back to the old copper days where to establish a connection Telstra would have to send a technician out to physically hook up the wire. In those days it did actually cost the wholesaler to do this. However nowadays with computerised provisioning (which takes just a few minutes), it costs the wholesaler nothing to do this. 

When NBN came onto the scene they decided from the beginning not to charge a connection charge, and based on this lack of charge we were able to develop our current pay as you daily internet product.

However Launtel are now providing services on other wholesale networks such as Opticomm and LBNCo. We were horrified to find that Opticomm were charging $69+GST and LBNCo were charging $99+GST. Since the merger of Opticomm and LBNCo, they are aligning the price at the higher of the two, $99+GST, in April 2021. The net result is that every time a service is “unpaused” we are charged $99+GST. This clearly makes pausing and unpausing completely unviable financially on their networks. Just to re-iterate this is a charge for an action that costs them virtually nothing due to their automated provisioning (which is very good, particularly on Opticomm, most services are provisioned in minutes).

However not only does this mean that we cannot offer our service pausing for the minority of customers on their networks, but it also has significant implications for competition as a whole. Every time a customer changes retail provider this fee also must be paid to the wholesaler (some retailers pass it on others do not). One wholesaler actually promoted this to us retailers as a way to reduce this “churn” – the retailers are more likely to get “sticky” customers, i.e. reduce competition. This feels very much like the old idea of a 12 or 24 month contract. We think this is a bad idea: customers should feel free to try multiple providers until they find one they like. Interestingly even in the old Telstra copper days the connection charge did not apply when changing retail providers.

We offer a no obligation free trial to our customers, while a good percentage go onto take the paid service, adding this connection charge further increases the outlay that we have to make to get a new customer.

There is also a perverse incentive around this connection charge: the wholesaler has an incentive to provide a poor service. Typically if a wholesaler’s network gets congested for an extended period of time due to overcommitted backhaul, customers tend to switch retail provider in the hope of avoiding the issue (which they can’t – all retailers use the same wholesale network). The wholesaler collects a connection charge each time the customer does this. In other words they profit from the overcommited backhaul.

We are surprised that the ACCC allows this charge at all – their stated position is that a wholesaler may not charge any more than an equivalent NBN service without special permission. Did they really not take into account these connection charges?

With the increasing use of 5G mobile broadband plans, wholesalers are coming under increased competition from the mobile network. The mobile network suits many people who move around a lot, which is particularly common in the apartment dwellings that many of the non-nbn wholesalers often specialise in, because they just pick up their router and take it with them. Our quick provisioning and daily pricing often also works in this market – people connecting up only when they need it. One wholesaler boasted to us that they had 80% of a block connected. What about the other 20% we asked, they probably have internet, just not fixed line internet.

It is worth pointing out that these wholesalers are pretty much always monopolies. While the idea of infrastructure competition (multiple wholesale networks offered to an address) sounds good in theory, it very rarely actually happens. You end up with a bunch of monopolies instead. No new provider is going to roll out infrastructure to a building that already has existing infrastructure just to go into competition with incumbent – they are more likely to pick a building without any competition (or only a competitor using a poor technology such as FTTB). So while the developer may have a choice of provider, the end user, probably just a tenant of the building, has no choice and must go with whatever the developer chose.

While I said earlier that NBN do not charge a connection charge, they are proposing to introduce a small one of about $5. This is probably small enough that while we may have to pass it through, will not stop people pausing their service. However, we believe it is a move in the wrong direction.

We believe that connection (and churn) charges are a hangover from the past and now are just being used as a way to raise more revenue by the monopoly wholesale providers while avoiding raising their monthly wholesale prices. It is produces an anti competitive environment where even changing retail providers is harder than it should be.

Damian Ivereigh
Launtel
11/01/21