Today, for about 4 hours, we saw a major disruption to our voice (telephone call) network. The problem was in the systems of our upstream provider, Vocus. While we don’t have any detailed information of what failed at Vocus, it is already clear that there is a fundamental design flaw in the telephone network of Australia.

During the outage we were able to send outbound calls via an alternate carrier, such that our clients were able to make phone calls without interruption. However we were unable to do this for inbound calls. Unfortunately inbound calls are the calls that most businesses rely on for sales and so this was particularly disruptive. Why were we unable to use an alternate carrier? 

The problem comes down to the mess that our major carriers are in around managing numbers. The basic system is that each carrier announces ahead of time to all the other carriers what numbers it will accept calls for. The process for moving a number between one carrier and another is a ridiculously arduous process called “number porting”, something that often takes days or weeks. So whilst we can send an outbound call via any carrier we like (and just say it comes from xyz number), we have to allocate that same to number to a particular carrier for inbound calls beforehand during the number port process.

You would think there would be some sophisticated protocol (like “SS7”) to do this, however my understanding is that it basically comes down to ad hoc spreadsheets and the file transfer of these between the carriers.

If one of the carriers has a failure there is no method or process for us to quickly tell an alternate carrier to tell all the other carriers to send calls to the alternate carrier instead (and then onto us).

There are only a small number of large carriers that are part of this club that are allowed to send and receive calls to each other. While there are plenty of smaller players, they all have to connect via one of the club members. As you can imagine competition and innovation are rare commodities in this environment.

A good comparison to the issue can be made in the data world with a protocol called BGP. While BGP has its issues, one of the great things about it is that all players (even the smaller ones like us) are allowed to “announce” our addresses (the equivalent of telephone numbers) to the world. We are also allowed to announce via multiple routes, so if one upstream carrier fails another can quickly (in a minute or two) take over, resulting in minimal disruption. Could this technically be done for phone calls, indeed it could – most modern signalling protocols (e.g. SIP that is used by your VoIP phone) can easily handle this.

So unfortunately there really was nothing that we, Launtel, could do about the Vocus outage today. We could choose a different carrier, but in reality all of them will have outages and this is the first system outage they have had in a few years. All providers have to do the same as we do and get their calls through a large provider who is a member of the club.

The one thing we cannot do and the only way to get a fully reliable system, is to use multiple carriers for inbound calls.

Damian Ivereigh
Launtel
15 June 2020

Launch Announcement

As we focus on being Australia’s most innovative ISP, and grow our number of direct connections across Australia, we’re launching a new connection option called FastAF.

#FastAF gives users the confidence that they’re getting the best download connection speed physically possible.

Launtel were the first to launch Gigabit connections in Australia 3 years ago and soon after implemented a User Portal that offered our customers the ultimate flexibility. 

Fast As Fibre is based on the new NBN 1000/50 connection being released on Friday 29th May. 

Not only are we offering FastAF at $4.60 per day, but the Launtel User Portal also allows you to pause the service (stop daily charges) for when you’re away on holiday, change speeds up and down and of course there’s a 7 day free trial for your first week with Launtel.

Assuming a user decides to stay on FastAF without switching to other options in the month, the price for this “up to Gigabit” speed connection amounts to a market leading $139.91 (inc GST) per calendar month.   

We’ve also invested heavily in our nbn network and backhaul capacity to ensure that you experience the fastest speeds possible without peak time slowdowns. You should expect minimum download speeds of 400mbps up to 950mbps. 

We’ve also banished the use of the word “plan”, and are instead reverting to ‘daily options’ to remind everyone that you can choose the speed you have on a daily basis. 

Over 450,000 Australian homes with FTTP and HFC will have access to #FastAF connections. It isn’t available to customers who are in our Vocus/LoLow connection areas, yet, but we’re constantly switching more areas on to direct Launtel Premium connections. There’s also a symmetrical option in there too for those wanting faster uploads!

Check which type of service and connection is available at your home anywhere in Australia here.

Thanks for reading,

Damo
Founder of Launtel

Damian will be live on our Facebook page on Thursday night at 7 doing an AMA with Tech Man Pat to answer your questions about FastAF.

So I’m sure, like many other Australians, you’ve noticed that we’re doing quite well in the battle against COVID19 and how there’s now talk of moving towards elimination rather than just suppression. If we could achieve elimination this would allow us to substantially lift the current lockdown and resume our lives in a relatively normal way.

Elimination is a bit of misnomer because it wouldn’t really be eliminated, but instead reduced in prevalence to such a degree, that we can reasonably hope to catch the virus every time it takes hold again and starts to spread. This is done through a combination of large scale testing and harsh isolation measures of people infected.

A big part of this process is contact tracing where every time a test comes back positive it’s important to quickly trace all of the people the infected person has been in contact with. This is where “The Trace Together App” has been proposed by the Australian Government as a key (but not the only) tool to do this. In essence the pitch is “give up some of your privacy and you can be released from the lockdown we’re in”.

Various authorities have described the need for this app to be taken up by at least 40% of people for it to be effective. The app is based on a similar one developed by the Singaporean government who’ve so far only obtained about a 20% take up in what’s typically regarded a significantly more compliant population than Australia. 

There’s been some hysteria about exactly how much privacy we’re being asked to give up. Some have described it as tracking all your movements and sending it back to the government, which isn’t the case. All reports so far say that it uses Bluetooth technology to record who you’ve been around not GPS, so it doesn’t know where you’ve been.

The government have a strong need for people to trust that their privacy will be protected and so far they’ve done a spectacularly bad job at doing this. After all the past scares about cyber security including the Census bungle, the Data Retention Laws etc, the Australian Government has some way to go before the Australian people consider it trustworthy around privacy. So my prediction is that this app will be a complete waste of time due to the low take up rate.

So could it be done better? Could it be done in such a way that people wouldn’t have to trust the government with their privacy? The answer is yes. Using a “Protocol” rather than “Platform” approach along with some open source code practices.

The Protocol 

 

The first way the government could gain a massive amount of trust would be to release the software source code to the app it’s developing. Then we can look at their code and see if truly does what it is only intended to do and in particular doesn’t do anything that they aren’t telling us about. We’re clearly not all coding experts, but there are enough experts who we trust that could give us their opinion.

However, why not go a stage further and actually throw away the code from Singapore and make it a home grown open source project here in Australia? It need not be limited to a single one – people could download the app created by the group that they trust most.

The government would set out the “protocol” which would define how these apps are to interact with each other, how the data is stored and how the data is released.

It would be totally possible to avoid the government ever having the information without your permission.

I imagine a scenario like this:-

1. The app is downloaded and a single random number is created – this number is only stored on the phone and is unique to you.
2. The app turns on bluetooth and watches for interactions with other phones.
3. Once an interaction has exceeded a criteria (as defined by the government protocol) the two phones running the app swap the random number generated in step 1. These numbers are stored along with the date/time on the phone only. Note that these are just numbers, nobody knows who they belong to.
4. Everyone’s app also subscribes to a list of electronically published numbers that the health authorities would like to contact.
5. If at some point in the future you test positive to COVID19. At that point you are asked, but not forced, if you would be willing to share either your random number and/or your list of numbers (interactions) – there are subtle privacy differences here but either will work.
6. Your number is then either added to the list of numbers that other people’s apps should check to see if they have had any interaction with, alternatively a number could be broadcast saying “we need to contact this person”.
7. Your app would then offer up a notification to you saying that the health authorities would like to contact you. You can choose to follow this or just ignore it as you see fit. 

Using the above protocol/procedure and open source code to actually perform the function, the aim of the health authorities could be achieved (contact tracing) without the people involved ever revealing their location, or even proximity to anyone else without their express permission, which would only be sought if it was needed for the purposed of COVID19 tracing. 

Damian Ivereigh
Launtel
20 April 2020

Unfortunately due to the generally poor state of Australia’s internet infrastructure (don’t get me started), testing your internet speed has become a bit of national obsession. We buy plans based on the speed of the connection and get grumpy if our provider doesn’t supply those speeds.

When you think about it this is a pretty crazy state of affairs and is really an indication that what we have probably isn’t enough. Compare that to Europe where most people have no idea how fast their connection is. “It’s fast enough for me”, said a friend of mine in Poland on what turned out to be a 400Mb connection after I suggested he run up a speedtest.

Measuring the speed of your connection is something akin to wanting to know what the amperage rating of the power cable coming into your house. We don’t know because it’s enough for a house and unless we are planning to build a factory there, we don’t and shouldn’t care about it.

Unfortunately to test your speed isn’t quite as simple as spinning up https://speedtest.net, connecting to whatever server it reckons is your closest and hitting GO. It will give you a measure, but is it useful? Will it tell you how fast you can download a file or whether it is enough to watch 4K Netflix? Unfortunately, it probably won’t.

To understand why, we need to delve a little into the way data is moved across the internet and in particular over the most widely used protocol called “TCP”. Moving data around is not like sending water down a pipe, it is more akin to sending water down a trench. With water if you put too much water at the beginning of your trench, it will overflow and will not make it to the other end. Assuming we have plenty of water available, we probably won’t care about this as long as enough water makes it. One bit of water is the same as the next bit.

However data is quite different, it all has to eventually get there – you don’t want holes in your downloaded file. So anything that overflows has to be detected and resent by the sender. This re-sending process is a waste of capacity and effectively reduces the overall speed of downloading, so should be minimised. So the TCP algorithm incorporates in it a system for pacing it’s sending rate until it just starts to see lost data as an indication that it is going too fast and is overrunning part of the link. It called an congestion avoidance algorithm. It gets this information from the receiver who is constantly sending back information about what packets it has received and therefore what is missing. The key thing to understand is that any lost data creates a signal (via the receiver) to the sender to slow down its sending rate.  

On a perfect data transmission channel no data will get lost until it exceeds the data rate of the channel, at which point data packets will be lost and the sender will peg back its sending rate to match the capacity of the channel. Given that most transmissions are over multiple channels with different data rates, the system quite correctly slows itself to match the speed of the slowest (least capacity) segment.

However what happens if part of the link isn’t working perfectly well and starts losing occasional random packets? The algorithm will mistakenly believe it needs to slow down to try an avoid this packet loss. You run up a speed test and your speed is much slower than it should be. What is the most likely cause of this packet loss? Your wifi! Wifi is a radio system, it is subject to interference and particularly in apartment building, congestion – too many people trying to use the available radio capacity at once. 2.4GHz wifi has just 3 channels, 5GHz has many more, around 30, but because of the higher frequency will not go so far. Once the signal gets too faint you have more packet loss.

Unfortunately this effect of random low level packet loss is not constant. Because of a side effect of the way the TCP algorithm works, latency, the time between a bit of data being sent and the signal back from the receiver, has a massive effect. A lower latency (closer) sender can cope with much higher packet loss than a higher latency one. This is due to delays in the sender algorithm getting updates on what is going on, it tends to overshoot or undershoot the correct sending rate more often.  

The net result is that if you run a speedtest to a server just 5ms away over  your dodgy wifi you will still get a high speed reading. Try the same test to one 30ms away and your reading is much lower. So which is correct? The answer depends on what you are using your connection for. If you are in Perth and most of your traffic comes from Sydney (because in Australia almost everything is hosted in Sydney), then getting a speedtest reading to a server in Perth isn’t very useful, because that is the not speed that you will be seeing in real life.

At Launtel we have recently taken on services via Vocus. The downside of this arrangement is that, at least in the short term, all the traffic is “backhauled” to Sydney before being handed to us. So the closest place we can put a speedtest server to you is in Sydney. Given that most people’s traffic is going to go to Sydney anyway, this is not an issue, but it will affect the speedtest results when comparing to another RSP who has a speedtest server much closer to you. However I would argue this isn’t real. What is worse is sometimes people test against a local server from our network and then the traffic has to make the trip to Sydney and then back to the local server, increasing the latency even more.

However at this point I need to remind you that it is not the latency that is the problem, it is the low level packet loss plus the latency that causes this discrepancy in speed between a local test server versus a remote one. So if you are seeing this, the first step has to be to remove the most likely cause: your wifi. Run the test using a cabled in PC (sorry mac people, I know you guys can’t stand cables, but you will have to get an ethernet adapter!). If that is still showing a difference then there is an issue in your NBN connection or our network. Remember if you seeing a fast reading from a local speedtest server, it is likely it masking your problem. Unless you can understand the result you should always run a speedtest to Sydney on any RSPs network, because that is probably the one you care about. 

In the meantime we will continue to roll out our network and put speedtest servers close to people, even though it is technically pointless, just because it is good for marketing and avoids me having to explain the above. 🙂

Damian Ivereigh
Launtel
March 9, 2020

There has been some controversy about an announcement by Seven Networks to broadcast the Japan Olympics in 4k over Optus’ 5G network. The controversy being that they will only be offering 4k through Optus.

Optus is clearly using this as a way to drive customers onto their new 5G network and have at least taken an implicit swipe at NBN and its lack of performance.

Many people are upset that they will be unable to access this streaming due to the lack of 5G coverage or that they don’t want to buy the appropriate equipment to allow them to receive this.

So was this a good decision by Seven? I believe it is, however the reasoning will surprise you. It does not mean that 5G is some great network that will beat NBN. The reason goes to the heart of a growing problem in the internet live streaming industry that unless solved is headed for a train wreck.

It is worth remembering that Optus has already had a big failure when they sub licensed some of the Soccer World Cup games from SBS and attempted to broadcast them over their mobile network. It was a massive failure, even triggering the then PM, Malcolm Turnbull, to put a call into the CEO of Optus.

So why did it fail? Optus completely underestimated the demand, indeed given the scale of the failure involved I actually wonder if anyone in management & marketing even consulted the IT and Networking engineers to ask about capacity.

With any streaming you always have to budget to send a separate copy of the video to every single person who asks for it. If a lot of people are asking for it, this traffic quickly multiplies up. So if a video streaming in HD (using about 5Mb) is being requested by 100,000 people that amounts to 500Gb of traffic. Now with appropriate banks of servers in datacenters that is quite doable. When sending pre-recorded video (youtube, Netflix etc) you can also play some tricks like send the data in bursts and allow the receiver to store (“buffer”) many seconds of video ahead that can make up for any slowdowns or congestion (i.e. render them invisible).

However live streaming creates a number of unique challenges. Firstly depending on your definition of “live” there really can be no prebuffering – otherwise you would be delaying the video (no longer “live”). This means that the servers must generate a very smooth stream of data at 500Gb, any congestion or delays and the video stops or breaks up, leading to a very unpleasant watching experience. Secondly players will get out of sync (maybe the user paused it briefly) with each other so that each stream is being sent slightly different data. That position in the stream has to be tracked for each individual user, more computation and data fetching power required in the servers.

I don’t know the number of subscribers to the Optus Soccer, but clearly either the servers being used or the network delivering the data could not keep up.

This is all bad enough in HD, but if you bump that up to 4k (which uses 25Mb per stream) and now instead of 500Gb of traffic, you now have 2.5Tb of traffic to support those 100,000 users.

So it is highly possible that Seven were actually staring down the barrel of not offering 4k at all: they just could not justify the expense of the server hardware required (even when rented – which is what content companies such as Akamai do) to supply the 4k streams for the two weeks of the Olympics. It is also extremely hard to test these things – it’s not like you can corral 100,000 users in for a system test!

So if Optus came to them offering to do it over 5G there are several distinct advantages for Seven: Firstly it would mean they could say they are actually streaming in 4k. Secondly it will be to a limited audience (due to the lack of 5G coverage), thus limiting any issues with capacity. Thirdly and this is most important: they will likely use a technology called “Multicast”, which is possible over 5G.

Multicast

So what is Multicast? As I explained above the problem with internet streaming is that it requires the data to be individually sent to each user who wants to watch the stream, every user sets up a new session all the way back to the server. This is called a “Unicast”.

If you compare this with the traditional (“over the airwaves”) broadcast TV, the transmitter just sends a single copy (stream of data) everywhere and each receiver just passively receives the signal. There is no extra load put on the transmitter just because someone new decides to start watching.

In attempting to duplicate this behaviour over the internet, some years ago the internet engineers came up with a concept called Multicast. To understand consider that the internet is built a bit like a tree, with the transmitting server at the base of the trunk and the users at the leaves. What Multicast does is allow users to register with the immediate branching point below it (a router) that it wants to receive the streaming traffic. If that stream is not in the routers list of streams it is “subscribed” to it will send a message (“subscribe me”) to it’s next router below that it wants a copy and so on down to the base.

The transmitter then just sends a single copy of the stream up the trunk to the first router which sends a copy up each branch to the next router until it reaches the user at the leaf. The key difference is that it scales well – each router only has to send a copy of the stream to the routers immediately above it, probably a few tens or hundreds of copies at most.

In the 5G context the final branching point will be the tower which will send out a single copy of the stream for all the users to receive.

The key point is that there is no extra bandwidth or server load required as each new user starts watching.

There only one downside: each user will be playing exactly the same thing, but for live streaming this is exactly what is needed. Any pausing or rewinding of the stream must be done by the player storing the data locally.

NBN and the bigger picture

When NBN was first designed they did incorporate Multicast into the FTTP product, however it was not done well. It was designed for companies (ironically they probably had Foxtel in mind) to have their own separate connection and to always use that connection to provide a multicast service. Of course there was no takeup since that is not how people buy streaming services these days. Multicast was then completely left off the other NBN technologies (FTTN, HFC etc).

Multicast does not work well for the video on demand type streaming (e.g. Netflix), because by making it “on demand”, you can’t join multiple streams into a single one – different people will be watching different things or watching different points in the stream.

However we are left with a problem, without multicast, live streaming will always be problematic over the NBN, not because the NBN isn’t fast enough (the usual criticism), but because unicasting a live stream at any scale will just put too much load on everything, not only the network but also the streaming servers.

It should be noted that 5G isn’t the magic answer here, it is only going to work in this particular instance, where a single company is essentially controlling the content creation, network delivery and the player hardware.

I fully expect that come the Olympics in 2020, we will see significant issues with live streaming of popular events. If we don’t then, then it will be at the next major live streamed event.

Damian Ivereigh
Launtel
10 Dec 2019

Summary: By manipulating who ends up own the infrastructure upgrade we can create a model that significantly incentivises people to perform a TCP upgrade earlier rather than later. Thus accelerating Australia’s eventual upgrade to FTTP.

It is fairly well known that the take-up of NBN’s Technology Choice Program has not been that great. When Malcolm Turnbull proposed his Multi Technology Mix model he said that those who wanted full fibre could upgrade for “a few thousand dollars”. It hasn’t turned out that way with most upgrades costing over $10,000 and many significantly higher than that.

So why is it so much more expensive? The reason is that NBN have used a “cost recovery” model, where anyone who asks for an upgrade has to pay for all the upgrade costs to the entire infrastructure needed to deliver FTTP to that person’s premises. In the early days, that even included purchasing an expensive OLT device – about $100,000 – that sits in the hub (often the old exchange). This OLT is capable of servicing hundreds of subscribers. We had an upgrade quote come back to us of over $120,000 because of this. They have since rolled out OLTs so thankfully they are no longer charging for this.

Assuming that this upgrade proceeded, the next person, maybe a neighbour, to upgrade would then be able to pay nothing for that infrastructure and just pay the extra bits that was needed – a significantly reduced amount. It is understandable why NBN did this, they want to make sure that any FTTP upgrades were at worst cost neutral to their budget.

This creates a significant disincentive to go first, a “first mover disadvantage”, which mean that few people want to go ahead unless they had a very high motivation. Unfortunately due to NBN’s high pricing of the higher speed tiers combined with a quirk in the CVC model, few RSPs actually offer speeds above 100Mbps (with the exception of us in Tasmania). So the speed motivation is not there, there are others like reliability and latency, but this often isn’t enough.

NBN in their latest pricing review is proposing some changes that will make offering the higher speed tiers (above 100Mbps) more financially viable for RSPs – in particular a cheaper 1000/50 speed tier. So this particular problem may be about to be fixed.

However what about the first mover disadvantage problem? Could we turn the problem on it’s head and actually create a first mover advantage and still not put any pressure on NBN’s finances?

Creating a first mover advantage

Suppose after an upgrade, the purchaser gets a certificate that details the upgrade performed and the costs of every segment of that upgrade. That person now “owns” all the parts of the upgrade. If another person comes along asking for an upgrade, that purchaser must buy all those shared parts from the previous owner – mediated by NBN. The new upgrader now owns those parts that were used in the new upgrade and is given a new certificate.

This would not require a significant amount of work, because NBN already keeps careful records of the parts of an upgrade and indeed many of the costs are standardised with their contractors.

The net result is that while the first mover still has to initially pay for all the upgrade costs, the quicker they do it, the more likely they are to be able to get a substantial portion of the upgrade costs back from other upgraders. Indeed the last person to upgrade would get landed with all the costs of the shared infrastructure. Of course there is nothing to stop the first upgrader encouraging the future upgraders by giving a discount on those costs. It would after all be a free market.

You could even see RSPs or other bodies financing these upgrades in return for receipt of the certificates. Banks could loan against them or add them to the mortgage. Consortiums could be formed that would pay for and receive all the certificates.

There are some variations on this that would allow each subsequent upgrader to pay a share of the costs rather than the full costs, but that would change it to being a neutral advantage. By making the default be “all the costs”, any variations can be done by individual negotiation. The important change that NBN have to agree with is to not allow subsequent upgraders a free ride on the earlier upgraders. The later upgraders must buy those certificates first.

Another minor variation is that NBN pays a lease payment for that infrastructure to whoever owns it (has the certificate). They are after all getting extra revenue from the higher speed tiers. While this may be a lot of process for a relatively small amount of money, it may be enough to partly defray the interest costs of the owner and further encourage takeup.

At any stage NBN could decide for operational reasons that they want to actually own the infrastructure (rather than paying lease payments), in which case they have the right to buy back those certificates at face value.

I believe this should be an attractive proposal for NBN. It ticks the boxes of not costing NBN any more money, turbo charging the inevitable FTTP upgrade that Australia needs and of course is essentially a free market approach that fits with the neo-liberal agenda that both sides of government seem to identify with these days. 

Let me know what you think.

Damian Ivereigh
Launtel