Unfortunately due to the generally poor state of Australia’s internet infrastructure (don’t get me started), testing your internet speed has become a bit of national obsession. We buy plans based on the speed of the connection and get grumpy if our provider doesn’t supply those speeds.

When you think about it this is a pretty crazy state of affairs and is really an indication that what we have probably isn’t enough. Compare that to Europe where most people have no idea how fast their connection is. “It’s fast enough for me”, said a friend of mine in Poland on what turned out to be a 400Mb connection after I suggested he run up a speedtest.

Measuring the speed of your connection is something akin to wanting to know what the amperage rating of the power cable coming into your house. We don’t know because it’s enough for a house and unless we are planning to build a factory there, we don’t and shouldn’t care about it.

Unfortunately to test your speed isn’t quite as simple as spinning up https://speedtest.net, connecting to whatever server it reckons is your closest and hitting GO. It will give you a measure, but is it useful? Will it tell you how fast you can download a file or whether it is enough to watch 4K Netflix? Unfortunately, it probably won’t.

To understand why, we need to delve a little into the way data is moved across the internet and in particular over the most widely used protocol called “TCP”. Moving data around is not like sending water down a pipe, it is more akin to sending water down a trench. With water if you put too much water at the beginning of your trench, it will overflow and will not make it to the other end. Assuming we have plenty of water available, we probably won’t care about this as long as enough water makes it. One bit of water is the same as the next bit.

However data is quite different, it all has to eventually get there – you don’t want holes in your downloaded file. So anything that overflows has to be detected and resent by the sender. This re-sending process is a waste of capacity and effectively reduces the overall speed of downloading, so should be minimised. So the TCP algorithm incorporates in it a system for pacing it’s sending rate until it just starts to see lost data as an indication that it is going too fast and is overrunning part of the link. It called an congestion avoidance algorithm. It gets this information from the receiver who is constantly sending back information about what packets it has received and therefore what is missing. The key thing to understand is that any lost data creates a signal (via the receiver) to the sender to slow down its sending rate.  

On a perfect data transmission channel no data will get lost until it exceeds the data rate of the channel, at which point data packets will be lost and the sender will peg back its sending rate to match the capacity of the channel. Given that most transmissions are over multiple channels with different data rates, the system quite correctly slows itself to match the speed of the slowest (least capacity) segment.

However what happens if part of the link isn’t working perfectly well and starts losing occasional random packets? The algorithm will mistakenly believe it needs to slow down to try an avoid this packet loss. You run up a speed test and your speed is much slower than it should be. What is the most likely cause of this packet loss? Your wifi! Wifi is a radio system, it is subject to interference and particularly in apartment building, congestion – too many people trying to use the available radio capacity at once. 2.4GHz wifi has just 3 channels, 5GHz has many more, around 30, but because of the higher frequency will not go so far. Once the signal gets too faint you have more packet loss.

Unfortunately this effect of random low level packet loss is not constant. Because of a side effect of the way the TCP algorithm works, latency, the time between a bit of data being sent and the signal back from the receiver, has a massive effect. A lower latency (closer) sender can cope with much higher packet loss than a higher latency one. This is due to delays in the sender algorithm getting updates on what is going on, it tends to overshoot or undershoot the correct sending rate more often.  

The net result is that if you run a speedtest to a server just 5ms away over  your dodgy wifi you will still get a high speed reading. Try the same test to one 30ms away and your reading is much lower. So which is correct? The answer depends on what you are using your connection for. If you are in Perth and most of your traffic comes from Sydney (because in Australia almost everything is hosted in Sydney), then getting a speedtest reading to a server in Perth isn’t very useful, because that is the not speed that you will be seeing in real life.

At Launtel we have recently taken on services via Vocus. The downside of this arrangement is that, at least in the short term, all the traffic is “backhauled” to Sydney before being handed to us. So the closest place we can put a speedtest server to you is in Sydney. Given that most people’s traffic is going to go to Sydney anyway, this is not an issue, but it will affect the speedtest results when comparing to another RSP who has a speedtest server much closer to you. However I would argue this isn’t real. What is worse is sometimes people test against a local server from our network and then the traffic has to make the trip to Sydney and then back to the local server, increasing the latency even more.

However at this point I need to remind you that it is not the latency that is the problem, it is the low level packet loss plus the latency that causes this discrepancy in speed between a local test server versus a remote one. So if you are seeing this, the first step has to be to remove the most likely cause: your wifi. Run the test using a cabled in PC (sorry mac people, I know you guys can’t stand cables, but you will have to get an ethernet adapter!). If that is still showing a difference then there is an issue in your NBN connection or our network. Remember if you seeing a fast reading from a local speedtest server, it is likely it masking your problem. Unless you can understand the result you should always run a speedtest to Sydney on any RSPs network, because that is probably the one you care about. 

In the meantime we will continue to roll out our network and put speedtest servers close to people, even though it is technically pointless, just because it is good for marketing and avoids me having to explain the above. 🙂

Damian Ivereigh
Launtel
March 9, 2020

There has been some controversy about an announcement by Seven Networks to broadcast the Japan Olympics in 4k over Optus’ 5G network. The controversy being that they will only be offering 4k through Optus.

Optus is clearly using this as a way to drive customers onto their new 5G network and have at least taken an implicit swipe at NBN and its lack of performance.

Many people are upset that they will be unable to access this streaming due to the lack of 5G coverage or that they don’t want to buy the appropriate equipment to allow them to receive this.

So was this a good decision by Seven? I believe it is, however the reasoning will surprise you. It does not mean that 5G is some great network that will beat NBN. The reason goes to the heart of a growing problem in the internet live streaming industry that unless solved is headed for a train wreck.

It is worth remembering that Optus has already had a big failure when they sub licensed some of the Soccer World Cup games from SBS and attempted to broadcast them over their mobile network. It was a massive failure, even triggering the then PM, Malcolm Turnbull, to put a call into the CEO of Optus.

So why did it fail? Optus completely underestimated the demand, indeed given the scale of the failure involved I actually wonder if anyone in management & marketing even consulted the IT and Networking engineers to ask about capacity.

With any streaming you always have to budget to send a separate copy of the video to every single person who asks for it. If a lot of people are asking for it, this traffic quickly multiplies up. So if a video streaming in HD (using about 5Mb) is being requested by 100,000 people that amounts to 500Gb of traffic. Now with appropriate banks of servers in datacenters that is quite doable. When sending pre-recorded video (youtube, Netflix etc) you can also play some tricks like send the data in bursts and allow the receiver to store (“buffer”) many seconds of video ahead that can make up for any slowdowns or congestion (i.e. render them invisible).

However live streaming creates a number of unique challenges. Firstly depending on your definition of “live” there really can be no prebuffering – otherwise you would be delaying the video (no longer “live”). This means that the servers must generate a very smooth stream of data at 500Gb, any congestion or delays and the video stops or breaks up, leading to a very unpleasant watching experience. Secondly players will get out of sync (maybe the user paused it briefly) with each other so that each stream is being sent slightly different data. That position in the stream has to be tracked for each individual user, more computation and data fetching power required in the servers.

I don’t know the number of subscribers to the Optus Soccer, but clearly either the servers being used or the network delivering the data could not keep up.

This is all bad enough in HD, but if you bump that up to 4k (which uses 25Mb per stream) and now instead of 500Gb of traffic, you now have 2.5Tb of traffic to support those 100,000 users.

So it is highly possible that Seven were actually staring down the barrel of not offering 4k at all: they just could not justify the expense of the server hardware required (even when rented – which is what content companies such as Akamai do) to supply the 4k streams for the two weeks of the Olympics. It is also extremely hard to test these things – it’s not like you can corral 100,000 users in for a system test!

So if Optus came to them offering to do it over 5G there are several distinct advantages for Seven: Firstly it would mean they could say they are actually streaming in 4k. Secondly it will be to a limited audience (due to the lack of 5G coverage), thus limiting any issues with capacity. Thirdly and this is most important: they will likely use a technology called “Multicast”, which is possible over 5G.

Multicast

So what is Multicast? As I explained above the problem with internet streaming is that it requires the data to be individually sent to each user who wants to watch the stream, every user sets up a new session all the way back to the server. This is called a “Unicast”.

If you compare this with the traditional (“over the airwaves”) broadcast TV, the transmitter just sends a single copy (stream of data) everywhere and each receiver just passively receives the signal. There is no extra load put on the transmitter just because someone new decides to start watching.

In attempting to duplicate this behaviour over the internet, some years ago the internet engineers came up with a concept called Multicast. To understand consider that the internet is built a bit like a tree, with the transmitting server at the base of the trunk and the users at the leaves. What Multicast does is allow users to register with the immediate branching point below it (a router) that it wants to receive the streaming traffic. If that stream is not in the routers list of streams it is “subscribed” to it will send a message (“subscribe me”) to it’s next router below that it wants a copy and so on down to the base.

The transmitter then just sends a single copy of the stream up the trunk to the first router which sends a copy up each branch to the next router until it reaches the user at the leaf. The key difference is that it scales well – each router only has to send a copy of the stream to the routers immediately above it, probably a few tens or hundreds of copies at most.

In the 5G context the final branching point will be the tower which will send out a single copy of the stream for all the users to receive.

The key point is that there is no extra bandwidth or server load required as each new user starts watching.

There only one downside: each user will be playing exactly the same thing, but for live streaming this is exactly what is needed. Any pausing or rewinding of the stream must be done by the player storing the data locally.

NBN and the bigger picture

When NBN was first designed they did incorporate Multicast into the FTTP product, however it was not done well. It was designed for companies (ironically they probably had Foxtel in mind) to have their own separate connection and to always use that connection to provide a multicast service. Of course there was no takeup since that is not how people buy streaming services these days. Multicast was then completely left off the other NBN technologies (FTTN, HFC etc).

Multicast does not work well for the video on demand type streaming (e.g. Netflix), because by making it “on demand”, you can’t join multiple streams into a single one – different people will be watching different things or watching different points in the stream.

However we are left with a problem, without multicast, live streaming will always be problematic over the NBN, not because the NBN isn’t fast enough (the usual criticism), but because unicasting a live stream at any scale will just put too much load on everything, not only the network but also the streaming servers.

It should be noted that 5G isn’t the magic answer here, it is only going to work in this particular instance, where a single company is essentially controlling the content creation, network delivery and the player hardware.

I fully expect that come the Olympics in 2020, we will see significant issues with live streaming of popular events. If we don’t then, then it will be at the next major live streamed event.

Damian Ivereigh
Launtel
10 Dec 2019

Summary: By manipulating who ends up own the infrastructure upgrade we can create a model that significantly incentivises people to perform a TCP upgrade earlier rather than later. Thus accelerating Australia’s eventual upgrade to FTTP.

It is fairly well known that the take-up of NBN’s Technology Choice Program has not been that great. When Malcolm Turnbull proposed his Multi Technology Mix model he said that those who wanted full fibre could upgrade for “a few thousand dollars”. It hasn’t turned out that way with most upgrades costing over $10,000 and many significantly higher than that.

So why is it so much more expensive? The reason is that NBN have used a “cost recovery” model, where anyone who asks for an upgrade has to pay for all the upgrade costs to the entire infrastructure needed to deliver FTTP to that person’s premises. In the early days, that even included purchasing an expensive OLT device – about $100,000 – that sits in the hub (often the old exchange). This OLT is capable of servicing hundreds of subscribers. We had an upgrade quote come back to us of over $120,000 because of this. They have since rolled out OLTs so thankfully they are no longer charging for this.

Assuming that this upgrade proceeded, the next person, maybe a neighbour, to upgrade would then be able to pay nothing for that infrastructure and just pay the extra bits that was needed – a significantly reduced amount. It is understandable why NBN did this, they want to make sure that any FTTP upgrades were at worst cost neutral to their budget.

This creates a significant disincentive to go first, a “first mover disadvantage”, which mean that few people want to go ahead unless they had a very high motivation. Unfortunately due to NBN’s high pricing of the higher speed tiers combined with a quirk in the CVC model, few RSPs actually offer speeds above 100Mbps (with the exception of us in Tasmania). So the speed motivation is not there, there are others like reliability and latency, but this often isn’t enough.

NBN in their latest pricing review is proposing some changes that will make offering the higher speed tiers (above 100Mbps) more financially viable for RSPs – in particular a cheaper 1000/50 speed tier. So this particular problem may be about to be fixed.

However what about the first mover disadvantage problem? Could we turn the problem on it’s head and actually create a first mover advantage and still not put any pressure on NBN’s finances?

Creating a first mover advantage

Suppose after an upgrade, the purchaser gets a certificate that details the upgrade performed and the costs of every segment of that upgrade. That person now “owns” all the parts of the upgrade. If another person comes along asking for an upgrade, that purchaser must buy all those shared parts from the previous owner – mediated by NBN. The new upgrader now owns those parts that were used in the new upgrade and is given a new certificate.

This would not require a significant amount of work, because NBN already keeps careful records of the parts of an upgrade and indeed many of the costs are standardised with their contractors.

The net result is that while the first mover still has to initially pay for all the upgrade costs, the quicker they do it, the more likely they are to be able to get a substantial portion of the upgrade costs back from other upgraders. Indeed the last person to upgrade would get landed with all the costs of the shared infrastructure. Of course there is nothing to stop the first upgrader encouraging the future upgraders by giving a discount on those costs. It would after all be a free market.

You could even see RSPs or other bodies financing these upgrades in return for receipt of the certificates. Banks could loan against them or add them to the mortgage. Consortiums could be formed that would pay for and receive all the certificates.

There are some variations on this that would allow each subsequent upgrader to pay a share of the costs rather than the full costs, but that would change it to being a neutral advantage. By making the default be “all the costs”, any variations can be done by individual negotiation. The important change that NBN have to agree with is to not allow subsequent upgraders a free ride on the earlier upgraders. The later upgraders must buy those certificates first.

Another minor variation is that NBN pays a lease payment for that infrastructure to whoever owns it (has the certificate). They are after all getting extra revenue from the higher speed tiers. While this may be a lot of process for a relatively small amount of money, it may be enough to partly defray the interest costs of the owner and further encourage takeup.

At any stage NBN could decide for operational reasons that they want to actually own the infrastructure (rather than paying lease payments), in which case they have the right to buy back those certificates at face value.

I believe this should be an attractive proposal for NBN. It ticks the boxes of not costing NBN any more money, turbo charging the inevitable FTTP upgrade that Australia needs and of course is essentially a free market approach that fits with the neo-liberal agenda that both sides of government seem to identify with these days. 

Let me know what you think.

Damian Ivereigh
Launtel

It is with a heavy heart that we’ve decided that Launtel will not be sponsoring the free wifi at the next Festivale event in Jan/Feb 2020.

We’ve provided Festivale with free wifi for the last three years and we’ve very much enjoyed growing and improving with Festivale. The 2019 event was extremely successful to the point that on the Saturday night, the gates had to be closed – declaring for the first time ever that the event was full.

While Festivale draws people from all over Australia and even the world, it’s still very much a locally organised event, enjoying a lot of support and sponsorship by many local Launceston businesses including ourselves.

One of the things we love about Launceston is its community. People are not anonymous here, most of the people in the business community know each other and have worked with each other for many years, developing and deepening their relationships. One of things you cannot do in Launceston is burn relationships and expect there to be no consequences.

It was thus with great surprise that we learned that two of our friends and customers, both long standing marketing supporters and sponsors of Festivale, Effective Naturally and Walker Designs, were told by the Festivale Committee that they had a week to prepare a substantial Expression of Interest (EOI) document to provide marketing services for the next 2020 event. Despite their past relationship (and apparently successful performance), they were asked to start a tender process from scratch with very little notice. It’s unclear to us why it was done this way, with no prior warning. Both companies declined due to this short notice and thus will not be providing their services for the 2020 event.

Frankly this is not the sort of behaviour we expect to see from the Festivale Committee. It may be normal, corporate style, cut-throat behaviour in the major cities, but not in Launceston and certainly not in a community event such as Festivale. We understand we are talking about business here, however businesses are filled with real people. People form relationships with people, often long standing ones, not businesses. 

The treatment of our two friends has caused us to question our relationship with Festivale. Our sponsorship of Festivale was because we wanted to contribute to the community in what is clearly a fantastic event. Sure we got exposure and marketing from it, but nothing that we couldn’t get much, much cheaper elsewhere.We had hoped to have a long continued relationship with Festivale, but we now wonder if actually we’re just a source of sponsorship dollars (or services in kind) to the committee – to be discarded when no longer useful to them.

We understand that a new marketing company, Purpose Marketing, whose owner is the partner of the Chairman of the Festivale Committee, has been appointed and so we don’t expect this decision to be reversed. One can’t help but wonder if the new appointment has been somewhat unethical and a conflict of interest. We have decided that we no longer want to be part of an event that treats its supporters and sponsors this way.

Damian Ivereigh
Launtel
13th September 2019

 

Any parent, pet owner or manager will know that the basic tenet of managing anything is that you do your best to encourage good behaviour and discourage bad behaviour. This is the origin of the old “carrot & stick” saying.

Each of us have our own motivations and what we try to do is find ways to encourage behaviour in the people we work with that lines up with our motivations.

NBN’s Goals

So what are NBN’s motivations? There are three main goals set partly by the Government and partly by NBN themselves;

1. Roll out the network as fast as possible while keeping capital expenditure within budget.
2. Ensure the end customers have a good experience (“CX” in NBN parlance), both connecting to and using the network.
3. Raise the average revenue they get from each connection (via their customers, the RSPs) to at least $51 per connection per month (known in the industry as ARPU – Average Revenue Per User). This is to ensure the long term financial viability of the project. 

I’m not going to comment on the first goal, because this has been written about extensively elsewhere, it’s also not something that we as an RSP (Retail Service Provider), have much influence on.

The other two goals are heavily influenced by NBN’s customers, the RSPs. So coming back to my opening statement… wouldn’t you expect that NBN would support and encourage those RSPs that are helping them meet their goals? Unfortunately their actions to date haven’t demonstrated this, indeed they appear to be either doing the opposite or at best, ignoring the behaviour of those RSPs that seem to be aligning with their goals. They are behaving like the teacher who’s time is taken up attempting to deal with a few noisy, recalcitrant pupils who don’t want to be there, rather than helping those pupils that are interested in actually learning something.

The Bundles

The most obvious example is the bundles. In 2017 NBN had a big problem with two of the goals (CX and Revenue). RSPs were allowing their networks to become congested by not buying enough CVC – the capacity that NBN charges RSPs to deliver data to the RSP’s customers. This meant that end customers were complaining about poor performance (“bad CX”), which big RSPs where able to blame-shift across to NBN’s choice of technology. It was also impacting NBN’s overall revenue due to delayed take-up of services and people not interested in buying higher speeds.

So instead of asking “OK which RSPs are providing better CX and more revenue for us (i.e. align with our goals), what are they doing and how can we encourage more of it?” They instead went straight to “How can we force the big RSPs to improve the CX by providing more CVC to their customers?”

NBN released Focus on 50 and then the bundles, which essentially gives everyone a pre-determined amount of CVC bandwidth with each connection. Phil Britt, CEO of Aussie Broadband, was quoted in a presentation he gave to AusNOG: “In November 2017 nbn rewarded the big boys for behaving badly and the landscape changed again.”

The bundles were a short term fix, but they’ve created a bigger problem. While there’s no question it did almost instantly improve CX by removing congestion, the change was only temporary as our bandwidth usage has grown and we’re again seeing congestion return to some carrier’s networks. However, the second problem is that this has made it even harder for NBN to reach its $51 ARPU goal. The reason is that the bundles have homogenised the product offerings of RSPs. NBN have essentially told the RSPs how they should run their networks (how much CVC they should buy) by making it much more expensive to operate outside the expected amount. NBN took away one of the key unique selling points of those RSPs already doing the right thing: they damaged the business models of those RSPs that were generating NBN the most revenue per user.

Why is this a problem? In any market you need to have a range of products catering to different needs around price versus quality/performance. I drive a cheap Hyundai Getz because I am not interested in cars, yet some people will pay 5 times what I paid for my Getz to get a vehicle they enjoy driving or does more than going from A to B. That doesn’t make the Getz a bad car, it is just targeted at a particular market that happens to include me.

Commoditised Market

If a market is homogenised, i.e. all products are much the same, then the only thing to discuss is the price. We call this a commoditised market. This is very prone to happening anyway in the telco market where product differences are at best technical and hard to explain. This has not only lead to a fall in revenue for everyone involved, but has also lead to some atrocious behaviour with some telcos using sales tricks or even downright deception to get or keep customers, rather than product differentiation. The commoditisation makes it very hard for anyone (NBN or RSPs) to raise extra revenue by providing higher performance services, that at least some people will buy. Without the higher paying customers, NBN and the RSPs cannot afford to provide services to their lower paying customers.

Meeting NBNs Goals

For the record, at Launtel we aim to provide a very high level of network performance and customer experience. We do this by simply buying more CVC (“overage”) than most other providers along with an attention to detail and care around the support we provide our customers. We also charge more than most RSPs for an internet service. Thus we don’t cater well for those who are happy to accept a lower performance and want to pay less – we turn away people that want to buy a Getz because we are selling Mercedes. We don’t use tricks to get sales, we just let people try the product and see if it is for them – see if it is worth paying the extra money.

It should come as no surprise that we also provide more revenue to NBN per connection. As of today we pay NBN an ARPU of $55.24. In other words we are among the RSPs who are bringing their revenue average up, helping NBN meet their goals, both in terms of revenue and customer experience. You would expect them to be interested in what we have done, however to date they seem to be more interested in how to stop the big players from holding their customers to ransom by providing poor CX. 

Pricing Review

To be fair, NBN have launched a pricing review, which we have contributed to. Our submission includes a very detailed suggestion on how the CVC pricing could be restructured to encourage the RSPs looking to sell high quality/high performance/high speed/high priced services to sell more of those services. Many commentators would like to see the CVC charge gone, or have argued for the bundles to include more CVC, I don’t believe this is a good approach. It would just further commoditise the market. It would hurt NBN’s goal of raising the revenue it needs to further upgrade the network (e.g. towards the FTTP we know Australia will absolutely need within 20 years) and more to the point, make it harder for NBN to cross-subsidise the services for those who can only afford to pay less.

Don’t get me wrong I’m not looking for special treatment here, I’m just asking NBN to stop closing off a market (the high performance one) that would align very well with two of their important goals.

Damian Ivereigh
Launtel

NBNCo (the company rolling out the nations broadband network) has announced a wholesale pricing review that. As a kick off to this review they also announced that they are going to increase the CVC inclusion on the 100Mbps plans from 2.5Mbps to 3Mbps.

Unfortunately there is perception that this as a performance boost to 100/40 services. It is, but it is nothing like the 20% boost you might think it is.

The way the bundles work is that all the CVC inclusion is put into a pool (the CVC) which is then shared among all of the RSP’s bundled services – including those on other, lower speed, tiers. Given that the number of 100Mbps services is, for most RSPs anyway, a small percentage (around 10%) of the total number of services, the increase per connection is much smaller when averaged over all of the connections.

Let’s put some example numbers into this. If your RSP has 1000 services in the pooled CVC you are connected to. Let us assume that your RSP has the typical spread of speeds, which is, 90% are on 50Mbps and below and 10% are on 100Mbps and above. The former gets 2Mbps of included CVC, the latter currently has 2.5Mbps of CVC, will soon be 3Mbps of CVC.

So currently the total CVC is 900 X 2 + 100 X 2.5 = 2050

After the change the total CVC is 900 X 2 + 100 X 3 = 2100

So the increase is just a little under 2.5%. Not something to get too excited about!

If you are with a good RSP, then they will already be buying extra CVC over and above the amount included in the bundles, called overage. This overage costs $8/Mbps, so the above example RSP will see a reduction in their costs of $400 – 40c per service. Again not something to get too excited about. If your RSP is not buying overage (which may be why you are seeing congestion), then you will see a 2.5% reduction in congestion (whatever that means!) as they increase the CVC size.

To be fair to NBN what they are trying to do is make the 100Mbps plans a little more attractive for RSPs to sell, they are not reducing the cost of the 100Mbps plans rather they are including more for the same price.

My concern is that NBN is currently in a game of chicken with the large RSPs who are essentially holding their users to ransom by constraining their CVC to just the included bundle amount and waiting for the bad press to accumulate blaming NBN for the problem. Since 2017, NBN has put customer experience (or “CX” as they call it) high up their priority list, we will see whether it is above getting revenue!

At Launtel we are already buying overage because we believe that having a high performing, ultra low congestion, network is our key point of difference.  This is the main reason we are more expensive than most of our competitors.

Damian Ivereigh

June 30, 2019