There has been some controversy about an announcement by Seven Networks to broadcast the Japan Olympics in 4k over Optus’ 5G network. The controversy being that they will only be offering 4k through Optus.

Optus is clearly using this as a way to drive customers onto their new 5G network and have at least taken an implicit swipe at NBN and its lack of performance.

Many people are upset that they will be unable to access this streaming due to the lack of 5G coverage or that they don’t want to buy the appropriate equipment to allow them to receive this.

So was this a good decision by Seven? I believe it is, however the reasoning will surprise you. It does not mean that 5G is some great network that will beat NBN. The reason goes to the heart of a growing problem in the internet live streaming industry that unless solved is headed for a train wreck.

It is worth remembering that Optus has already had a big failure when they sub licensed some of the Soccer World Cup games from SBS and attempted to broadcast them over their mobile network. It was a massive failure, even triggering the then PM, Malcolm Turnbull, to put a call into the CEO of Optus.

So why did it fail? Optus completely underestimated the demand, indeed given the scale of the failure involved I actually wonder if anyone in management & marketing even consulted the IT and Networking engineers to ask about capacity.

With any streaming you always have to budget to send a separate copy of the video to every single person who asks for it. If a lot of people are asking for it, this traffic quickly multiplies up. So if a video streaming in HD (using about 5Mb) is being requested by 100,000 people that amounts to 500Gb of traffic. Now with appropriate banks of servers in datacenters that is quite doable. When sending pre-recorded video (youtube, Netflix etc) you can also play some tricks like send the data in bursts and allow the receiver to store (“buffer”) many seconds of video ahead that can make up for any slowdowns or congestion (i.e. render them invisible).

However live streaming creates a number of unique challenges. Firstly depending on your definition of “live” there really can be no prebuffering – otherwise you would be delaying the video (no longer “live”). This means that the servers must generate a very smooth stream of data at 500Gb, any congestion or delays and the video stops or breaks up, leading to a very unpleasant watching experience. Secondly players will get out of sync (maybe the user paused it briefly) with each other so that each stream is being sent slightly different data. That position in the stream has to be tracked for each individual user, more computation and data fetching power required in the servers.

I don’t know the number of subscribers to the Optus Soccer, but clearly either the servers being used or the network delivering the data could not keep up.

This is all bad enough in HD, but if you bump that up to 4k (which uses 25Mb per stream) and now instead of 500Gb of traffic, you now have 2.5Tb of traffic to support those 100,000 users.

So it is highly possible that Seven were actually staring down the barrel of not offering 4k at all: they just could not justify the expense of the server hardware required (even when rented – which is what content companies such as Akamai do) to supply the 4k streams for the two weeks of the Olympics. It is also extremely hard to test these things – it’s not like you can corral 100,000 users in for a system test!

So if Optus came to them offering to do it over 5G there are several distinct advantages for Seven: Firstly it would mean they could say they are actually streaming in 4k. Secondly it will be to a limited audience (due to the lack of 5G coverage), thus limiting any issues with capacity. Thirdly and this is most important: they will likely use a technology called “Multicast”, which is possible over 5G.

Multicast

So what is Multicast? As I explained above the problem with internet streaming is that it requires the data to be individually sent to each user who wants to watch the stream, every user sets up a new session all the way back to the server. This is called a “Unicast”.

If you compare this with the traditional (“over the airwaves”) broadcast TV, the transmitter just sends a single copy (stream of data) everywhere and each receiver just passively receives the signal. There is no extra load put on the transmitter just because someone new decides to start watching.

In attempting to duplicate this behaviour over the internet, some years ago the internet engineers came up with a concept called Multicast. To understand it if you consider the internet is built a bit like a tree, with the transmitting server at the base of the trunk and the users at the leaves. What Multicast does is allow users to register with the immediate branching point below it (a router) that it wants to receive the streaming traffic. If that stream is not in the routers list it will send a message to it’s next router below that it wants a copy and so on down to the base.

The transmitter then just sends a single copy of the stream up the trunk to the first router which sends a copy up each branch to the next router until it reaches the user at the leaf.

In the 5G context the final branching point will be the tower which will send out a single copy of the stream for all the users to receive.

The key point is that there is no extra bandwidth or server load required as each new user starts watching.

There only downside is that each user will be playing exactly the same thing, but for live streaming this is exactly what is needed. Any pausing or rewinding of the stream must be done by the player storing the data locally.

NBN and the bigger picture

When NBN was first designed they did incorporate Multicast into the FTTP product, however it was not done well. It was designed for companies (ironically they probably had Foxtel in mind) to have their own separate connection and to always use that connection to provide a multicast service. Of course there was no takeup since that is not how people buy streaming services these days. Multicast was then completely left off the other technologies.

Multicast does not work well for the video on demand type streaming (e.g. Netflix), because by making it “on demand”, you can’t join multiple streams into a single one – different people will be watching different things.

However we are left with a problem, without multicast, live streaming will always be problematic over the NBN, not because the NBN isn’t fast enough (the usual criticism), but because unicasting a live stream at any scale will just put too much load on everything, not only the network but also the streaming servers.

It should be noted that 5G isn’t the magic answer here, it is only going to work in this particular instance, where a single company is essentially controlling the content, the delivery and the player hardware.

I fully expect that come the Olympics in 2020, we will see significant issues. If we don’t then, then it will be at the next major live streamed event.

Damian Ivereigh
Launtel
10 Dec 2019

Summary: By manipulating who ends up own the infrastructure upgrade we can create a model that significantly incentivises people to perform a TCP upgrade earlier rather than later. Thus accelerating Australia’s eventual upgrade to FTTP.

It is fairly well known that the take-up of NBN’s Technology Choice Program has not been that great. When Malcolm Turnbull proposed his Multi Technology Mix model he said that those who wanted full fibre could upgrade for “a few thousand dollars”. It hasn’t turned out that way with most upgrades costing over $10,000 and many significantly higher than that.

So why is it so much more expensive? The reason is that NBN have used a “cost recovery” model, where anyone who asks for an upgrade has to pay for all the upgrade costs to the entire infrastructure needed to deliver FTTP to that person’s premises. In the early days, that even included purchasing an expensive OLT device – about $100,000 – that sits in the hub (often the old exchange). This OLT is capable of servicing hundreds of subscribers. We had an upgrade quote come back to us of over $120,000 because of this. They have since rolled out OLTs so thankfully they are no longer charging for this.

Assuming that this upgrade proceeded, the next person, maybe a neighbour, to upgrade would then be able to pay nothing for that infrastructure and just pay the extra bits that was needed – a significantly reduced amount. It is understandable why NBN did this, they want to make sure that any FTTP upgrades were at worst cost neutral to their budget.

This creates a significant disincentive to go first, a “first mover disadvantage”, which mean that few people want to go ahead unless they had a very high motivation. Unfortunately due to NBN’s high pricing of the higher speed tiers combined with a quirk in the CVC model, few RSPs actually offer speeds above 100Mbps (with the exception of us in Tasmania). So the speed motivation is not there, there are others like reliability and latency, but this often isn’t enough.

NBN in their latest pricing review is proposing some changes that will make offering the higher speed tiers (above 100Mbps) more financially viable for RSPs – in particular a cheaper 1000/50 speed tier. So this particular problem may be about to be fixed.

However what about the first mover disadvantage problem? Could we turn the problem on it’s head and actually create a first mover advantage and still not put any pressure on NBN’s finances?

Creating a first mover advantage

Suppose after an upgrade, the purchaser gets a certificate that details the upgrade performed and the costs of every segment of that upgrade. That person now “owns” all the parts of the upgrade. If another person comes along asking for an upgrade, that purchaser must buy all those shared parts from the previous owner – mediated by NBN. The new upgrader now owns those parts that were used in the new upgrade and is given a new certificate.

This would not require a significant amount of work, because NBN already keeps careful records of the parts of an upgrade and indeed many of the costs are standardised with their contractors.

The net result is that while the first mover still has to initially pay for all the upgrade costs, the quicker they do it, the more likely they are to be able to get a substantial portion of the upgrade costs back from other upgraders. Indeed the last person to upgrade would get landed with all the costs of the shared infrastructure. Of course there is nothing to stop the first upgrader encouraging the future upgraders by giving a discount on those costs. It would after all be a free market.

You could even see RSPs or other bodies financing these upgrades in return for receipt of the certificates. Banks could loan against them or add them to the mortgage. Consortiums could be formed that would pay for and receive all the certificates.

There are some variations on this that would allow each subsequent upgrader to pay a share of the costs rather than the full costs, but that would change it to being a neutral advantage. By making the default be “all the costs”, any variations can be done by individual negotiation. The important change that NBN have to agree with is to not allow subsequent upgraders a free ride on the earlier upgraders. The later upgraders must buy those certificates first.

Another minor variation is that NBN pays a lease payment for that infrastructure to whoever owns it (has the certificate). They are after all getting extra revenue from the higher speed tiers. While this may be a lot of process for a relatively small amount of money, it may be enough to partly defray the interest costs of the owner and further encourage takeup.

At any stage NBN could decide for operational reasons that they want to actually own the infrastructure (rather than paying lease payments), in which case they have the right to buy back those certificates at face value.

I believe this should be an attractive proposal for NBN. It ticks the boxes of not costing NBN any more money, turbo charging the inevitable FTTP upgrade that Australia needs and of course is essentially a free market approach that fits with the neo-liberal agenda that both sides of government seem to identify with these days. 

Let me know what you think.

Damian Ivereigh
Launtel

It is with a heavy heart that we’ve decided that Launtel will not be sponsoring the free wifi at the next Festivale event in Jan/Feb 2020.

We’ve provided Festivale with free wifi for the last three years and we’ve very much enjoyed growing and improving with Festivale. The 2019 event was extremely successful to the point that on the Saturday night, the gates had to be closed – declaring for the first time ever that the event was full.

While Festivale draws people from all over Australia and even the world, it’s still very much a locally organised event, enjoying a lot of support and sponsorship by many local Launceston businesses including ourselves.

One of the things we love about Launceston is its community. People are not anonymous here, most of the people in the business community know each other and have worked with each other for many years, developing and deepening their relationships. One of things you cannot do in Launceston is burn relationships and expect there to be no consequences.

It was thus with great surprise that we learned that two of our friends and customers, both long standing marketing supporters and sponsors of Festivale, Effective Naturally and Walker Designs, were told by the Festivale Committee that they had a week to prepare a substantial Expression of Interest (EOI) document to provide marketing services for the next 2020 event. Despite their past relationship (and apparently successful performance), they were asked to start a tender process from scratch with very little notice. It’s unclear to us why it was done this way, with no prior warning. Both companies declined due to this short notice and thus will not be providing their services for the 2020 event.

Frankly this is not the sort of behaviour we expect to see from the Festivale Committee. It may be normal, corporate style, cut-throat behaviour in the major cities, but not in Launceston and certainly not in a community event such as Festivale. We understand we are talking about business here, however businesses are filled with real people. People form relationships with people, often long standing ones, not businesses. 

The treatment of our two friends has caused us to question our relationship with Festivale. Our sponsorship of Festivale was because we wanted to contribute to the community in what is clearly a fantastic event. Sure we got exposure and marketing from it, but nothing that we couldn’t get much, much cheaper elsewhere.We had hoped to have a long continued relationship with Festivale, but we now wonder if actually we’re just a source of sponsorship dollars (or services in kind) to the committee – to be discarded when no longer useful to them.

We understand that a new marketing company, Purpose Marketing, whose owner is the partner of the Chairman of the Festivale Committee, has been appointed and so we don’t expect this decision to be reversed. One can’t help but wonder if the new appointment has been somewhat unethical and a conflict of interest. We have decided that we no longer want to be part of an event that treats its supporters and sponsors this way.

Damian Ivereigh
Launtel
13th September 2019

 

Any parent, pet owner or manager will know that the basic tenet of managing anything is that you do your best to encourage good behaviour and discourage bad behaviour. This is the origin of the old “carrot & stick” saying.

Each of us have our own motivations and what we try to do is find ways to encourage behaviour in the people we work with that lines up with our motivations.

NBN’s Goals

So what are NBN’s motivations? There are three main goals set partly by the Government and partly by NBN themselves;

1. Roll out the network as fast as possible while keeping capital expenditure within budget.
2. Ensure the end customers have a good experience (“CX” in NBN parlance), both connecting to and using the network.
3. Raise the average revenue they get from each connection (via their customers, the RSPs) to at least $51 per connection per month (known in the industry as ARPU – Average Revenue Per User). This is to ensure the long term financial viability of the project. 

I’m not going to comment on the first goal, because this has been written about extensively elsewhere, it’s also not something that we as an RSP (Retail Service Provider), have much influence on.

The other two goals are heavily influenced by NBN’s customers, the RSPs. So coming back to my opening statement… wouldn’t you expect that NBN would support and encourage those RSPs that are helping them meet their goals? Unfortunately their actions to date haven’t demonstrated this, indeed they appear to be either doing the opposite or at best, ignoring the behaviour of those RSPs that seem to be aligning with their goals. They are behaving like the teacher who’s time is taken up attempting to deal with a few noisy, recalcitrant pupils who don’t want to be there, rather than helping those pupils that are interested in actually learning something.

The Bundles

The most obvious example is the bundles. In 2017 NBN had a big problem with two of the goals (CX and Revenue). RSPs were allowing their networks to become congested by not buying enough CVC – the capacity that NBN charges RSPs to deliver data to the RSP’s customers. This meant that end customers were complaining about poor performance (“bad CX”), which big RSPs where able to blame-shift across to NBN’s choice of technology. It was also impacting NBN’s overall revenue due to delayed take-up of services and people not interested in buying higher speeds.

So instead of asking “OK which RSPs are providing better CX and more revenue for us (i.e. align with our goals), what are they doing and how can we encourage more of it?” They instead went straight to “How can we force the big RSPs to improve the CX by providing more CVC to their customers?”

NBN released Focus on 50 and then the bundles, which essentially gives everyone a pre-determined amount of CVC bandwidth with each connection. Phil Britt, CEO of Aussie Broadband, was quoted in a presentation he gave to AusNOG: “In November 2017 nbn rewarded the big boys for behaving badly and the landscape changed again.”

The bundles were a short term fix, but they’ve created a bigger problem. While there’s no question it did almost instantly improve CX by removing congestion, the change was only temporary as our bandwidth usage has grown and we’re again seeing congestion return to some carrier’s networks. However, the second problem is that this has made it even harder for NBN to reach its $51 ARPU goal. The reason is that the bundles have homogenised the product offerings of RSPs. NBN have essentially told the RSPs how they should run their networks (how much CVC they should buy) by making it much more expensive to operate outside the expected amount. NBN took away one of the key unique selling points of those RSPs already doing the right thing: they damaged the business models of those RSPs that were generating NBN the most revenue per user.

Why is this a problem? In any market you need to have a range of products catering to different needs around price versus quality/performance. I drive a cheap Hyundai Getz because I am not interested in cars, yet some people will pay 5 times what I paid for my Getz to get a vehicle they enjoy driving or does more than going from A to B. That doesn’t make the Getz a bad car, it is just targeted at a particular market that happens to include me.

Commoditised Market

If a market is homogenised, i.e. all products are much the same, then the only thing to discuss is the price. We call this a commoditised market. This is very prone to happening anyway in the telco market where product differences are at best technical and hard to explain. This has not only lead to a fall in revenue for everyone involved, but has also lead to some atrocious behaviour with some telcos using sales tricks or even downright deception to get or keep customers, rather than product differentiation. The commoditisation makes it very hard for anyone (NBN or RSPs) to raise extra revenue by providing higher performance services, that at least some people will buy. Without the higher paying customers, NBN and the RSPs cannot afford to provide services to their lower paying customers.

Meeting NBNs Goals

For the record, at Launtel we aim to provide a very high level of network performance and customer experience. We do this by simply buying more CVC (“overage”) than most other providers along with an attention to detail and care around the support we provide our customers. We also charge more than most RSPs for an internet service. Thus we don’t cater well for those who are happy to accept a lower performance and want to pay less – we turn away people that want to buy a Getz because we are selling Mercedes. We don’t use tricks to get sales, we just let people try the product and see if it is for them – see if it is worth paying the extra money.

It should come as no surprise that we also provide more revenue to NBN per connection. As of today we pay NBN an ARPU of $55.24. In other words we are among the RSPs who are bringing their revenue average up, helping NBN meet their goals, both in terms of revenue and customer experience. You would expect them to be interested in what we have done, however to date they seem to be more interested in how to stop the big players from holding their customers to ransom by providing poor CX. 

Pricing Review

To be fair, NBN have launched a pricing review, which we have contributed to. Our submission includes a very detailed suggestion on how the CVC pricing could be restructured to encourage the RSPs looking to sell high quality/high performance/high speed/high priced services to sell more of those services. Many commentators would like to see the CVC charge gone, or have argued for the bundles to include more CVC, I don’t believe this is a good approach. It would just further commoditise the market. It would hurt NBN’s goal of raising the revenue it needs to further upgrade the network (e.g. towards the FTTP we know Australia will absolutely need within 20 years) and more to the point, make it harder for NBN to cross-subsidise the services for those who can only afford to pay less.

Don’t get me wrong I’m not looking for special treatment here, I’m just asking NBN to stop closing off a market (the high performance one) that would align very well with two of their important goals.

Damian Ivereigh
Launtel

NBNCo (the company rolling out the nations broadband network) has announced a wholesale pricing review that. As a kick off to this review they also announced that they are going to increase the CVC inclusion on the 100Mbps plans from 2.5Mbps to 3Mbps.

Unfortunately there is perception that this as a performance boost to 100/40 services. It is, but it is nothing like the 20% boost you might think it is.

The way the bundles work is that all the CVC inclusion is put into a pool (the CVC) which is then shared among all of the RSP’s bundled services – including those on other, lower speed, tiers. Given that the number of 100Mbps services is, for most RSPs anyway, a small percentage (around 10%) of the total number of services, the increase per connection is much smaller when averaged over all of the connections.

Let’s put some example numbers into this. If your RSP has 1000 services in the pooled CVC you are connected to. Let us assume that your RSP has the typical spread of speeds, which is, 90% are on 50Mbps and below and 10% are on 100Mbps and above. The former gets 2Mbps of included CVC, the latter currently has 2.5Mbps of CVC, will soon be 3Mbps of CVC.

So currently the total CVC is 900 X 2 + 100 X 2.5 = 2050

After the change the total CVC is 900 X 2 + 100 X 3 = 2100

So the increase is just a little under 2.5%. Not something to get too excited about!

If you are with a good RSP, then they will already be buying extra CVC over and above the amount included in the bundles, called overage. This overage costs $8/Mbps, so the above example RSP will see a reduction in their costs of $400 – 40c per service. Again not something to get too excited about. If your RSP is not buying overage (which may be why you are seeing congestion), then you will see a 2.5% reduction in congestion (whatever that means!) as they increase the CVC size.

To be fair to NBN what they are trying to do is make the 100Mbps plans a little more attractive for RSPs to sell, they are not reducing the cost of the 100Mbps plans rather they are including more for the same price.

My concern is that NBN is currently in a game of chicken with the large RSPs who are essentially holding their users to ransom by constraining their CVC to just the included bundle amount and waiting for the bad press to accumulate blaming NBN for the problem. Since 2017, NBN has put customer experience (or “CX” as they call it) high up their priority list, we will see whether it is above getting revenue!

At Launtel we are already buying overage because we believe that having a high performing, ultra low congestion, network is our key point of difference.  This is the main reason we are more expensive than most of our competitors.

Damian Ivereigh

June 30, 2019

We’re currently evaluating whether we continue offer 250/100 (250Mbps down and 100Mbps up) services in Canberra. Unfortunately due to NBN’s CVC pricing construct it is extremely expensive to offer these services for a relatively small number of people and we are very disappointed to say that it looks like we will just have to stop offering it.

So what’s CVC and what’s the problem with it?

NBN charges providers, such as ourselves, in two parts. The first is the AVC charge and is a single price per connection based on the peak speed of that connection. Higher speeds are higher priced as you would expect.

The second part is the CVC charge, which is the total bandwidth requested by the RSP to be used across all services in an area (called a CSA, which is pretty much just a POI – point of interconnect). In Canberra there are two CSAs: Civic and Queanbeyan. If an RSP does not purchase enough CVC then the result is congestion at peak times as there isn’t enough bandwidth to go around. NBN and the RSPs assume that a certain amount of CVC, typically just a few Mbps, is used on average per connection.

So for example if 2.5Mbps of CVC is allocated per connection, then if you have 1000 services, then your CVC should be 2500Mbps.

The problem with that calculation is that it doesn’t scale down. You have to have a minimum amount of CVC just to allow a service to work at all. Clearly you have have at least 100Mbps of CVC just to allow a single connection to peak at 100Mbps. However when you have two you probably need 200Mbps so they can both operate at the same time. However when you get more, you can start going back to the old formula. The received wisdom is that need about 3 times the highest speed of your services as a minimum on your CVC. 

NBN recognised this scaling issue and allows an RSP to get a minimum of 300Mbps of CVC effectively for free to get past this. This was clearly aimed at supporting 100Mbps services.

Great what about higher speed ones? Unfortunately NBN has done nothing to help. If RSPs want to sell 250Mbps and above, they need to buy more CVC, at $8/Mbps/month. The problem is that all that extra CVC is really just to support the higher speed services and it gets really expensive per service.

Let put some numbers on this. To support 250/100 services we really need to buy 750Mbps of CVC – that’s an extra 450Mbps, $3600 per month over what we need for the 100/40 services. We don’t actually buy that much, about 600Mbps and unfortunately we are seeing some occasional congestion when two people on 250Mbps service fire up at the same time, during peak times.

We don’t have many 250/100 services – around 15 – so that’s $240/month per connection – around $8/day. This is just the CVC! Add in the AVC charge and it’s about $11/day just for the NBN component. So yes we are making a big loss on 250/100 services!

The problem is that the higher the price, as you would expect, the less people will want to buy it, but the CVC cost is the same just spread over fewer people, making our cost per service higher. A vicious cycle.

To make matters worse our daily pricing works against us – people, quite reasonably, will just upgrade for the day just for some big downloads. The problem is we have to keep (i.e. buy) the bandwidth on hand regardless.  

Why don’t we have this problem in Tasmania?

Because we are well past the scale issue – we have about 10 times the number of connections in Tasmania that we have in Canberra, plus we setup our gigabit network in Tasmania mainly for businesses, whose CVC capacity we can use at night!

So where to from here?

We recognise that the 250/100 service is one of our unique points of difference and we are loathed to turn this off. However we have been running for almost a year and it is clear this problem is not going to fix itself.

We also recognise that some people get the 250/100 because of the higher upload speed. We could keep this part of it, by offering a 100/100 service (which is just a 250/100 service with the downstream side speed limited to 100Mbps). This would be cheaper than the current 250/100, but still at least $1.25/day more than the 100/40. Would people want to buy this?

We have considered providing a sort of “best efforts” speed where people can use the high speed if no-one else is using the CVC, however this could get quite complicated to explain and I really don’t want people to buy something thinking it’s one thing and then being disappointed.

I would appreciate any comments you have on the above – feel free to message resi-support@launtel.net.au where our staff will ensure that the emails are forwarded onto me.

Damian Ivereigh
17 Jun 2019