What happens if Bitcoin mining companies vertically integrate?

Creeping centralization can come in many forms – are the Web Application Server wars a lesson from the past?!

First some history of the IT industry (you can skip this bit if you just want the meat…)

Travel back in time with me to the late nineties…. I started my career during the Web Application Server wars. The web was taking off in the late nineties and companies like WebLogic (quickly acquired by BEA) spotted a need in the marked for a Web Application Server.

They were solving an obvious problem: it was a real pain to write proper applications. Sure: you could write HTML and run it on Apache. And you could use the CGI interface to have scripts run to generate basic dynamic content. But it was painful to do anything more sophisticated

If you were a bank that wanted to build an online banking service that interfaced with your core banking system, it needed a huge amount of effort and care… and you were wholly responsible for figuring out how to manage session state, failover, transactionality and everything else. And what we quickly found was that everybody needed a similar set of code “middleware” services – and were building for themselves each time.

And one solution to this problem was the web application server. This was a server, usually written in Java, that provided a defined set of APIs and an execution environment for web applications.    You could think of it as having two faces: one face spoke “web” – it sent and received HTTP. The other face spoke “enterprise” – it provided a set of APIs, SLAs and common behaviours that app developers could rely on to build their applications.

Products like BEA WebLogic, as it became, Netscape Enterprise Server, IBM WebSphere Application Server and Microsoft’s Internet Information Services platform were all designed to solve this problem… and it became a billion-dollar industry.

Sure – somebody still had to buy, configure, install and manage these platforms (usually in corporate datacenters, running on expensive hardware) but at least there was now a clear separation: and a division of labour emerged: J2EE developers, App Server administrators, infrastructure engineers and so on.

Today is different but not that different!

Fast-forward to today and the world is different: we take a rich execution environment for granted and the state-of-the-art consists of cloud offerings that protect the developer from worrying about anything apart from their code: infrastructure, connectivity, bandwidth, middleware configuration is all delegated to the platform. (IBM’s BlueMix is a recent example of such a service)

But the insight is the same: in any field, we see lower levels of the stack become standardized and commoditized. A small number of specialized players, with economies of scale and benefitting from experience-curve effects, begin to dominate.

So I’ve often wondered if we’ll see something similar in the Bitcoin space.   Are there any parallels we can draw and, if so, what can we learn from them?

Can we see any parallels with Bitcoin infrastructure?

Look back a few years and anybody wanting to build a product or service on Bitcoin had to do it all for themselves.

For example, it seems that Mt.Gox wrote their own custom implementation of the Bitcoin protocol – which is not, perhaps, surprising given the nature of its business and the maturity of the ecosystem when it was founded. I guess that’s similar to people who wrote their own HTTP servers or implemented custom schemes for producing dynamic content back in the nineties.

More recently, APIs like BitcoinJ and platforms like BitsOfProof have emerged as credible abstraction layers upon which interesting products and services can be built. In both cases, developers typically run and maintain their own instance(s) of  Bitcoinj or BitsOfProof, etc. So one could argue this is a parallel to the on-premise app-server days of the web.  

So it’s tempting to ask… are the parallels with the web appropriate? If so, what happens if cloud-based services for Bitcoin developers emerge?

And, of course, we don’t need to imagine. These services already exist. chain.com is one such example.

Chain.com is a cloud-based Bitcoin API for developers. From their website, it appears that they maintain a set of Bitcoin nodes and ensure they are well-run and well-connected, perhaps in the way that Amazon or IBM might maintain app servers, infrastructure, routers and connectivity to major peering points on the internet.

And developers simply write their Bitcoin product or service in a language of their choice, calling the Chain APIs as and when they need to interface with the Bitcoin network.  One can imagine that coding against Chain.com makes it an order of magnitude more easy to get a new Bitcoin company up and running. You don’t need to worry about any of the infrastructure detail: Bitcoin is abstracted away to a set of easy-to-use, well-documented, high-level APIs.  

You might hardly even know you were even using Bitcoin!

And that’s a really important subtlety: if you are not running your own Bitcoin nodes, how do you know the information your service provider is giving you is accurate? How do you know you’re connected to the “real” network and seeing the real blockchain and that your transactions are making it across the network?

Now I’m sure the chain.com team are hugely competent and ethical people. This post is emphatically not a criticism of them or their service. But one does need to be open-eyed about the tradeoffs one is making as an engineer.  And one approach app developers could take might simply be to maintain their own independent copy of Bitcoin core to act as a monitor/sentinel.  It’s probably not a show-stopper.

And I can see this sort of service being immensely popular: it is a lot of effort to install and maintain Bitcoin network infrastructure… keeping it patched, making sure you really are well-connected to the Bitcoin network and so on… and if your business has latency dependencies (mining? Exchanges?) then you also need to worry about making sure your nodes are “close” to other major nodes. And so on and so forth.

Figuring out how to do this sort of stuff well has good scale effects and there will be big learning/experience curves. Just as cloud vendors can run web infrastructure better and cheaper than most companies could do themselves, we should probably expect to see small startups outsourcing their Bitcoin network integration to firms like chain.com – and maybe even hosting providers who include Bitcoin APIs and network abstraction as a core part of their service offering.

So, leaving aside the concerns this might raise around decentralization and ask yourself:

if a cloud-based Bitcoin API/application platform were to become really popular, who is best placed to deliver it?

Until a few weeks ago, I would have said I expected more firms like chain.com to be founded over the coming months and for a handful of them to gain critical mass and become the dominant application platform players of the future.

And then CoinTerra bought BitsOfProof

It was one of those lightbulb moments for me when I heard about it. Of course! How Obvious!

It makes perfect sense. Think about the capabilities and advantages that big mining firms have that they could use to build a powerful hosting proposition for developers:

  • Hosting in low-cost locations
    • They already maintain massive farms of servers, usually in locations with cheap power. They could easily offer colocation to their clients
  • Massive hashing power
    • They could commit SLAs to application clients that nobody else could make – “At this SLA, we will guarantee your transactions make it into a block within x hours”, perhaps?
  • Optimised network connectivity
    • The mining firms must surely have deep knowledge of the Bitcoin network topology so I suspect they could make promises about transaction propagation times and other time-critical requirements that would measure to some clients – “You will see transactions from other people faster on our platform than if you ran your own node”, perhaps?
  • 100% Virgin coins on tap
    • Mining companies can offer businesses who use their APIs access to freshly-minted coins… no taint, no need to ask where they came from
    • … and I’m sure there are more

Many observers, myself included, have tended to ignore mining, regarding it as an infrastructure function, with the main debate being around the threat of concentration of hashing power. What I missed until now is that a well-run mining platform is also a phenomenally powerful platform from which to build up the stack to middleware/APIs and beyond.

So the question for me is: what happens if (when?) one of the mining firms offers a hosted application environment for Bitcoin companies with a really easy-to-use abstraction API over the Bitcoin network?   Commentators such as Tim Swanson have written extensively about the incentive structure of Bitcoin mining but I’ve not yet seen an analysis that thinks through the implications of vertical integration of the sort I outline here – and what it would mean for any security analysis of the system as a whole.  

If anybody’s aware of one, I’d love to read a copy.

Disclaimer: I’ve not spoken to any of the firms named in this post and I have no insight into their product plans

A simple explanation of fees in the payment card industry

I wrote a piece last month explaining how the payment card industry works.  I talked about the various actors (acquirers, issuers, schemes, merchants, etc) and pointed out how weird it is that everybody knows the Mastercard and Visa brand names yet nobody actually has a relationship with them. One of the questions I didn’t address there was fees.  Who makes all the money? Why does it seem so expensive?

Let’s start with the standard four-party model: Merchants, Acquirers, Issuers and Schemes:

Four Party Model Fees 1

The four-party model: Merchants obtain card processing services from Acquirers, who route transactions via Schemes to Issuers, who debit Consumers’ accounts.

A Worked Example

The key point is that one firm from each category is going to be involved in every payment card transaction.  So it’s interesting to ask: how much do they get paid?

Let’s take a concrete example and work it through.  Bear in mind: this is just an example. As you’ll see, there are almost infinite variations and some merchants will pay fees considerably higher than the ones I discuss below.  Also, note: this information all comes from public sources.  I use company names below for clarity but I have no private insight or information into their fee structures

The Scenario

Let’s imagine I’m using a Visa Debit card, issued by a US bank (let’s say Bank of America) to buy $100 of goods from an online retailer.   What happens?  From my perspective, of course, it’s obvious: I’m paying $100!

Four Party Model Fees 2

Imagine I am using my Visa Debit Card, issued by Bank of America to pay for $100 goods from an online retailer.

The Merchant’s Perspective: The Merchant Discount Fee

What does the merchant see?  Well, the merchant will have a contract with an acquirer.  What does that look like?  Let’s take an example.  Costco have a page on their website that refers small merchants to Elavon for acquiring services.  Let’s use the pricing displayed on that page for Online transactions:

1.99% plus 25c per transaction (plus some other recurring/monthly fees, etc)

Many readers will be thinking that seems low but let’s go with it for now.

So, for our $100 transaction, we can calculate how much money the merchant will actually receive from Elavon/Costco:

  • Transaction value: $100
  • Elavon/Costco takes 1.99% + 25c = $2.24. This is often called the “merchant discount fee
  • So merchant gets $97.76

So our picture now looks like the one below:

Four Party Model Fees 3

Merchant receives $97.76 from the $100 transaction. Elavon gets $2.24.  But how is the $2.24 distributed between the acquirer, issuer and scheme?

The Issuer’s Perspective: The Interchange Fee

So we know how much money the merchant has paid to the “credit card industry”. But how is that money allocated between all the participants?   Visa Inc has a very helpful document on their website, which tells us part of the story: Visa U.S.A. Interchange Reimbursement Fees.

The key word here is “Interchange”.  Interchange is the fee that gets paid to whoever issued the card – and it’s set by the scheme (Visa in this case).   You’ll see in that document that this is not straightforward… there are pages and pages of rates:  the interchange fees vary based on whether the card was present or not – and on the type of good or service being bought, whether it was a debit or credit card, whether it was a corporate card, whether it was an international transaction and lots of other criteria…

So let’s just pick a simple example.  We’ll go with page 2 – “CPS/e-Commerce Basic, Debit” (Card not present).

Aside: CPS means that the merchant has complied with various Visa rules (such as validating customer address to reduce fraud risk, etc) and has thus qualified for a low cost option.  

So the issuer is entitled to 1.65% + 15c

  • Transaction value: $100
  • Issuer receives 1.65% + 15c = $1.80.  This is the interchange fee
  • So issuer owes $98.20 to the other participants (Visa, Elavon and the Merchant)

And we already know that the merchant only gets $97.76 of that money (their merchant discount fee was $2.24, remember?).  So that means there is 44c left to share between Visa and Elavon.

The diagram below shows the current state of the calculation:

Four Party Model Fees 4

Interchange Fee (what the issuer gets) is $1.80

So how is the remaining 44c allocated?

We’ve assumed the switch is Visa so we need to know much they charge.  CardFellow.com has a good explanation.

We’ve assumed a Visa Debit card so, according to that site, Visa’s fee, which we call the “Assessment” is 0.11%.   There is a menu of other charges that might apply but we’ve assumed this is a low-risk “CPS” transaction so we’ll assume none of them apply.  (In reality, the 1.55c “Acquirer Processing Fee” probably applies)

  • Transaction value: $100
  • Visa assessment is 0.11% so Visa charges 11c. 
  • So there is $98.09 to pass on to the acquirer.

And if there is $98.09 to pass on to the acquirer and we know that the merchant receives $97.76, that must mean there is 33c left for Elavon.

So there we have it… in this VERY SIMPLE, highly contrived – and probably unrepresentative – example, we end up with the result in the diagram below:

  • Consumer pays $100
  • Issuer receives $1.80
  • Visa receives $0.11
  • Acquirer receives $0.33
  • Merchant receives $97.76 – overall fee $2.24

Four Party Model Fees 6

Final picture showing how the merchant’s $2.24 fee is allocated

As I’ve stressed above, this is just a simple example but it shows two key points:

1) It is the issuer who receives the bulk of the fees (this is, in part, how they fund their loyalty schemes, etc)

2) The schemes actually earn the least, per transaction, of any of the participants.  This underlines, again, how powerful their business model is:  by being at the centre of a very sticky network, they can earn a lot of money overall by charging very low per-transaction fees.   [Edit 2013-08-10 10:35 : it’s also worth noting that the acquirers and schemes have pretty much fixed-cost infrastructures – unlike issuers, who need to hire customer service and debt collection staff in proportion to number of cards issued. So the schemes and acquirers also benefit disproportionately from rising volumes.  So: low fees for schemes/acquirers for sure… but HUGE volume is what enables them to make big profits]

[Note: I use blog posts like this to help clarify my own thinking and understanding – as well as to share knowledge…  and there are one or two pieces here where I’m not 100% confident I got it totally correct… so please do tell me where I’m wrong if you spot something]

[Update 2014-08-09 18:47 Minor typos and replaced last diagram]