A brief history of middleware and why it matters today

 

This blog post is a lightly edited reproduction of a series of tweets I wrote recently
  • This tweetstorm is a mini history of enterprise middleware. It argues that the problems we solved for firms 10-20 years ago are ones we can now solve for markets today. This blog post elaborates on a worked example with @Cordablockchain (1/29)
  • I sometimes take perverse pleasure in annoying my colleagues by using analogies from ancient enterprise software segments to explain what’s going on with enterprise blockchains today… But why should they be the only ones that suffer? Now you can too! (2/29)
  • Back in the late 90s and early 2000s, people began to notice that big companies in the world had a problem: they’d built or installed dozens or hundreds of applications on which they ran their businesses… and none of these systems talked to each other properly… (3/29)
  • IT systems in firms back then were all out of sync… armies of people were re-keying information left, right and centre. A total mess and colossal expense (4/29)
  • The solution to this problem began modestly, with products like Tibco Rendezvous and IBM MQSeries. This was software that sat in the _middle_ connecting applications to each other… if something interesting happened in one application it would be forwarded to the other one (5/29)
  • Tibco Rendezvous and IBM MQSeries were like “email for machines”. No more rekeying. A new industry began to take shape: “enterprise middleware”. It may seem quaint now but in the early 2000s, this industry was HOT. (6/29)
  • But sometimes the formats of data in systems were different. So you needed to transform it. Or you had to use some intelligence to figure out where to route any particular piece of data. Enterprise Application Integration was born: message routing and transformation. (7/29)
  • Fast forward a few years and we had “Enterprise Service Buses” (ESBs – the new name for EAI) and Service Oriented Architecture (SOA). Now, OK… SOA was a dead end and part of the reason middleware has a bad name today in some quarters. (8/29)
  • But thanks to Tibco, IBM, CrossWorlds, Mercator, SeeBeyond and literally dozens of other firms, middleware was transforming the efficiency of pretty much every big firm on the planet. “Middleware” gets a bad name today but the impact of ESB/EAI/MQ technologies was profound (9/29)
  • Some vendors then took it even further and realised that what all these parcels of data flying around represented were steps in _business processes_. And these business processes invariably included steps performed by systems _and_ people. (10/29)
  • The worlds of management consulting (“business process re-engineering”) and enterprise software began to converge and a new segment took shape: Business Process Management. (11/29)
  • The management consultants helped clients figure out where the inefficiencies in their processes were, and the technologists provided the software to automate them away. (12/29)
  • In other words, BPM was just fancy-speak for “figure out all the routine things that happen in the firm, automate those that can be automated, make sure the information flows where it should, when it should, and put some monitoring and management around the humans” (13/29)
  • Unfortunately, Business Process Management was often oversold – the tech was mostly just not up to it at that point – and its reputation is still somewhat tarnished (another reason “middleware” is a dirty word!) (14/29)
  • But, even given these mis-steps, the arc of progress from “systems that can barely talk to each other” to “systems and people that are orchestrated to achieve an optimised business outcome” was truly astounding. (15/29)
  • Anyway… the point is: this was mostly happening at the _level of the firm_. The effect of the enterprise middleware revolution was to help individual firms optimise the hell out of themselves. (16/29)
  • But few back then even thought about the markets in which those firms operated. How could we have? None of the software was designed to do anything other than join together systems deployed in the same IT estate. (17/29)
  • So now let’s fast forward to today. Firms are at the end of their middleware-focused optimisation journeys and are embarking on the next, as they migrate to the cloud. But the question of inefficiencies between firms remains open. (18/29)
  • Take the most trivial example in payments: “I just wired you the funds; did you get them?”… “No… I can’t see them. Which account did you send them to? Which reference did you use? Can you ask your bank to chase?” (19/29)
  • How can we be almost a fifth of the way through the 21st century and lost payments are still a daily occurrence? How can it be that if you and I agree that I owe you some money, we can still get into such a mess when I actually try to pay you? (20/29)
  • As I argue in this post, the problems we solved for firms over the last two decades within their four walls are precisely the ones that are still making inter-firm business so inefficient (21/29)
  • What stopped us solving this 20 years ago with the emerging “B2B” tech? Easy inter-firm routing b/w legal entities (the internet was scary…), broad availability of crypto techniques to protect data, orchestration of workflows between firms without central controller etc (22/29)
  • But we also hadn’t yet realised: it’s not enough merely to move data. You need to agree how it will be processed and what it means. This was a Bitcoin insight applied to the enterprise and my colleague @jwgcarlyle drew this seminal diagram that captures it so well. (23/29)
  • And my point is that the journey individual firms went on: messaging… integration… orchestration… process optimisation – is now a journey that entire markets can go on. The problems we couldn’t solve back then are ones we now can solve. (24/29)
  • What has changed? Lazy answer: “enterprise blockchain”… lazy because not all ent blockchains are designed for same thing, plus the enabling tech and environment (maturation of crypto techniques, consensus algorithms, emergence of industry consortia, etc) is not all new (25/29)
  • But the explosion of interest in blockchain technology was a catalyst and made us realise that maybe we could move to common data processing and not just data sharing at the level of markets and, in so doing, utterly transform them for the better. (26/29)
  • In my blog post I make this idea concrete by talking about something called the Corda Settler. In truth, it is an early – and modest – example of this… a small business process optimisation (27/29)
  • The Corda Settler optimisation is simple: move from asking a payee to confirm receipt once sent and instead pre-commit before the payment is even made what proof will convince them it was done, all enabled through secure inter-legal-entity-level communication and workflow (28/29)
  • But the Settler is also profound… because it’s a sign the touchpaper has truly been lit on the next middleware revolution… but this time focused on entire markets, not just individual firms (29/29)

 

Process Improvement and Blockchain: A Payments Example

“I just wired you the funds; did you get them?”… “No… I can’t see them. Which account did you send them to? Which reference did you use? Can you ask your bank to chase?”

red and white metal mail box

The cheque is in the post…

How can we be almost a fifth of the way through the twenty first century and this is still a daily occurrence? How can we be in a world where even if you and I agree that I owe you some money, it’s still a basically totally manual, goodwill-based process to shepherd that payment through to completion?!

The “Settler pattern” reduces opportunities for error and dispute in any payments process – and does so by changing the process

This was the problem we set out to solve when we built the Corda Settler. And I was reminded about this when I overheard some colleagues discussing it the other day. One of them wondered why we don’t include the recipient of a payment in the set of parties that must agree that a payment has actually been made. Isn’t that kinda a bit of an oversight?!

Screenshot 2019-08-09 at 10.35.03.png

The Corda Settler pattern works by moving all possible sources of disagreement in a payment process to the start

As I sketched out the answer, I realised I was also describing some concepts from the distant past… from my days in the middleware industry. In particular, it reminded me of when I used to work on Business Process Management solutions.

And there’s a really important insight from those days that explains why, despite all the stupid claims being made about the magical powers of blockchains and the justifiable cynicism in many quarters, those of us solving customer problems with Corda and some other enterprise-focused blockchain platforms are doing something a little bit different… and its impact is going to surprise a lot of people.

Now… I was in two minds about writing this blog post because words like “middleware” and “business process management” are guaranteed to send most readers to the “close tab” button… Indeed, I fear I am a figure of fun amongst some of my R3 colleagues… what on earth is our CTO – our CTO of all people! – doing talking about boring concepts from twenty years ago?!

But, to be fair, I get laughed at in the office by pretty much everybody some days… especially those when I describe Corda as “like an application server but one where you deploy it for a whole market, not just a single firm” or when I say “it’s like middleware for optimising a whole industry, not just one company.

“Application Servers? Middleware? You’re a dinosaur! It’s all about micro-services and cloud and acronyms you can’t even spell these days, Richard… Get with the programme, Grandad!”

Anyway… the Corda Settler discussion reminded me I had come up with yet another way to send my colleagues round the bend…  because I realised a good way to explain what we’re building with Corda – and enterprise blockchains in general – isn’t just “industry level middleware” or “next generation application servers”… it’s also a new generation of Business Process Management platform…  and many successful projects in this space are actually disguised Industry Process Re-Engineering exercises.

Assuming you haven’t already fallen asleep, here’s what I mean.

Enterprise Blockchains like Corda enable entire markets to move to shared processes

Think back to the promise we’re making with enterprise blockchains and what motivated the design of Corda:

“Imagine if we could apply the lessons of Bitcoin and other cryptocurrencies in how they keep disparate parties in sync about facts they care about to the world of regular business…  imagine if we could bring people who want to transact with each other to a state where they are in consensus about their contracts and trades and agreements… where we knew for sure that What You See Is What I See – WYSIWIS. Think of how much cost we could eliminate through fewer breaks, fewer reconciliation failures and greater data quality… and how much more business we could do together when we can move at pace because we can trust our information”

And that’s exactly what we’ve built. But… and sorry if this shocks anybody… Corda is not based on magic spells and pixie dust…  Instead, it works in part because we drive everybody who uses it to a far greater degree of commonality.

Because if you’re going to move from a world where everybody builds and runs their own distinct applications, which are endlessly out of sync, to one where everybody is using a shared market-level application, what you’re actually saying is: these parties have agreed in some way to align their shared business processes, as embodied in this new shared application.  And when you look at it through that lens, it’s hardly surprising that this approach would drive down deviations and errors…!

I mean: we’re documenting – in deterministically executed code – and for each fact we jointly care about: who can update which records, when and in what ways. And to do that we have to identify and ruthlessly eliminate all the places where disagreements can enter the process.

Because if we know we have eliminated all areas of ambiguity, doubt and disagreement up-front, then we can be sure the rest of our work will execute as if it’s like a train on rails.

Just like trains, if two of them start in the same place and follow the same track… they’ll end up in the same place at the end.

Reducing friction in payments: a worked example

So, for payments, what are those things? What are those things that if we don’t get them right up front can lead to the “I haven’t received your payment” saga I outlined at the start of the post?

Well, there’s the obvious ones like:

  • How much needs to be paid?
  • By whom?
  • To whom?
  • In what kind of money/asset?

There are trickier ones such as:

  • Over what settlement rail should I pay?
  • To which destination must we pay the money?
  • With any reference information?

These are trickier since there is probably a bit of automated negotiation that needs to happen at that point… we need to find a network common to us both… and the format of the routing strings is different for each and so forth. But if you have an ability to manage a back-and-forth negotiation (as Corda does, with the Flow Framework) then it’s pretty simple.

But that still leaves a problem… even if we get all of these things right, we’re still left hanging at the end. Because even if I have paid you the right amount to the right account at the right time and with the right reference, I don’t know that you’ve received it.

And so there’s always that little bit of doubt. Until you’ve acknowledged it you could always turn around in the future and play annoying games with me by claiming not to have received it and force us into dispute… and we’d be back to square one! We’d be in exactly the same position as before: parties who are not in consensus and are instead seeing different information.

And it struck us as a bit mad to be building blockchain solutions that kept everybody in sync about really complicated business processes in multiple industries, only for the prize to be stolen from our grasp at the last moment… when we discover the payment that is invariably the thing that needs to happen at the end of pretty much every process hasn’t actually been acknowledged.

It would be as if our carefully tuned train had jumped off the rails and crashed down the embankment just at the last moment. Calamity!

So we added a crucial extra step when we designed the Corda Settler. We said: not only do you need to agree on all the stuff above, you also need to agree: what will the recipient accept from the sender as irrefutable proof that the payment has been made?

And with one bound, we were free!

Because we can now… wait for it… re-engineer the payment process. We can eliminate the need for the recipient to acknowledge receipt. Because if the sender can secure the proof that the recipient has already said they will accept irrefutably then there is no need to actually ask them… simply presenting them with the proof is enough, by prior agreement.

And this proof may be a digital signature from the recipient bank, or an SPV proof from the Bitcoin network that a particular transaction is buried under sufficient work… or whatever the relevant payment network’s standard of evidence actually is.

But the key point is: we’ve agreed it all up front and made it the sender’s problem… because they have the incentive to mark the payment as “done”. As opposed to today, where it’s the recipient who must confirm receipt but has no incentive to do so, and may have an incentive to delay or lie.

But building on this notion of cryptographic proof of payment, the Corda Settler pattern has allowed us to identify a source of deviation in the payment process and moved it from the end of the process, where it is annoying and expensive and makes everybody sad… and moved it to the start of the process and, in so doing, allows us to keep the train on the rails.

And this approach is universal. Take SWIFT, for example. The innovations delivered with their gpi initiative are a perfect match for the payment process improvements enabled by the Settler pattern.

The APIs made available by Open Banking are also a great match to this approach.

Middleware for markets, Business Process Management for ecosystems, Application Servers for industries..!

And this is what I mean when I say platforms like Corda actually achieve some of their magic because they make it possible to make seemingly trivial improvements to inter-firm business processes and, in so doing, drive up levels of automation and consensus.

So this is why I sometimes say “Corda is middleware for markets”.

It’s as if the first sixty years of IT were all about optimising the operations of individual firms… and that the future of IT will be about optimising entire markets.

Corda: Open Source Community Update

The Corda open source community is getting big… it’s time for a dedicated corda-dev mailing list, a co-maintainer for the project, a refreshed whitepaper, expanded contribution guidelines, and more..!

 

It feels like Corda took on board some rocket fuel over the last few months. Corda’s open source community is now getting so big and growing so fast that it’s just not possible to keep up with everything any more — a nice problem to have, of course. And I think this is a sign that we’re reaching a tipping point as an industry as people make their choices and the enterprise blockchain platforms consolidate down to what I think we’ll come to describe as “the big three”.

Read the rest of this post over at the Corda Medium blog…

Introducing the Corda Technical Advisory Council

I’m delighted to announce the formation of the Corda Technical Advisory Council (the Corda TAC). This is a group of technical leaders in our community — who most of you know well — who have volunteered to commit their time over and above their existing contributions to the Corda ecosystem to provide advice and guidance to the Corda maintainers.

Members of the TAC are invited by the maintainers of the Corda open source project (Mike and Joel) and will change over time — the inaugural members are listed below. If you’re also interested in contributing to the TAC, please do let us know — most usefully through your technical leadership and contribution to the ecosystem!

Read the rest of this post over at the Corda medium blog…!

Universal Interoperability: Why Enterprise Blockchain Applications Should be Deployed to Shared Networks

Business needs the universal interoperability of public networks but with the privacy of private networks. Only the Corda network can deliver this.

The tl;dr of this post is:

  • Most permissioned blockchains use isolated networks for each application, and these are unable to interoperate. This makes no sense.
  • We should instead aspire to deploy multiple business applications to an open, shared network. But this needs the right technology with the right privacy model.
  • Corda, the open source blockchain platform we and our community are building, was designed for just this from day one. But there was a piece missing until now: the global Corda network.
  • In this post I describe the global Corda network for the first time in public and how it will be opened up to the entire Corda community in the coming months.
  • If you’re building blockchain solutions for business, you need to read this post…

Think back to how excited you were (well, was!) when you first heard about Ethereum. The idea of a platform for smart contract applications, all running across a common network, with interoperability between all these different applications written by different people for different purposes. It was mind-blowing.

And it’s not just a vision, of course. The public Ethereum community have actually delivered it! Indeed, emerging standards such as ERC20 are a demonstration of the power of a shared, interoperable network and the power of standardisation.

So the question we asked ourselves at R3 back in 2015 was: imagine if you could apply that idea to business… imagine if different groups of people, each deploying applications for their own commercial purposes, woke up one day and discovered that those apps could be reassembled and connected in ways unimaginable to their creators but in a way that respected privacy and which could be deployed in real-world businesses with all the complexity that entails.

It seemed obvious to us that this was the right vision. And that it would require a universal, shared, open network, the topic of this post.

But it dawned on me recently that this is not how everybody in the permissioned blockchain space sees it. The consequences for users could be serious.

The rest of this post is continued at our medium site here!

View at Medium.com

What problem are we trying to solve with Corda?

Todd pointed me at a great piece about “crypto finance” versus “regular finance” by Bloomberg’s Matt Levine earlier.

I thought he did a good job of nailing the essential contradiction that arises if one tries naively to apply Bitcoin or Ethereum principles directly to traditional finance. He uses the example of an Interest Rate Swap (IRS) and how a fully pre-funded model would kind of defeat the point…

This caught my attention because an IRS was the very first project we ever did on Corda! So it’s something I know a little about… Anyway, I think the key to understanding the mismatch is captured in a post of mine from 2015 about how two revolutions are playing out in parallel.

Anyway… full details over on my Medium page.

New to Corda? Start here!

Are you just hearing about Corda for the first time? Want to understand how Corda differs from other platforms and how its unique architecture is perfectly suited to address the real problems faced by today’s businesses?

I just posted to the Corda Medium page with a list of links and background info that should help answer that question…