IoT, Corfu and the future of research at VMware

At the heart of VMware’s research organization, Chief Research Officer (CRO) David Tennenhouse champions research and innovation, assuming an anticipatory stance towards technological investment and customer experience. To have any impact on the strategic direction of the enterprise, research-based insights must be actionable, advocated for, and assimilated across the entire enterprise.

At VMware, Tennenhouse and his team work to make the link between a host of technologies — notably the Internet of Things (IoT) and the in-house consistency platform Corfu — and practical business execution.

Innovation culture: Linking data to action

“A lot of [innovation-focused activity] will be somewhat roadmap-driven,” Tennenhouse explains. “And then separately, we try to have activities that are more ‘off the roadmap.’ What are things that are going to lead to new products or services, or are going to allow us to significantly change direction on existing products and services? That’s area that my team and I focus on.”

Hackathons are known as “Borathons.” They were so named by VMware engineers to reflect their last code repository island name during the transition from Concurrent Versions System (CVS) to Perforce: an homage to the South Pacific island of Bora Bora. A 2017 Borathon brought together 22 teams of VMware engineers to Palo Alto, California, in a competitive collaboration, with first prize going to the “vHackers” team for their new workflow for merging clusters.

“We also have a major internal conference,” Tennenhouse adds. “I think this year it’s going to be close to 1,600 or 1,800 of our developers — so a very large fraction of our engineering and development base comes together effectively at an R&D offsite.”

VMware

Finally, Tennenhouse underscores the value of governance and attendant proximity to innovative colleagues and new ideas, in the search for enterprise innovation. His colleague Chris Wolf, who serves as CTO, Global Field and Industry, works with top field engineers who have a demonstrated pulse on customer behavior, specifically how customers select, adopt, and integrate emerging technologies.

“So, for us, [these engineers] are actually a great interface to the company’s field organization and to our customers,” Tennenhouse says. “It’s bidirectional. In fact, they just had a big event, and I had the opportunity to go with some of my team members. We showed them some of the things we’re doing in research. Then we encouraged them to come back to us and say, ‘Here are a couple of my customers who could really use those [findings].’”

IoT and the necessity of ‘small’ innovation

VMware has long looked with institutional enthusiasm toward the IoT space. CEO Pat Gelsinger had been involved in the space through his past work at Intel on embedded computing. And Greg Bollella, an Oracle and Sun Microsystems veteran, was appointed CTO for the IoT group in 2016.

“There was enthusiasm for the space, but also it’s tempered by knowing ‘Boy, IoT is a tough space,’” Tennenhouse says.

Tennenhouse has been enamored of the technology since his days at the Defense Advanced Research Project Agency (DARPA) and its potential to fuse the virtual and physical worlds using sensors and actuators. He knew, however, that a gradual, tempered approach would help VMware better navigate some potential hurdles.

“It’s very hard to launch one opportunity in the IoT space that’s going to get traction and become large unless you actually create a vertically integrated business,” Tennenhouse explains. “We’re, of course, more of an infrastructure company.

“Similarly, the challenge to us sitting at VMware is saying, ‘Look, our customers know us more in infrastructure,’” he adds. “We’re trusted by pretty much every large enterprise in the world, but they don’t necessarily see us as a machine learning company. In the future, I hope they will see us as the company that’s going to help them do the data analysis from IoT — but that’s not where we are today.”

In a move that Tennenhouse describes as “almost anti-visionary,” yet simultaneously highly reflective of the practicalities of execution, the company instead focused on securing and managing the gateways to IoT devices.

“We could see that customers were not worrying about these challenges while conducting IoT field trials and proofs of concept (POCs) with relatively small numbers of dedicated gateways — but that they were going to be show-stoppers when rolled out at scale with millions of gateways and billions of IoT devices,” he says.

“A lot of what we had to do was think through [IoT security and management] and then say, ‘What’s the smallest thing we could do to help customers with that problem?’” Tennenhouse says. “Initially it was Greg [Bollella] and an intern, and then Greg, Mimi Spier and a few people building a prototype and getting it out there as fast they could and doing proofs of concept with customers, so they could sharpen their sense of what VMware Pulse IoT Center had to be. They could also bring those customer testimonials to the executive staff and show people that there was going to be a large opportunity for VMware in the IoT space.”

The move also represented an excellent launching point for customer dialogue, as VMware’s IoT division was learning from its OT-focused clients about their own general integration and adoption challenges.

Finally, Tennenhouse says the tilt toward IoT represented an opportunity to connect even more deeply with the global field organization. He recalls a German colleague who was especially keen on IoT applications in connected cars, and quite a bit of similar pull from Japan. This represented wonderful news to Tennenhouse because, ultimately, VMware is an enterprise company focused on the many applications — such as industry automation and supply chain — that IoT can help expedite.

Corfu: Championing an idea from the inside out

VMware Research’s open-source data project, Corfu, is a log-structured system originally designed with high consistency and high throughput. The project is representative of the potential impact of research on organizational strategy — not only in how VMware Research employees helped champion Corfu as a viable cloud-scale consistency platform from the outset, but also in the ways in which the project evolved over time.

“When you’re working in large organizations, you actually have two challenges,” Tennenhouse says. “One is finding something the customers really want. The other is, we’re just a small, little research group within a small CTO organization within a large company.”

That means to have impact, it helps for the office of the CRO to have a unified message about an idea’s practical utility and potential for return on investment. When it came to Corfu, it was enormously beneficial to have had team members join the research program with a nascent Corfu project in tow.

From there, Tennenhouse and his team relied on user feedback to morph Corfu from a high-throughput log-based object store to a more programmer-friendly and scalable platform. Tennenhouse gives the hypothetical example of a Java programmer who has defined their classes to be derived from some of the Corfu classes and, on the back end of the interface, benefits from transactional semantics superimposed in an automatic and highly scalable fashion.

“Behind the scenes, [such programmers are] getting a strong-consistency distributed object store,” Tennenhouse says. “But to them, it just looked like an in-memory object store, and they are just kind of invoking Java methods on the objects they’ve set up and that they know and love. It’s a really nice programmer interface. Again, that focus on what developers really want I think is really important.”

Tennenhouse emphasizes that the project has stimulated dialogue and collaboration between teams.

“What I think has been very remarkable is, internally, the research team really has this attitude of wanting to foist their work on the world, of wanting people to use it,” he says. “What we’ve done over the past year is really a wonderful case of the researchers really working hand in hand with developers, rather than just handing [projects] off to the networking team and saying, ‘have fun.’”

The digital horizon: Discerning actionable research areas at VMware

As VMware Research looks to the future, Tennenhouse and his colleagues will continue to solicit and synthesize a variety of perspectives — both inside and outside the company.

The VMware Academic Program (VMAP) is a major conduit for critical analysis. The program connects VMware with top researchers in what Tennenhouse describes as product-focused and product-adjacent areas, including big data, machine learning, and mobile computing.

“For example, in security, we were concerned that academic work had become too focused on exploits, [with many academics] publishing papers on new kinds of exploits as opposed to thinking systematically about security,” Tennenhouse explains. “So, there we’ve partnered with the National Science Foundation (NSF) to launch a whole new program. There was actually a joint solicitation between VMware and the NSF to call for proposals on how to use all sorts of virtualization technologies to create systemic improvements in the security landscape.”

Tennenhouse takes an enthusiastic but tempered stance on blockchain, which in his view has immense potential to create federated ledgers. Many customers want to federate with each other; banks do it through clearinghouses and enterprises through supply chains.

Federated ledgers can be run using Bitcoin’s blockchain ledger, but according to Tennenhouse, there are potential regulatory and economic implications to that approach.

“The catch is now you’re sharing,” he says. “The same ledger you’re using for your operational, really important information is being used by a bunch of folks for money laundering and other purposes.”

Private blockchains interoperating with each other made the most sense. The team examined the throughput and latency of current proof-of-work techniques of the Bitcoin blockchain, whose consensus protocol, Tennenhouse says, “is immoral in the sense that it deliberately wastes energy.”

VMware Research began to seek consensus protocol alternatives that would be less expensive per transaction, while avoiding the “very high latency, very low throughput” of the Bitcoin blockchain. Some members of the Corfu team were some of the world’s experts on consensus protocols, Tennenhouse says, and were aware of work in Byzantine Fault Tolerance (BFT).

In this context, BFT reflects the ability of a distributed computing environment — one with multiple users — to tolerate failures caused by potential bad actors. This is essential in a decentralized ledger that, by definition, is not controlled by a central authority. The research team came up with new, more scalable approaches to BFT known as Scalable Byzantine Fault Tolerance, or SBFT.

“Pushing that team as quickly as possible to develop a proof of concept, we’re now starting to work with customers on it,” Tennenhouse says.

Ultimately, whether the topic is big data or machine learning, Tennenhouse emphasizes the importance of direct communication, proofs of concept, and tangible connections between analytical insights and business needs.

“A lot of this just involves trust, obviously at the executive level, but also the individual researcher level,” he concludes. “We’ll encourage our researchers to go out there and engage with folks in the product teams and show them that they’re going to be a really trustworthy partner early on. So, later, when you come to them with the thing that’s big and disruptive, they already know they can count on you.”