SDO 010 - Fostering Category Leading Data Products - Mark Palmer
What are your thoughts on data in Formula One racing?
As a huge data nerd, Formula One provides me with the intersection of fast cars, deep engineering, and unfathomable amounts of data. If a Formula One team (especially McLaren) called me and said, “drop everything, move to the UK, and help us win championships with data,” I would be on a flight within an hour. While I wait for that call, we can live vicariously through this issue’s guest, Mark Palmer— who led the team behind TIBCO’s Spotfire, which powered the real-time streaming analytics for the Mercedes AMG Formula One team (they won 1st from 2014-2021). Formula one is the perfect use case of how real-time data can transform our workflows and how data doesn’t replace us but rather augments our work and increase what’s possible.
Photo by Kévin et Laurianne Langlais on Unsplash
Hear from Mark Palmer, Board Advisor at Correlation One, Data Visualization Society, and Talkmap:
Words cannot even begin to describe how appreciative I am that Mark Palmer agreed to share his experience and expertise with the Scaling DataOps audience. Mark Palmer is the former CEO of StreamBase systems and head of products and engineering for TIBCO. He’s the co-author of “10 Things to Know About ModelOps,” World Economic Forum Technology Pioneer, and one of CDO Magazine’s Leading Voices in Data. He serves on the advisory board of Conversational AI company Talkmap, the Data Visualization Society, and Correlation One, creator of the Data Science for All educational program for underrepresented communities in tech.
*Note: There will also be more links than usual, as Mark has provided some awesome resources for us!
You have had a track record of success, throughout multiple companies, in repeatedly jumping into a new product, building effective teams, and fostering 17 category leading data products. What has been core to your technical leadership approach in enabling this success?
Mark: “It's teamwork. I know it sounds hokey, but it really is, and I know it also sounds hokey when people say, "oh, I surround myself with people who are smarter than me." But it really is a matter of not necessarily smarter; it's just different, right? If you know your own limitations, you can find people that, like, I'm very creative but also distracted, right?
My downside is I'm not, you know I'm not the most attentive of details. So my favorite people to work with our people that sort of keep me grounded. I generate ideas, they tell me when they are kind of overly ambitious. And, you know, it really is teamwork. So I don't think it's one size fits all either, but I think that's been it.
Also, just the ability to use those ideas to constantly change and fail. You know, failure, there's a great book now out by Annie Duke called Quit-- of course, she's a world champion poker player. She's had the book Thinking in Bets, and she talks a lot about the key to looking at it as one huge game, right? Like not each individual hand as one huge game where you're constantly failing or quitting and being able to quit. When I grew up I got pretty good at not getting things right. So, I guess maybe that's my biggest skill is not getting things right.
Everybody sees entrepreneurs at the end when they're big, and they're in the news. But what they don't see is when you are trying something, you're failing, your customers don't buy it, it breaks. You think you know the market, but you don't.
So I think, again, from a teamwork perspective, it's working with engineers, particularly when you're building software products that are okay with banging things out quickly. The antithesis of this idea is sort of the traditional, slow, non-agile development approach.
Seth Godin's idea, I'm a huge fan of his, he talks about artists having to ship every week, every day. He's written a blog post for 5,000 straight days for whatever last ten years or decades. Same with software. You gotta ship it as quickly as possible, not just take quick turns. So I think that working with people that have that engineering mindset is kind of special. A lot of engineers wanna get it exactly right, but that's a barrier to, I think, innovation.”
Resources provided by Mark Palmer:
The 50X Innovation Axiom
One of the products you led helped support a world leading F1 racing team. How do you get an organization, at that caliber and competitiveness, to recognize the value of your data infrastructure product?
Mark: “It's been such an incredible experience. You know, TIBCO, my company, sponsored the Mercedes AMG Formula One Racing Team, multi-time champion, one of the greatest of all time. But it wasn't just the sponsorship; they used Spotfire particularly, but our whole infrastructure, right? Like our real-time analytics software, et cetera.
So yeah, there's a lot we can say about this. And that has been said, but to directly answer your question about what was the fit really was, it comes back to what makes your product great. For the market, in our case, our philosophy of the product was about teams that are sort of hyper-competitive and hyper-connected. Agile is one word for it, but if anyone watches Formula One and particularly Toto Wolf, who's the lead for the team, you could see 'em on the headphones all the time. If you've ever been to a race or been fortunate enough to sit in the sort of the paddock area and go down into the pit, which we got to do with the team, you know, they're constantly talking.
I mean, this guy is not sitting in a corner office, and you know, figuring out if he's gonna win the race. He's on the headset all the time with everyone, the whole team. So the product fit there is. It wasn't a static dashboard. So many business intelligence dashboards are just, "here it is, here's the story, go interpret it." No, they're constantly going and exploring and figuring something out.
What a lot of people don't recognize… I love the way Toto said it once. He said "I get a performance review in public every two weeks." imagine if you're at. And your boss came to you every two weeks and said, you know, the last two weeks we're gonna dock your pay, or whatever, but worse than that, in public. So the agility that you require and the kind of tools you use to, you know, simulate. The other thing that a lot of people don't think about or know about is the prep work for the race, so the trials and stuff, it's not just a warmup. They're constantly changing the car, they're analyzing it, they're re-simulating, they're making hundreds of decisions. And so from a product fit perspective, that's what we had always designed for, which is we believe that analytics is a team sport, and to find customers, where it's sort of the ultimate team sport, was the way we designed our products that way and the way we found good customers like Mercedes.”
Resources provided by Mark Palmer:
The data market has become extremely fractured and competitive the past few years. If you were to establish a completely new data product as a category leader, what would you focus on in the first six months to ensure its success?
Mark: “I love Stuart Copeland, the drummer for The Police. He talked about sort of the evolution of the band and what it took for them to find their sound. That's what he calls this biggest lesson if, you know, take the time to find your sound. And for startups, you spend some time to find your sound, and sometimes the sound you're playing people don't like. So the two things about the six months framing, which is a really interesting way to ask the question, is I really view startups as taking six years, not six months.
I think the gestation period of new technology is a lot longer than people think. However, of course, you're constantly gigging. Like if you're The Police, you gotta go out and do a show every night and find your sound. And to me, on the tech side, that means, like we were talking about before, you know, agile, ship constantly.
And not just ship a new version or bug fixes, but ship a new idea, hack a new idea, and see if people like it. I'm not a musician, but I imagine that's what it's like on the stage, right? You try something else, try a twist, try a different syncopation. So that's what I would think of in the first six months.
If you have an idea, if you have a vision, okay, well, now go test that vision. Constantly, the more you test, the more you fail-- that's almost like hyperbole. But it's really true, and it really is hard. I was listening to Reid Hoffman, the founder of LinkedIn, and he is like, "you know, entrepreneurs have to be willing to lose a limb." It's a little bit grotesque, but it's a lot harder and grungy work. A lot of people go, "oh, I'm gonna go to a tech company to start up and make a billion dollars or something." But it's actually just kind of grungy gigging work, and that's the way I think of the first six months, though, is really to find the sound, find a customer that agrees with you and start with a few. And then you can go through the next six years to actually make a successful startup or product.”
Bonus: we had a great tangent on why streaming data is so exciting for both of us and Mark Palmer shares some insights from the trends since 2002.
Mark: “It's fantastic since, I mean, my gosh, you know, dating myself, right? Yeah. Back to 2002, got into this, and that's when, like MIT, Cambridge, Stanford, the early research was done on, you know, Mike Stone Breaker, who arguably, if there were a Pulitzer Prize for technology, he'd have won it.
You know, the inventor of a lot of the database constructs, and the idea and the vision was, is that the world was gonna become data in motion, not data addressed. And, of course, now that's true. However, you know, to your point, it's been one of the coolest, quietest market growth patterns that I've seen, and it just goes to show you how long the gestation period for technology is.
But I'm totally with you. You know that that was one of the other things about Mercedes is like what industry generates as much streaming data all the time as Formula One? When they start testing the cars, they have hundreds of sensors on board. They'll send stream information. You can push that into analytics. Something like half a terabyte of data each race weekend they generate. And yeah, there's no better playground for streaming data fans.
There are still people like Roy Schulte at Gartner, the founders of Streambase. The founders of Palma, the founders of a bunch of companies and researchers from that day that are still out there. And it's great, it's just gotten better, more pervasive. Kafka has been a revelation, the rise of it. Cuz more and more, you know, they talk about streaming data, they actually don't do streaming analytics is what we would classically define it. But it's all good, the more, the merrier, the rising tide lifts all boats.
I think it's becoming more and more commonplace in the industry. And it's really a different way of thinking, which is what I love about it is it's really thinking about what's happening now. Temporal processing of information, looking for patterns and making the connection to human decision-making.
As things are changing, which of course the world is, you know, with social media and everything being connected, it's there's no better time to be playing in the streaming space. But a lot of people kind of misperceive the idea of streaming data, cuz you've nailed it, right? Like, but a lot of people think it's all about automation, and it is about automation, but what's most powerful is when the human is in the loop sort of evaluating and guiding and nudging the system based on real-time feedback and continuous analytics.”
Resources provided by Mark Palmer:
How to Query the Future:
Person Profile:
Mark Palmer serves on the advisory board of Talkmap, the Data Visualization Society, and Correlation One. Feel free to connect with him on LinkedIn to learn more about his work.