SDO 018 - What Happens When Your Infrastructure Doesn’t Scale Anymore? - Panel Discussion
Panel discussion with a CTO, associate VP, and an experienced IC
Hear from… an entire panel of experts:
This edition of Scaling DataOps is a little different in that I have three technical leaders sharing their insights. In January, we did a live version of the Scaling DataOps Newsletter at the Data Teams Summit, which was awesome. I’ve taken the highlights of this panel from each speaker to share with you all! At the time of this recording, we had the following guests with respective titles [follow them on LinkedIn]:
Sarah Floris - Senior Data & ML Engineer, and founder of Dutch Engineering
Ben Doremus: Chief Technology Officer - Magenta Care Continuum
Richad Nieves-Becker: Sr. Associate VP, Data Science - Revantage
My goal with this panel was to look at a complex problem from three different perspectives of experienced IC, senior manager, and c-suite… and my data friends DELIVERED on this panel. I highly encourage you to check out the full panel, as there are some awesome discussions that I didn’t have space to include here. Enjoy!
— Markl
Real quick… my goal is to keep this newsletter free for my audience and have sponsors pay. Engaging with the following would be a huge help if you want to support this newsletter!
This edition of the Scaling DataOps Newsletter is sponsored by SingleStore who is hosting the free webinar: How to Build GPT-chatbot Apps with SingleStore & MindsDB on March 30th at 10 AM PST.
You can register for the event using my featured link, where every signup goes a long way in supporting the Scaling DataOps newsletter:
What are the early signs that your data infrastructure isn't scaling anymore? Specifically, what is the impact you see on the technology, product roadmap, and data teams themselves?
Sarah: “This a problem that I see on a regular basis, right? You're scaling your small startup, you're medium company, you're scaling and suddenly you get all of these warnings from places. And that's hopefully only if you have monitoring.
Otherwise, your pipeline just start crashing, your data is stale, the things that you see on a regular basis when your infrastructure doesn't scale anymore. And so, when I see it as a data platform engineer, I am immediately alerted to that, and then I have to go in and do those reactive changes, and it becomes a very much more reactive environment, which is not always the best way to approach these problems, right?
So the first thing I always typically want to do when I start a new data project or things like that: have monitoring. Usually, the first sign is getting those monitors, having those alerts set up, right? And so that's really what happens. You start having a lot of people, customers, stakeholders, leadership come to you and be like, "Hey, what is happening to my data?" And so that often happens. So the first one is the backup of logs. That's what you all see in the data teams, you'll get a lot of folks who can't take on any more requests because we are being so reactive in our responses to the data pipelines.
And then the other thing is the projects are not getting done. You'll see the technology also giving warning signs, you'll see leaders starting to complain, but you'll also see customers come in and talk about what is happening.
And that is usually like when you have an embedded analytics product. And so those are the things that I would definitely watch out for What are your data engineers doing? What are your data scientists doing? Making sure data analysts, what are their complaints? And a lot of the time you'll hear a common theme and that's, like the queries are taking slower than they're supposed to be. And then, the data engineers are not getting my work. I'm not receiving those of the data for those ad hoc requests. And so those are definitely something you have to really watch out for.”
Scaling often puts you within the build vs. buy debate to meet your new data needs. As a leader, how do you navigate through this decision while also accounting for the needs of your growing data team?
Ben: “This is a very easy question and a very hard one at the same time. If you've got a wide open budget, you've got the options of doing whatever you want. If you have constraints, then you gotta figure out what's the most valuable. And I think this really comes down to a philosophy of who you are as a company.
You look at this with like the cloud, right? Why is the cloud so popular? Because people don't wanna be managing their own infrastructure. That's not the strength of their business. They don't want that taking up their brain space. They wanna focus on solving the problems and just have computers exist so that they can use them.
Yeah, it's more expensive, but sometimes that's worth it. You know, it's not all just a cost thing, it's who are you, what is the thing that you wanna focus on? Can you clear out your brain by offloading all these other things? You know, everyone's got Salesforce. Nobody wants to build the integrations. That's not a part of any company is managing their own Salesforce integrations.
It's just buy that, get that figured out somewhere else. But there's also really interesting questions between like, okay, do you buy it from a bespoke company. Do you get a consulting firm to build this for you? Do you go with an open source thing and try to hack it on your top? A lot of that still comes back to your philosophy as a company.
Who is it we want to be as a team? Do we want to scale up in this? Sometimes it's worth it to say like, "it's gonna take us longer, it's probably not gonna save us a ton of money, but people wanna do this, they want to give it a shot." I'm not talking resume driven development here, but there's something to be said for using the strengths in the way that people want to use them, allowing them to grow and allowing them to stretch.
So, in a world where everything's free, you just offload everything except what you need to focus on as a business. But when it's not, you just need to figure out what is the truest to who we are? What can we afford and what can we get away with for now? Not buying until we're really gonna need it later.”
Investment to scale data infrastructure is unique because it requires high upfront costs for long-term ROI. How do you get buy-in from leadership to make such investments when the pain of data scaling isn't felt yet by the broader organization?
Richad: “There's a couple options. Nothing speaks like the pain of experience, right? So I think the best way to do this is to simply give an executive a slow app. Infrastructure sells itself. Here's all these logical arguments, or you could just give them the slow app, that you built MVP style, which you should, no matter how big your company is. Right? And then you're like "okay. Is this valuable? Oh yeah. Yes. By the way, it's slow. In order to make it not slow, we need X, Y, Z." So I would say that's one way to do it.
The more broad answer is storytelling. So instead of giving them that experience. You can also paint a picture of what the experience will be like in a very concrete manner. Because ultimately I think things that you can't see always come back to the things that you can, the outcomes, the dashboards, the ML, the performance.
And so basically you have to often yourself think and do the creative work, hard work of figuring out how it ties to business outcomes. You also can tie it into where you want to be in three years. If you are defining a roadmap or you have one defined, you could point to those things and say, "this will be very difficult to do with this many customers versus these few customers," or "this is gonna be really difficult when we scale to Europe and Asia versus just here." So you basically use things that you know, they already care about as hooks and then you hang your idea on them.
I think Ben gave a good example where it's like, We hire the team ourselves and then you draw out the cost for them. Or you buy this thing and maybe you try to point out or estimate, okay, where does buying the thing become more expensive? And you say " by the time we get there, we'll be concerned about these other things." You try to make it fairly airtight, but it's impossible to make it fully airtight. Ultimately buy-in is people's beliefs, and the beliefs are changed one person at a time, one experience, and one story at a time.
General principles, tips, I guess work backwards from value. Show don't tell, and hopefully you're working with a team with a long-term view because if, back to the first question, if everyone's super reactive, it's gonna be very difficult. There's no magic answer to convince anyone of anything. If we had that, then there would be peace in the world and stuff, right? But obviously you can't always convince everyone that of everything you need. Ultimately it does depend a little bit on your environment. There's no perfect answer.”
What are others saying in the DataOps space?
Meaningful metrics: How data sharpened the focus of product teams
What: Duolingo highlights how moving away from aggregate data for key product metrics opened up new insights and possibilities.
Why: This is a great article to learn more about the challenges faced by our data colleagues towards the end of data pipelines.
Who: You either work in or support a product-focused team with your data infrastructure.
Timeless Obstacle for Data Products: Data Quality
What: A great overview of the various components of data quality within a product.
Why: The author highlights how ubiquitous data is in all products and how data quality is essential to a product’s success.
Who: You are starting to take data quality seriously within your org and want a starting point to understand how.
Scalable Annotation Service — Marken
What: Data labeling is essential for ML and search within a data product and Netflix shares how they do this at scale with Marken.
Why: The article includes an awesome architecture diagram and in-depth explanations.
Who: You love seeing the innovations of big tech players and looking into how you can potentially apply them yourself.
About On the Mark Data:
On the Mark Data helps brands connect to data professionals through captivating content, such as this newsletter and other featured content! Please feel free to check out my website to learn how I can support your data brand via influencer marketing or content and go-to-market strategy consulting.